Image processing system and image processing method

Abstract

The present invention aims to provide a technology that enables users to easily select a position for measuring the three-dimensional size of an object. [Solution] The image processing system of the present invention comprises means for capturing multiple images containing the same subject; means for extracting feature points from each of the multiple images and determining corresponding points by comparing that the feature points in each image are corresponding points that indicate the same part of the subject; means for converting the corresponding points into a 3D point cloud, which is a point cloud in 3D space; measurement position input means for specifying a measurement position for measuring the size of the subject; and size estimation means for estimating the size of the measurement position. The measurement position input means specifies the measurement position using coordinates in 2D space and projects the coordinates in 2D space into 3D space, and the size estimation means measures the size of the measurement position in 3D space.

Description

Technical Field

[0001] The present invention relates to an image processing system and an image processing method for estimating the size of a subject using input data.

Background Art

[0002] Image processing technologies for photographing a subject such as a person or an object and inferring the planar or three-dimensional size (length, size, area, volume, etc.) of the subject based on the image thereof are widely used in various fields such as industry, city, transportation, medical, energy, and IT.

[0003] For example, in an automobile driving assist function, safe driving is supported by recognizing the situation around the automobile, such as the distance and size of obstacles in front or on the side, from the image of an in-vehicle camera and various sensor information. Also, by observing a building or a bridge from the image of a camera mounted on a drone and measuring the presence or absence and size of cracks or collapses, it becomes possible to perform maintenance inspections of facilities. Also, at a manufacturing site, by photographing a manufactured product with a camera and sensors arranged on a production line and determining the presence or absence and size of abnormalities such as scratches, it is possible to improve the quality of product manufacturing.

[0004] The form of subject size measurement includes a case where a determined part of the subject is automatically measured and a case where a person checks the size of an arbitrary part as part of analyzing the situation of the subject. Although the needs for the former have been increasing with the recent development of AI, there are still many scenes where the latter person (user) and machine perform image analysis while interacting with each other, and a technology for efficiently specifying an arbitrary part on an image and measuring the size is also important. Also, practically important is to obtain a three-dimensional size, and by obtaining a three-dimensional size, it may be possible to know detailed information of a subject that cannot be grasped only by planar information.

[0005] Technologies related to obtaining the size of an object include, for example, those described in Patent Documents 1 to 3. Patent Document 1 addresses the problem of performing high-precision calibration of a wide-angle stereo camera without increasing the size of the chart, and discloses the following invention of a stereo image processing device and image correction means: "This stereo image processing device comprises a stereo matching unit that performs stereo matching of multiple images captured by an imaging unit that images an object with multiple cameras and detects parallax, and a pixel shift correction processing unit that corrects the pixel shift when the object is imaged. The field of view of the imaging unit has a first field of view range and a second field of view range that is wider horizontally than the first field of view range. The pixel shift correction processing unit performs pixel shift correction processing of the second field of view range based on the information of the first field of view range." Furthermore, Patent Document 2 aims to provide a point cloud data processing device, a method, and a program that can optimize 3D point cloud data to be processed, and discloses the following as an invention of a point cloud data processing device, a method, and a program: "The point cloud data processing device A of the present invention determines whether or not there are data gaps in the 3D point cloud data of a target object, performs a data gap determination interpolation process to interpolate the data gaps if it is determined that there are data gaps, determines whether or not there are outlier data in the 3D point cloud data after the data gap determination interpolation process that deviates from the data values ​​representing the target object, performs an outlier determination removal process to remove the outlier data if it is determined that there are outlier data, and repeats the data gap determination interpolation process and the outlier determination removal process, respectively, until it is no longer determined that there are outlier data or until a predetermined upper limit is reached." Furthermore, Patent Document 3 addresses the issue of reducing the burden of image management by automating the linking of multiple images and the retrieval of specific images, and discloses the following inventions for an image management device, an image management method, and a program: "The image management device 10 includes a 3D point identification unit 11 that identifies 3D points corresponding to feature points extracted from a 2D image obtained by photographing an object, among the 3D points that constitute the 3D point cloud data of the object; an image management unit 12 that associates the identification information of the 2D image from which the extracted feature points were extracted with the 3D points corresponding to the extracted feature points; a 2D image identification unit 13 that, when a specific part of the object is specified, identifies 3D points corresponding to the specified part, and further identifies the identification information of the 2D image associated with the identified 3D points; and a display unit 14 that displays the 2D image with the identified identification information on the screen." [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2022-109555 [Patent Document 2] Japanese Patent Publication No. 2023-059688 [Patent Document 3] Japanese Patent Publication No. 2023-20180 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] Estimating the three-dimensional size of an object using a camera or sensor employs various methods to reconstruct a 3D point cloud of the object based on different principles, such as Structure from Motion (SfM) using images with parallax, Multi View Stereo (MVS), Light Detection and Ranging (LiDAR), radar surveying, and Time of Flight (ToF). In any of these methods, if the reconstruction accuracy of the 3D point cloud is low (the original shape cannot be determined due to missing or blurred data), the measurement error increases, and in particular, the measurement error varies depending on the location of measurement.

[0008] As a result of this, when a user specifies a location on an image, particularly in 3D, to measure its size, the 3D point cloud is incomplete, making it difficult to confirm its correspondence with the original shape. This makes it difficult to specify the measurement area in 3D space using a GUI. None of the patent documents from 1 to 3 address this issue. Therefore, the present invention aims to provide a technology that enables users to easily select a position for measuring the three-dimensional size of an object. [Means for solving the problem]

[0009] To solve the above problems, one representative image processing system of the present invention comprises means for capturing multiple images containing the same subject; means for extracting feature points from each of the multiple images and determining corresponding points by comparing that the feature points of each image are corresponding points that represent the same part of the subject; means for converting the corresponding points into a three-dimensional point cloud, which is a point cloud in three-dimensional space; measurement position input means for specifying a measurement position for measuring the size of the subject; and size estimation means for estimating the size of the measurement position, wherein the measurement position input means specifies the measurement position using coordinates in two-dimensional space and projects the coordinates in two-dimensional space into three-dimensional space, and the size estimation means measures the size of the measurement position in three-dimensional space. [Effects of the Invention]

[0010] According to the present invention, the user can easily select a position for measuring the three-dimensional size of an object. Other problems, configurations, and effects will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 shows an example of the configuration of the image processing system in Example 1. [Figure 2] Figure 2 shows an example of the program structure stored in memory in Example 1. [Figure 3] Figure 3 shows an example of the processing procedure of the image processing system that estimates the three-dimensional size of an arbitrary position of an object, as described in Example 1. [Figure 4] Figure 4 shows an example of the input UI screen for the position of the object to be measured in step S304 of Figure 3. [Figure 5] Figure 5 shows an example of a UI for inputting the position of the part of the subject to be measured in steps S312 and S313 of Figure 3. [Figure 6] Figure 6 is an explanatory diagram illustrating the case in steps S314 and S315 of Figure 3 where size estimation is performed using a 3D point cloud and a size-specified line segment. [Figure 7] Figure 7 shows an example of a processing flow for predicting the level of error during size estimation. [Figure 8] Figure 8 is a schematic diagram showing the projection point cluster distributed in the vicinity of a specified size line segment. [Figure 9] Figure 9 shows an example of the distribution of the error rates for evaluation values, final scores, and size estimation. [Figure 10] Figure 10 shows an example of displaying the size of the measured subject and the level of error. [Modes for carrying out the invention]

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following description and drawings are examples for explaining the present invention, and for the sake of clarity of explanation, appropriate omissions and simplifications have been made. The present invention can also be implemented in various other forms. Unless otherwise limited, each component may be singular or plural.

[0013] In the drawings, the positions, sizes, shapes, ranges, etc. of each component shown may not represent the actual positions, sizes, shapes, ranges, etc. in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the positions, sizes, shapes, ranges, etc. disclosed in the drawings.

[0014] Also, in the following, when explaining the process mainly based on the program, it is shown that arithmetic devices such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) execute the program using computer resources such as a storage device and a communication interface. Note that the functions realized by the program may also be realized using a virtual computer, a computer system, or a dedicated circuit such as an FPGA (Field-Programmable Gate Array) and an ASIC (Application Specific Integrated Circuit).

Embodiment

[0015] In Example 1, a case of an image processing system that utilizes a drone for measuring the size of a subject in maintenance inspection work of structures or artifacts such as buildings, bridges, and roads will be described. The drone is equipped with a camera, an acceleration sensor, a gyro sensor, and a GPS (Global Positioning System) sensor. The drone captures the maintenance inspection target by the user's operation or the automatic navigation function, and utilizes the video and sensing information obtained at that time to measure the size of the subject. For example, it is possible to estimate the size of cracks on the surface of a building or the size of fallen rocks on a road during disasters such as earthquakes and floods. The user of the system confirms the size estimation result and utilizes it for subsequent maintenance inspection work and restoration work. Although the case of applying the drone is described, the present disclosure is not limited to this case. If it is a configuration capable of specifying the shooting position, the present disclosure can also be applied to a general aircraft.

[0016] In addition, in this example, unless otherwise specified, a building will be described as the subject, but the measurement target is not limited to a building and can be arbitrarily set. Also, in this example, the case of estimating the size of the subject will be described, but various information such as the position, type, and performance of the subject may be estimated together using the image and sensor information. Further, the size estimation result can also be utilized as basic information for estimating various information.

[0017] (System Configuration) FIG. 1 is a diagram showing an example of the configuration of the image processing system of Example 1. FIG. 1(a) shows the configuration of the image processing system, and FIG. 1(b) shows an example when the image processing system is applied.

[0018] As shown in FIG. 1(a), the image processing system 1 includes a terminal 100 and a photographing device 101. The terminal 100 and the photographing device 101 are connected via a network not shown.

[0019] Terminal 100 is a device for the user to operate the camera 101 and estimate the whole or partial size of the building 103, which is the subject of the camera, and is, for example, a smartphone or tablet. Terminal 100 has a CPU 110, memory 111, input device 112, output device 113, and network interface (NW IF) 114.

[0020] Memory 111 stores a program that performs object size estimation. For example, the user can use this program to obtain an estimated size of an object of interest during maintenance inspection while reviewing the captured image.

[0021] In this embodiment, size includes linear length, curved length, and the area or volume of a region enclosed by a line or curve. However, in subsequent embodiments, unless otherwise specified, the size of the subject will represent a straight line with a finite length indicated by a line segment.

[0022] The output device 113 is, for example, the display of a smartphone or tablet, and displays the user interface provided by the program. The input device 112 is, for example, the touch panel of a smartphone or tablet, and provides the user with the ability to input information for controlling the imaging device 101 and to select a location for which size estimation is to be made. The program stored in memory 111 may be designed to include a function for inputting image analysis results related to maintenance and inspection, such as the location, type, and comments of any problems that may arise during maintenance and inspection. In this case, for example, the output device 113 may display a text box, and the input device 112 may input this information into the program.

[0023] The imaging device 101 is a device for photographing a subject whose size is to be estimated, and is, for example, a drone with imaging capabilities. The imaging device 101 has a camera 115, a sensor 116, and a network interface 117.

[0024] Camera 115 is a device that captures still images or videos of a subject, and is generally implemented using a combination of a lens and an optical sensor. It is also assumed to be equipped with an autofocus function that automatically focuses on the subject, an auto white balance function that adjusts color information, and an auto exposure function that automatically adjusts brightness. Note that the camera is not limited to a color camera; it may also be a monochrome camera. The camera has internal parameters that represent camera-specific characteristic values ​​such as focal length and aspect ratio, and external parameters that represent the camera's position and orientation in each frame of multiple captured still images or videos. At least the internal parameters are assumed to be known, and it is possible to roughly determine the external parameters using the sensor information described later. Note that when there is no need to particularly distinguish between internal and external parameters, they may simply be referred to as camera parameters.

[0025] Sensor 116 is a device that acquires sensing information including information about the surroundings of the imaging device other than the image itself. Examples include an acceleration sensor or gyro sensor that detects the orientation of the camera or the imaging device itself when taking an image of a subject, an illuminance sensor that measures the brightness of the shooting environment, a GPS sensor that measures the absolute position of the shooting location, and an altitude sensor that measures the height of the shooting location.

[0026] The network interface 117 is an interface for communicating information between the terminal 100 and the imaging device 101, and can send communication data 105 bidirectionally. The communication data includes, for example, images and sensor information transmitted from the imaging device to the terminal, and control signals that control the movement of the camera and the imaging device itself, transmitted from the terminal to the imaging device. Public wireless communication and wired communication can be used as means for transmitting data.

[0027] Figure 1(b) shows a case where a drone 121 equipped with a camera and having a shooting function is used as an example of a shooting device 101, and a tablet 120 is used as the terminal 100. For example, user 102 operates the drone 121 with the tablet 120 and performs maintenance inspections on building 103 as the subject. As an example of maintenance inspection, user 102 checks areas of the building 103 that require maintenance and repair, such as cracks, peeling, corrosion, fading, and paint peeling on the side walls. The memory 111 of the tablet 120 stores a program for performing size estimation of the subject. User 102 uses this program to specify the maintenance and repair area to be measured and detects the size of the area. User 102 uses the size estimation results as a reference for analyzing the cost and duration of repairs, and for estimating the asset value of the building. In the following explanation, the case of Figure 1(b) may be referred to.

[0028] When a drone 121 is used as the imaging device 101, the components included in the imaging device 101 include a propeller, motor, battery, software for controlling flight via drone operation instructions from terminal 100 or autonomous automatic navigation functions, CPU, memory, etc., but these are not shown in this embodiment.

[0029] (Program structure) Figure 2 shows an example of the program configuration stored in the memory 111 of Embodiment 1. The memory 111 stores the subject shooting unit 200, the size estimation unit 201, the measurement position input unit 202, and the result output unit 203. Each program contains processing instructions for executing the processes described later and is executed by the CPU 110. For example, when the CPU 110 executes the subject shooting unit 200, the CPU 110 can also function as the subject shooting unit 200. The same applies to other programs. Although the description focuses on the case where each program is executed by the CPU 110 of the terminal 100, this disclosure is not limited to this case. Each program may be included in the shooting device 101, or the processing instructions defined in each program may be executed by the image processing system 1 as a whole.

[0030] The subject shooting unit 200 is a program that performs the function of controlling the shooting device 101 to photograph a subject. The subject shooting unit 200 provides a user interface necessary for the user to control the position and orientation of the shooting device and the camera, and also transmits control signals to the shooting device 101. Alternatively, the subject shooting unit 200 may be configured to automatically control the shooting device 101 without user intervention.

[0031] The size estimation unit 201 is a program that performs the function of estimating the size of a specified location. The size estimation unit 201 receives images of the subject, sensing data, and camera parameters included in the communication data 105, and also receives data regarding the measurement location specified by the user 102 via the measurement location input unit 202, which will be described later. Based on the acquired data, the size estimation unit 201 estimates the size of the specified measurement location.

[0032] The measurement position input unit 202 is a program that performs the function of acquiring information indicating the location of an object whose size is to be estimated, as specified by the user. The measurement position input unit 202 presents an image of the object included in the communication data 105 to the user via the output device 113 and acquires information from the user via the input device 112 regarding the measurement position for which the user wants to measure the size. Examples of information indicating the part to be measured include coordinate information of a bounding box that specifies the whole or part of the object as a rectangle, coordinate information of the start and end points when the part to be measured is specified as a line segment, and multiple combinations of such information.

[0033] The result output unit 203 is a program that performs the function of outputting the size estimation results. The result output unit 203 shows the user the size estimation results estimated by the size estimation unit 201. In addition to the size estimation results, the result output unit 203 may also show the user the level of error included in the estimated size. The level of error will be described later.

[0034] (Image processing procedure) Figure 3 shows an example of the processing procedure of the image processing system 1 for estimating the three-dimensional size of an arbitrary position of a subject, as described in Example 1. This processing procedure is carried out by the CPU 110, which functions as a subject shooting unit 200, a size estimation unit 201, a measurement position input unit 202, and a result output unit 203.

[0035] First, an example of the processing procedure of the subject shooting unit 200 is shown. The subject shooting unit 200 functions as a means of capturing multiple images that include the same subject. The subject shooting unit 200 takes a picture of the subject (step S300), and then takes a picture of the same subject from a different angle (step S301). Referring to Figure 1(b), and relating this to the operation of user 102, user 102 operates the drone 121 while looking at the screen of the tablet 120 in their hand and takes a picture of building 103 with the mounted camera (step S300). Then, similarly, takes another picture of building 103 from a different angle (another angle) (step S301). The subject shooting unit 200 controls the camera mounted on the drone 121 and takes a picture of building 103. Note that this series of images can be a video or still images. At least two or more angles should be used to photograph the subject. Note that two consecutive images should include overlapping areas, and the overlap rate should be, for example, 80% or more. Then, the subject shooting unit 200 inputs the acquired multiple images into the size estimation unit 201 (step S302).

[0036] Next, an example of the processing procedure for the size estimation unit 201 and the measurement position input unit 202 is shown. The size estimation unit 201 is a size estimation means that measures the size of the measurement position (the position where the size of the subject is measured), and the measurement position input unit 202 is a measurement position input means that specifies the measurement position. First, the measurement position input unit 202 specifies the position of the subject for the image input to the size estimation unit 201 (step S303). This process is performed to indicate which subject in the image is the subject to be measured. First, the measurement position input unit 202 displays a UI (user interface) for inputting the position of the subject to be measured, allowing the user to specify the position of the subject appearing in multiple captured images (step S304). A specific example of the UI for specifying the position of the subject will be described later. Using this UI, the user encloses the subject to be measured with a rectangle (bounding box) that circumscribes it, thereby clarifying the subject to be measured. Subsequently, the measurement position input unit 202 transfers the coordinate information of the enclosed rectangle on the image to the size estimation unit 201 (step S305). Note that this rectangle positioning must be performed for all captured images and for the same subject. However, as described later, the user may perform this operation for only one image, and the rectangle may be automatically detected for the other images.

[0037] Next, in steps S306 and S307, the size estimation unit 201 is a correspondence point determination means that extracts feature points from each of the multiple images and compares whether the feature points of each image are corresponding points that indicate the same part of the subject. Specifically, the size estimation unit 201 extracts feature points within a specified rectangle (S306). Feature points are key points on the image where the amount of change in brightness value, color, etc. is large, and are extracted, for example, at the edges and corners of the subject. Feature points can be extracted using general keypoint detection methods, such as SIFT (scale-invariant feature transformation), AKAZE (Accelerated KAZE), BRISK (Binary Robust Invariant Scalable Keypoints), and ORB (Oriented fast and Rotated BRIEF).

[0038] Next, in the multiple images from which feature points have been extracted, the size estimation unit 201 compares the feature points between different images and associates feature points that appear at the same location on the subject, i.e., obtains corresponding points (step S307). In each image taken from multiple viewpoints, the same part of the subject moves to different coordinates. Therefore, in order to find out which coordinates a certain feature point in one image moves to in another image, all feature points in one image and all feature points in another image are compared exhaustively, and the similarity is calculated for each to find the most similar feature points, thereby obtaining information on corresponding points between feature points in two images. Specifically, if the feature point with the highest similarity among all feature points in another image C to feature point B in image A is feature point D, and the feature point with the highest similarity among all feature points in image A to feature point D is feature point B, then these two feature points B and D are considered to point to the same location on the subject, and these two feature points B and D are considered corresponding points. By performing this operation on all images and all feature points, corresponding points can be determined for any two input images.

[0039] Furthermore, if the parallax between each image (the difference in the position of corresponding parts between two images) falls within a certain range, feature points with coordinates that are farther apart will not become corresponding points. Therefore, if the coordinate distance between feature points to be compared is greater than a certain distance, the matching is not performed. This suppresses the generation of incorrect corresponding points. Also, in the processing example described above, only the feature points with the highest similarity become corresponding points, but due to the influence of noise, etc., it is conceivable that even if they are corresponding points, the similarity may not be maximized. Therefore, when calculating the total similarity between a given feature point and all feature points in another image, a threshold N may be set to consider it a corresponding point if it falls within the top N (N is a positive integer) of that similarity. In other words, when the size estimation unit 201 matches feature points contained in multiple images, it matches only feature points that are closer than a predetermined distance in 2D space, calculates the similarity of the feature quantities calculated in the feature point matching, ranks the similarity, and selects a point as a corresponding point if it is higher than the predetermined rank. This reduces errors in correctly detecting points that are actually corresponding points.

[0040] Next, the size estimation unit 201 removes outliers from the corresponding points (step S308). Since RANSAC (Random Sample Consensus) is widely used as a method to remove errors from the obtained corresponding points, the present invention also uses RANSAC.

[0041] Next, in step S309, the size estimation unit 201 is a means for converting the corresponding points into a 3D point cloud, which is a point cloud in 3D space. Specifically, the size estimation unit 201 performs a 3D reconstruction process to calculate the three-dimensional structure based on the obtained information on the corresponding points, the initial values ​​of the camera's intrinsic parameters, and the initial values ​​of the extrinsic parameters (step S309). As a specific calculation example, the case based on the generally known SfM (Structure from Motion) method will be explained. First, initial values ​​for the camera parameters and 3D coordinates are given. As a method for obtaining the initial values ​​of the camera parameters, the generally known method using the basic matrix and fundamental matrix can be used. Next, the 3D coordinates are obtained by triangulation. Then, bundle adjustment is performed using these values ​​as initial values. Bundle adjustment is also a generally well-known method, and iteratively optimizes the camera parameters and the coordinate values ​​of each point in the 3D point cloud to minimize the reprojection error. Through this process, a 3D point cloud representing the three-dimensional coordinates of the feature points of the subject is finally generated. The above explanation shows an example of the 3D reconstruction process, and this disclosure is not limited to this case.

[0042] Next, the size estimation unit 201 calculates the size per pixel (step S310). The size per pixel can be determined from information such as focal length and pixel size included in the optimized camera parameters. In other words, parameters used when converting coordinate systems in 3D space and 2D space to actual size are calculated. In the following explanation, there are cases where the coordinate system in 2D space is displayed to the user by the output device 113. In this case, the 2D space may also be referred to as a 2D image.

[0043] Next, the size estimation unit 201 acquires the measurement position on the 2D image for which the size is to be estimated (step S311). The measurement position is detected by the measurement position input unit 202. The measurement position input unit 202 specifies the measurement position using coordinates in 2D space and projects the coordinates in 2D space onto 3D space. Specifically, the measurement position input unit 202 displays the image (step S312) and displays the measurement position input UI (step S313). Referring to Figure 1(b), and relating this to the operation of user 102, user 102 specifies the position to be measured from the displayed image of the captured image using a line segment on the 2D image (step S313). The specification on the 2D image is done by touching the screen of the tablet 120, or by mouse operation if a mouse is provided as the input device 112 (step S313). Finally, the measurement position input unit 202 transfers the line segment information of the start and end points (hereinafter also referred to as "start point / end point information") to the size estimation unit 201 (step S314). Note that the part to be measured can be specified not only with a single line segment, but also with a line made up of multiple straight lines, such as a polyline. Furthermore, it is not limited to straight lines; curves may also be specified, as well as closed curves. The point for projecting coordinates in 2D space into 3D space will be described later.

[0044] Subsequently, the size estimation unit 201 calculates the length in three-dimensional space for the two-dimensionally specified line segment from the length between the start and end points calculated from the acquired line segment information and the result of the size per pixel calculated above (step S315). Next, the size estimation unit 201 performs measurement error estimation processing to predict the degree of error included in the calculated size (step S316). The specific methods for calculating the size of the measurement position and estimating the level of error will be described later.

[0045] Finally, the size estimation unit 201 outputs the calculated size measurement result and the error level (step S317). This process is performed by the result output unit 203. Specifically, the result output unit 203 displays the size measurement result at the specified location on the screen (step S318), followed by the display of the error level (step S319). An example of a specific display screen will be described later.

[0046] (User Interface) Figure 4 shows an example of the input UI screen for the position of the object to be measured in step S304 of Figure 3. Figure 4 shows the display on the tablet 120's display unit (output device 113). Images 400 and 401 taken by the drone capture the building 103, which is the object to be measured, and the house 404, which is not the object to be measured, with a slight parallax. The user checks images 400 and 401 displayed on the tablet 120's screen and surrounds the building 103 with bounding boxes 402 and 403. This makes it possible to clearly indicate which of the multiple subjects in the image is the object to be measured.

[0047] Here, images 400 and 401 are referred to as stereo pair images. A stereo pair image is a collection of multiple images of the same subject taken from different angles. In this disclosure, the case where a stereo pair image is a set of two images taken from two angles is described, but this disclosure is not limited to this case. The number of angles and images may be three or more. The term stereo pair image is used for convenience and does not indicate that the number of images included in the set of images is limited to two. In this embodiment, the case where there are two stereo pair images is illustrated, but the position specification of the subject to be measured as described above must be performed for all of the multiple images taken. Therefore, if a third image or later is taken, for example, the images may be scrolled horizontally to display the third image or later on the screen, and the process of enclosing the subject with a bounding box may be performed in the same way.

[0048] However, if the user has to specify the target subject for every image, the workload may increase and the convenience may decrease. Therefore, it is possible to have the user draw a frame around the target subject in only one of the obtained images, and for the other images, the same subject as the framed subject can be automatically detected using general image similarity object detection technology, and the positions of subjects in the other images can be automatically detected. Alternatively, instead of having the user draw frames around multiple subjects in an image, general object detection processing can be applied to automatically detect the frames of subjects, and the user can choose which subject within the frame they want to focus on. This simplifies the effort required of the user to draw frames.

[0049] Furthermore, if there is only one subject and it is in the center of the image, the image itself can be considered to capture the position of the subject, and the process of enclosing the subject with a frame can be omitted. Instead, the edges of the image can be used as the bounding box for the subject.

[0050] Figure 5 shows an example of a UI for inputting the position of the part of the subject to be measured in steps S312 and S313 of Figure 3. Figure 5 shows the display on the tablet 120's display unit (output device 113). One image 400 is displayed from among multiple images showing the target subject, building 103. The user inputs the position of the subject to be measured from this screen. Here, as an example of measuring the height of building 103, a size-specified line segment 500 is specified. The size-specified line segment 500 is a line segment formed by specified coordinates in two-dimensional space. First, the user specifies the starting point 501 of the size-specified line segment 500 on the tablet 120 screen using an input means such as a stylus or mouse. Next, the user similarly specifies the ending point 502 for which they want to measure the length. This sets the size-specified line segment 500.

[0051] The size estimation unit 201 estimates the three-dimensional length of the portion specified by the size-specified line segment 500. However, as described later, the three-dimensional information of the subject may not be reproduced accurately, so it is not always possible to obtain the exact size of this specified line segment. Therefore, a correction process is performed as described later.

[0052] (Size estimation process) Figure 6 is an explanatory diagram illustrating the case where size estimation is performed using a 3D point cloud and a size-specified line segment in steps S314 and S315 of Figure 3. In the 3D reconstruction process in step S309, if the number of stereo pair images is small, the parallax between consecutively captured images is large, or the brightness change of the subject is small, the number of feature points and corresponding points tends to decrease, and it may not be possible to obtain a dense 3D point cloud of the subject. If a dense 3D point cloud is not obtained, it is necessary to perform size estimation with missing information, so even in that case, it is necessary to perform correction processing to reduce the error as much as possible. A dense 3D point cloud can be defined as, for example, a state where the original shape of the subject can be visually recognized when the 3D point cloud is displayed. The criterion for whether or not it is dense is not limited to this case, and can be set as appropriate according to the required estimation accuracy, system configuration, and other conditions.

[0053] Figure 6(a) shows the triangular pyramid 600 as seen in the 2D image of image 400, the ideal 3D point cloud 601 of the triangular pyramid, and the 3D point cloud 602 reconstructed by step S309 described above, when the subject is a three-dimensional triangular pyramid. Figure 6(a)(1) shows the 2D image, and Figure 6(a)(2) shows the 3D space corresponding to the image in Figure 6(a)(1). The 3D coordinates are defined as x, y, and z in the direction of the figure, and image 400 is a 2D plane defined by the x and y directions. For convenience, the ideal 3D point cloud 601 is represented by a solid line, but it is assumed that these are points with multiple 3D coordinates. Also, for ease of understanding, points in the reconstructed 3D point cloud 602 that are located on the sides of the triangular pyramid are represented by small circles. Furthermore, points placed on one side of the triangular pyramid are labeled 6021 to 6026. Figure 6(a)(2) shows that the points constituting the reconstructed 3D point cloud 602 are sparsely reconstructed. In other words, if an ideal reconstruction process were performed, points would be placed without gaps on the sides of the triangular pyramid represented by the ideal 3D point cloud 601. However, in the 3D point cloud 602 reconstructed by the realistic reconstruction process shown in Figure 6, points are placed with gaps on the sides of the triangular pyramid of the ideal 3D point cloud 601, and no points are placed at the vertices of the triangular pyramid.

[0054] In this embodiment, the size in 3D space is basically estimated by a size-specified line segment set on a 2D image. The general procedure for estimation is as follows: First, all groups included in the 3D point cloud are projected onto the 2D image on which the size-specified line segment is set (hereinafter referred to as the projected point cloud). Next, one point closest to each endpoint of the size-specified line segment is selected from the projected point cloud. Then, the two points selected from the projected point cloud are returned to 3D space, treated as a 3D line segment, and their length is calculated, which is obtained as the 3D length. However, if there are no points in the projected point cloud near the endpoint of the size-specified line segment, accurate size estimation becomes impossible, so it becomes necessary to perform a correction process that does not reduce accuracy as much as possible. Note that the vicinity of the endpoint can be defined as, for example, a range of up to a predetermined number of pixels from the endpoint. For example, the predetermined number of pixels can be 20 pixels.

[0055] The size-specified line segment 500 is based on the start and end point information specified, for example, in step S314 of Figure 3. The start and end points of the size-specified line segment 500 are specified near the vertices of the triangular pyramid 600, and the user intends to measure the three-dimensional length of the edge connecting the upper vertex of the triangular pyramid 600 (the point with the largest Y coordinate) to the left vertex (the point with the smallest X coordinate). Since the size-specified line segment 500 is specified in coordinates on a two-dimensional plane, it is necessary to associate the coordinates on the two-dimensional plane with the ideal three-dimensional point cloud 601. Therefore, in this invention, each point of the reconstructed three-dimensional point cloud 602 is projected onto the xy plane as described above, and a projected point cloud is obtained that is projected onto the two-dimensional plane. This operation is performed by setting the z=0 of the three-dimensional coordinate of each point in the reconstructed three-dimensional point cloud 602. In Figure 6, for ease of understanding, two points, 603 and 604, are shown as projected point clouds. Point 603 is a projection of point 6021, which is included in the reconstructed 3D point cloud 602, and point 604 is a projection of point 6027, which is included in the reconstructed 3D point cloud 602.

[0056] The measurement position input unit 202 then detects points located near specified coordinates in 2D space from the projected 3D point cloud, and the size estimation unit 201 estimates the size from the detected points in 2D space. The size estimation unit 201 detects the nearest point to the endpoint of the line segment (size-specified line segment 500) in 2D space from the projected 3D point cloud. Specifically, all points of the reconstructed 3D point cloud 602 are projected onto a 2D plane in this manner, and the point cloud with the closest 2D spatial distance between the projected point cloud and the size-specified line segment 500 is selected. Here, the point projected onto the 2D plane closest to the start point 501 is shown as 603, and the point closest to the end point 502 is shown as 604. From this figure, it can be seen that although the projected point cloud exists near the endpoints (start point 501 and end point 502) of the size-specified line segment 500, it does not exist in a location that perfectly coincides with the endpoints. Note that the vicinity of the endpoint can be defined as, for example, a range of less than or equal to a predetermined number of pixels from the endpoint. For example, the predetermined number of pixels can be set to 100 pixels.

[0057] (Correction process) Figure 6(b) is an explanatory diagram showing the correction process after selecting a group of projected points located near a specified size line segment. Figure 6(b)(1) is a re-illustration of the points on the 2D image shown in Figure 6(a). As described above, after obtaining points 603 and 604, which are the projection points closest to the specified size line segment 500, points 603 and 604 are returned to (projected) in 3D space to obtain the measured line segment 605 in 3D space. By treating the measured line segment 605 as a three-dimensional line segment and measuring its length, the line segment set in 2D space is converted into a 3D line segment, and its length C can be obtained. (Note that the measured line segment 605 is the measurement position in 3D space, and the size of the measured line segment 605 can be measured by the size estimation unit 201 in step S311. Furthermore, the process of returning (projecting) points 603 and 604 on the 2D image to 3D space can be performed, for example, by the measurement position input unit 202 in step S313.)

[0058] However, in the example shown in Figure 6 in particular, the points at the vertices of the triangular pyramid representing the ideal 3D point cloud 601 are not reconstructed, and there is a distance E between the starting point 501 and point 603 in the projected point cloud, and a distance F between the ending point 502 and point 604 in the projected point cloud. If we let A be the length of the size-specified line segment 500, and B be the length of the line segment between points 603 and 604 in the projected point cloud, then due to the effects of the errors in distances E and F, length B is shorter. Similarly, the measured line segment 605 in 3D space is represented by length C, which is shorter than the side length of the triangular pyramid representing the actual ideal 3D point cloud 601.

[0059] In this embodiment, the size estimation unit 201 corrects the size estimation result, which represents the estimated size, based on the ratio of the length of the line segment (length of the size-specified line segment 500) to the length of the line segment formed by the detected points (length of the measured line segment 605). Specifically, the ratio of the length A of the size-specified line segment 500 and the length B of the measured line segment 605 in 2D space is determined, and the length C of the measured line segment 605, determined in 3D space, is multiplied by the coefficient A / B, and the result is considered the final size estimation result. When the converted measured line segment 606, which is converted by multiplying the measured line segment 605 of length C by the coefficient A / B, is shown, the length D of the measured line segment 606 is expressed as C × A / B, and as shown in Figure 6, it is converted to something close to the side length of a triangular pyramid representing an ideal 3D point cloud 601. This correction makes it possible to improve the accuracy of size estimation even when the 3D point cloud has not been completely reconstructed.

[0060] In the example above, one point was detected from the projection point group near each of the two endpoints of the size-specified line segment 500 to form the measured line segment 605 and estimate the length of the line segment in 3D space. However, this method lacks robustness because it forms the measured line segment from information of at most two points. Therefore, instead of just the nearest projection point group, points up to the Nth nearest point group can be obtained, and the line segment can be determined from their centroids. This allows for a more accurate estimation of the line segment.

[0061] As described above, the present invention allows the user to easily select the position for measuring the three-dimensional size of an object. By specifying the position to be measured on a two-dimensional image, it becomes possible to measure the three-dimensional size, enabling accurate size estimation without the need for complex operations such as 3D viewers.

[0062] (Prediction of size estimation error) Figure 7 shows an example of a processing flow for predicting the level of error during size estimation. As mentioned above, if the 3D point cloud cannot be detected densely, the error in size estimation is expected to be large. In particular, if the points included in the projected point cloud, which is a 3D point cloud projected onto 2D space as mentioned above, are not in the vicinity of the size-specified line segment specified by the user, the error is expected to be large even after applying the correction process described above. Furthermore, if the total number of points included in the projected point cloud is small, the error is also expected to be large.

[0063] Therefore, evaluation criteria are set to assess the tendency for errors to increase, quantitative values ​​of the evaluation criteria are calculated, and their correlation with the errors that may be included during measurement (hereinafter also referred to as "measurement errors") is examined. The level of measurement error is then calculated from the relationship between the quantitative values ​​and the measurement errors.

[0064] Error calculation can be performed, for example, by the size estimation unit 201. The size estimation unit 201 is a means for estimating the error of the size estimation result, and estimates the error of the size estimation result based on the coordinates specified in 2D space, the distance in 2D space to the nearest point, and the density of the 3D point cloud. Specifically, first, the error rate between the length of the size-specified line segment and the measured line segment is calculated as the evaluation value V1 (step S701). This error rate is the ratio of the difference between length A and length B shown in Figure 6(b), and can be expressed as ABS(AB) / A. Next, the sum of the Euclidean distances between the starting point of the size-specified line segment and the nearest point in the projected point cloud, and between the ending point of the size-specified line segment and the nearest point in the projected point cloud is obtained as the evaluation value V2 (step S702). This is the sum of distances E and F shown in Figure 6(b). Next, as evaluation value V3, the ratio of the number of points in the projection point cloud located near the specified size line segment to the length of the specified size line segment (hereinafter also referred to as "specified line segment length") is calculated (step S703). From experience, it has been found that the higher the density of the 3D point cloud, the more accurately the 3D structure is reconstructed. Therefore, the number of points in the point cloud per unit length is calculated, and it is expected that the larger this value, the smaller the measurement error.

[0065] (When the accuracy of 3D point cloud reconstruction is low) Here, with reference to Figure 8, we will explain the case where a low-density point cloud is reconstructed by 3D point cloud reconstruction. Figure 8 is a schematic diagram showing the projected point cloud distributed in the vicinity of the specified size line segment 500. The low-density 3D point cloud 612 shows the point cloud reconstructed by step S309. The low-density 3D point cloud 612 is reconstructed as a point cloud with lower density compared to the reconstructed 3D point cloud 602 in Figure 6(a)(2). Specifically, three points in the projected point cloud (point 6131 is a projection of point 6121 included in the 3D point cloud 612, point 6132 is a projection of point 6122 included in the 3D point cloud, and point 6133 is a projection of point 6123 included in the 3D point cloud 612) are distributed in the vicinity of the size-specified line segment 500. However, in Figure 8(a), the density of points in the upper part of the triangular pyramid of the 3D point cloud 612 is relatively lower compared to the points in the lower part (points on the edges parallel to the X direction). In other words, it can be seen that reconstruction was relatively successful for the lower edges, but 3D reconstruction was not successful for the other edges. Thus, it can be inferred that the error in size estimation accuracy varies depending on the location of the size-specified line segment 500. The evaluation value V3 mentioned above is calculated by counting the number of projected points within a certain distance from the line segment specified as size 500 and dividing by the length of the line segment specified as size 500. A larger value indicates a denser point cloud with many points, resulting in a smaller error in size estimation accuracy. Conversely, a smaller value tends to result in a larger error.

[0066] Returning to the explanation of the processing flow in Figure 7, the final score is calculated by linearly combining the obtained scores of V1, V2, and V3 (step S704). If the coupling coefficients of the linear combination are α, β, and γ, then the final score = α × V1 + β × V2 + γ × V3. At this time, as one example of how to determine α, β, and γ, a method can be applied in which a large number of size estimation results and their correct values ​​are obtained in advance using evaluation data to calculate size estimation error data, and the coefficients are optimized so as to maximize the correlation between the final score and the size estimation error rate. As a method to solve this, mathematical programming methods such as the GRG (Generalized Reduced Gradient method) nonlinear method can be used.

[0067] (Relationship between size estimation error rate and evaluation value) Figure 9 shows an example of the distribution of the error rates between evaluation values ​​V1, V2, V3, and the final score and the size estimation error rate. The error rate is a numerical representation of the level of error. Figure 9(a) shows the relationship between evaluation value V1 and the size estimation error rate, Figure 9(b) shows the relationship between evaluation value V2 and the size estimation error rate, Figure 9(c) shows the relationship between evaluation value V3 and the size estimation error rate, and Figure 9(d) shows the relationship between the final score and the size estimation error rate. For example, as shown in Figures 9(a) and 9(b), in the case of evaluation values ​​V1 and V2, the larger the evaluation value, the larger the size estimation error rate, while as shown in Figure 9(c), the smaller the evaluation value V3, the smaller the size estimation error rate. An example of a linear combination of these evaluation values ​​is shown in Figure 9(d), a graph representing the relationship between the final score and the size estimation error rate, which shows that the correlation is increasing.

[0068] Returning to the explanation of the processing flow in Figure 7, the next step is to obtain a result determination of the error level relative to the final score (step S705). Here, the result output unit 203 is a means for outputting the size estimation result and the error estimation result (error level result) that shows the error of the size estimation result, and outputs the error estimation result. Specifically, as shown in the graph in Figure 9(d), for example, if the final score falls within the range of "minimal error", it can be seen that the error rate of the size estimation will not exceed 10%. Similarly, if the final score falls within the range of "small error", the error rate will generally be around 20% or less. If the final score falls within the range of "medium error", the error rate will be 30% or less. If the final score is higher than that, it can be predicted that the error rate will exceed 30%. In this way, by determining which range the final score falls into, an estimate of the error can be grasped. Finally, this error level (error rate) is output to the screen to show the user (step S706).

[0069] (UI for size estimation results and error estimation results) Figure 10 shows an example of displaying the size of a measured object and the level of error. Figure 10 shows how it appears on the display unit (output device 113) of the tablet 120. In the example of Figure 10(a), a size specification line segment 500 is set at the position indicating the height of the building 103, and the estimated length (size) and final score are calculated according to the above-described embodiment 1, and an output message 1001 including the level of error is output. In the example of Figure 10(a), the height of the building 103 was estimated to be 15m, but from the calculation of the final score mentioned above, it can be seen that the level of error is about 20%, so a warning to that effect is also displayed. Here, a message such as "Estimated length: 15m (Error range: 12m~18m) *Measurement error may exceed 20%. Read the result as is or remeasure at a different location" is displayed to alert the user that the measurement error is about 20%. Based on output message 1001, the user can decide whether to read the estimated length as is and use it for maintenance work, change the measurement location and recalculate, or take new images.

[0070] Furthermore, the size estimation unit 201 estimates a measurement error based on a 3D point cloud, and the result output unit 203 can also display the measurement error on a 2D image. Specifically, as shown in Figure 10(b), before the user specifies a size-specified line segment, the error level can be displayed as a heatmap based on the density information of the point cloud shown in Figure 8, for example, allowing the user to visually understand which areas are prone to large errors during measurement. In Figure 10(b), in image 400, a display according to the error rate indicated by a color bar is superimposed on the building 103, which is the subject. This makes it easy to identify measurement locations with small errors. Alternatively, the same effect can be obtained by superimposing the aforementioned projected point cloud onto this image 400. Although a heatmap is described as an example of an error map for displaying errors, this disclosure is not limited to this case. Any method of visualizing and displaying the distribution trend of errors when specifying a size-specified line segment can be applied to this disclosure.

[0071] (Effects / Actions) As described above, this disclosure allows users to easily select a position for measuring the three-dimensional size of an object, and to accurately estimate that three-dimensional size. Furthermore, the degree of error can be visually grasped, and users can select acceptable estimation results according to their purpose.

[0072] For example, in prior art, when using a method to reconstruct a 3D point cloud from a 2D disparity image, it is possible to specify the location for size estimation from the 2D image. However, because the reconstruction of the 3D point cloud is incomplete, the size of the specified location may not be accurately determined. Furthermore, if the reconstruction of the 3D point cloud is poor and there is variation in the density of the points, the degree of measurement error will vary depending on the specified area, making it difficult for the user to determine the reliability of the size estimation.

[0073] In response to the problems of the prior art described above, this disclosure presents a process for presenting the estimated error to the user in advance when estimating the size of an object, and a correction process for the 3D point cloud when performing size estimation. In this disclosure, although a complete correlation with the error rate cannot be grasped, by calculating a final score that quantifies the factors of the error, it is possible to get an idea of ​​the error in size estimation, and the user can make decisions such as using the measurement results again in maintenance work or changing the measurement location and re-measuring, thereby realizing more reliable maintenance work.

[0074] In the above example, we used a three-dimensional length specified by a line segment as the target for size estimation. However, even if we specify the length of a curve, for example, we can similarly obtain a highly accurate curve length by acquiring a cloud of points near the start and end points and converting the final result based on the ratio of the lengths.

[0075] As described above, the present invention is not limited to cases of measuring the size of buildings or cracks for maintenance and inspection of structures, but can also be applied to cases of measuring the size of an object using a general camera and sensor that includes the aforementioned problems. For example, it goes without saying that it can be applied to a function that estimates the size of an object from several images of the object taken with a smartphone. This function can be used, for example, in a system that automatically determines the purchase price of a used car in online car sales, to measure the length of scratches and the size of dents on a photographed car. In this case, for example, if the car owner takes a picture of their car and sends the image and sensing data to the car buying company's server, the car buying company can measure the size of the object on the server and confirm the size of the scratches and dents that are captured. This allows for an objective assessment of the car's condition and enables an accurate estimate of the purchase price of the photographed car. In other words, this technology can be used to realize an accurate online car purchase price estimation service.

[0076] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. Furthermore, for example, the embodiments described above are detailed explanations of the configuration in order to clearly illustrate the present invention, and are not necessarily limited to those having all the configurations described. In addition, some of the configurations in each embodiment can be added to, deleted from, or replaced with other configurations.

[0077] Furthermore, some or all of the above-described configurations, functions, processing units, and processing means may be implemented in hardware, for example, by designing them as integrated circuits. The present invention can also be implemented by software program code that realizes the functions of the embodiment. In this case, a storage medium on which the program code is recorded is provided to a computer, and the processor of that computer reads the program code stored in the storage medium. In this case, the program code read from the storage medium itself realizes the functions of the embodiment described above, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media used to supply such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs (Solid State Drives), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, and ROMs.

[0078] Furthermore, the program code that implements the functions described in this embodiment can be implemented in a wide range of programming or scripting languages, such as assembler, C / C++, Perl, Shell, PHP, Python, and Java (registered trademark).

[0079] Furthermore, the program code for the software that implements the functions of the embodiment may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the computer's processor may read and execute the program code stored in the storage means or storage medium.

[0080] In the above-described embodiment, the control lines and information lines shown are those deemed necessary for explanation and do not necessarily represent all control lines and information lines in the actual product. All components may be interconnected.

[0081] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications are possible without departing from the spirit of the present invention.

[0082] The following describes, but is not limited to, embodiments that may constitute the present invention. (Aspect 1) A means of taking multiple images that include the same subject, A means for determining corresponding points that extracts feature points from each of the multiple images and compares whether the feature points of each image are corresponding points that represent the same part of the subject, A means for converting the aforementioned corresponding points into a 3D point cloud, which is a point cloud in 3D space, A measurement position input means for specifying a measurement position for measuring the size of the subject, A size estimation means for estimating the size of the measurement location, Equipped with, The measurement position input means specifies the measurement position using coordinates in a two-dimensional space, and projects the coordinates in the two-dimensional space into the three-dimensional space. The size estimation means measures the size of the measurement position in the three-dimensional space. An image processing system characterized by the following. (Aspect 2) In embodiment 1, the measurement position input means is Means for projecting the three-dimensional point cloud onto the two-dimensional space, A means for detecting a point located in the vicinity of a specified coordinate in the two-dimensional space from the projected three-dimensional point cloud, Equipped with, The size estimation means estimates the size from the detected point in the two-dimensional space. An image processing system characterized by the following. (Aspect 3) In embodiment 1 or embodiment 2, the size estimation means is A line segment is formed in the aforementioned two-dimensional space by the specified coordinates, From the projected three-dimensional point cloud, the nearest neighbor point to the endpoint of the line segment is detected in the two-dimensional space. The size estimation result, which indicates the estimated size, is corrected based on the ratio between the length of the line segment and the length of the line segment formed by the detected point. An image processing system characterized by the following. (Aspect 4) In any one of the embodiments 1 to 3, Means for estimating the error in the size estimation result, Output means for the size estimation result and error estimation result showing the error of the size estimation result, Equipped with, The means for estimating the error in the size estimation result is: The distance in two-dimensional space between a specified coordinate in the two-dimensional space and the nearest point cloud, and the density of the three-dimensional point cloud. Based on this, the error in the size estimation result is estimated, The output means outputs the error estimation result, An image processing system characterized by the following. (Appendix 5) In any one of embodiments 1 to 4, The means for estimating the error in the size estimation result is: Based on the three-dimensional point cloud, the measurement error indicating the measurement error is estimated. A means for displaying the estimated measurement error on a two-dimensional image, An image processing system characterized by having the following features. (Aspect 6) In any one of embodiments 1 to 5, The means for determining the corresponding point is, In matching feature points contained in the aforementioned multiple images, only feature points that are closer than a predetermined distance in two-dimensional space are matched. The similarity of the features calculated in the matching of the aforementioned feature points is calculated, the similarity is ranked, and if it is higher than a predetermined rank, it is selected as the corresponding point. An image processing system characterized by the following. (Aspect 7) The steps include taking multiple images containing the same subject using a means of taking images, A correspondence point determination step involves extracting feature points from each of the multiple images using a correspondence point determination means, and comparing whether the extracted feature points of each image are corresponding points that represent the same part of the subject, The steps include: converting the corresponding points into a 3D point cloud, which is a point cloud in 3D space, using a means for converting them into a 3D point cloud; A measurement position input step involves specifying a measurement position for measuring the size of the subject using a measurement position input means, A size estimation step in which the size of the measurement position is estimated by a size estimation means, Includes, In the measurement position input step, the measurement position is specified by coordinates in a two-dimensional space, and the coordinates in the two-dimensional space are projected onto the three-dimensional space. In the size estimation step, the size of the measurement position in the three-dimensional space is measured. An image processing method characterized by the following. [Explanation of Symbols]

[0083] 100 devices 101 Imaging device 102 users 103 Building 105 Communication data 110 CPU 111 memory 112 Input device 113 Output device 114, 117 Network Interfaces 115 Camera 116 Sensors 120 tablets 121 Drones 200 Subject photography department 201 Size Estimation Section 202 Measurement position input section 203 Result Output Section Images 400, 401 402, 403 Bounding Boxes 404 Housing 500 specified size line segments 501 Starting point of line segment 502 End point of line segment 600 Triangular pyramid subjects in 2D images 601 Ideal 3D point cloud 602 Reconstructed 3D point cloud 603, 604 Three-dimensional point cloud projected onto a two-dimensional plane 605 Measurement line segments in 3D space 606 Measured line segment after conversion in 3D space 612 Low-density 3D point clouds 1001 Output message

Claims

1. A means of taking multiple images that include the same subject, A means for determining corresponding points that extracts feature points from each of the multiple images and compares whether the feature points of each image are corresponding points that represent the same part of the subject, Means for converting the aforementioned corresponding points into a three-dimensional point cloud, which is a point cloud in three-dimensional space, A measurement position input means for specifying a measurement position for measuring the size of the subject, A size estimation means for estimating the size of the measurement location, Equipped with, The measurement position input means specifies the measurement position using coordinates in two-dimensional space, and projects the coordinates in two-dimensional space into three-dimensional space. The size estimation means measures the size of the measurement position in the three-dimensional space. An image processing system characterized by the following.

2. In claim 1, the measurement position input means is Means for projecting the three-dimensional point cloud onto the two-dimensional space, A means for detecting a point located in the vicinity of a specified coordinate in the two-dimensional space from the projected three-dimensional point cloud, Equipped with, The size estimation means estimates the size from the detected point in the two-dimensional space. An image processing system characterized by the following.

3. In claim 2, the size estimation means is A line segment is formed in the aforementioned two-dimensional space by the specified coordinates, From the projected three-dimensional point cloud, the nearest neighbor point to the endpoint of the line segment is detected in the two-dimensional space. The size estimation result, which indicates the estimated size, is corrected based on the ratio between the length of the line segment and the length of the line segment formed by the detected point. An image processing system characterized by the following.

4. In claim 3, Means for estimating the error in the size estimation result, Output means for the size estimation result and error estimation result showing the error of the size estimation result, Equipped with, The means for estimating the error in the size estimation result is: The coordinates specified in the two-dimensional space, the distance in the two-dimensional space between them and the nearest point cloud, and the density of the three-dimensional point cloud. Based on this, the error in the size estimation result is estimated, The output means outputs the error estimation result, An image processing system characterized by the following.

5. In claim 4, The means for estimating the error in the size estimation result is: Based on the three-dimensional point cloud, the measurement error indicating the measurement error is estimated. A means for displaying the estimated measurement error on a two-dimensional image, An image processing system characterized by having the following features.

6. In claim 1, The means for determining the corresponding point is, In matching feature points contained in the aforementioned multiple images, only feature points that are closer than a predetermined distance in two-dimensional space are matched. The similarity of the features calculated in the matching of the aforementioned feature points is calculated, the similarity is ranked, and if it is higher than a predetermined rank, it is selected as the corresponding point. An image processing system characterized by the following.

7. The steps include taking multiple images containing the same subject using a means of taking images, A correspondence point determination step involves extracting feature points from each of the multiple images using a correspondence point determination means, and comparing whether the extracted feature points of each image are corresponding points that represent the same part of the subject, The steps include converting the corresponding points into a three-dimensional point cloud, which is a point cloud in three-dimensional space, using means for converting them into a three-dimensional point cloud, A measurement position input step involves specifying a measurement position for measuring the size of the subject using a measurement position input means, A size estimation step in which the size of the measurement position is estimated by a size estimation means, Includes, In the measurement position input step, the measurement position is specified by coordinates in a two-dimensional space, and the coordinates in the two-dimensional space are projected onto the three-dimensional space. In the size estimation step, the size of the measurement position in the three-dimensional space is measured. An image processing method characterized by the following.

Images