Stone material matching device and program

Abstract

The present invention provides a stone material matching device, a stone material matching method, and a program that can match stone material images while further reducing the workload on the operator. [Solution] The stone material matching device acquires the outlines of the stones before the collapse included in images of the stone wall taken before the collapse, and acquires the outlines of the stones after the collapse included in images taken after the collapse of the stone wall. For each image of the stones before the collapse and the images of the stones after the collapse, it extracts feature quantities that include at least the relationship between the relative angle with respect to a reference direction and the distance, which is determined based on the distance from the average position in each direction to the outline, centered on the average position of the outline or stone material area. The feature quantities of the stones before the collapse and the feature quantities of the stones after the collapse are compared to match the stones before the collapse and the stones after the collapse.

Description

Technical Field

[0001] This invention relates to a stone matching device and a program.

Background Art

[0002] Old buildings such as castles and fortifications, and their remains often include stone walls. Stone walls may collapse due to natural disasters such as large earthquakes.

[0003] When a large amount of stones collapse disorderly, there is a problem that it is very time-consuming to identify each of the collapsed stones while referring to the photos of the stone wall that was held and return them to their original positions. In the stone wall management system described in Patent Document 1, feature points such as corners and edges of stones are extracted from the image of the stone wall taken by an optical camera and the images of each stone after collapse, and the information of the stone images extracted in the order of the highest similarity of the feature points is output as a matching result.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the conventional technology, there is a problem that it may be difficult to appropriately associate due to damage, scratches, dirt, etc. at the time of the collapse of the stone wall, and it often requires a lot of labor for the operator.

[0006] An object of this disclosure is to provide a stone matching device and a program that can match stone images while further reducing the labor of the operator.

Means for Solving the Problems

[0007] To achieve the above objective, the stone material matching device of this disclosure is characterized by comprising: a first region acquisition means for acquiring stone material regions of multiple pre-collapse stone materials included in an image of the stone wall before collapse; a second region acquisition means for acquiring stone material regions of post-collapse stone materials included in an image taken after the collapse of the stone wall; a feature quantity extraction means for determining, for both the pre-collapse stone material and the post-collapse stone material, which of the predetermined circumscribing rectangle and inscribing rectangle of the stone material region has the smallest area difference with the stone material region, and extracting a feature quantity that includes at least an index relating to the degree of irregularity corresponding to the smallest area difference; and a matching means for comparing the feature quantity of the pre-collapse stone material and the feature quantity of the post-collapse stone material to match the pre-collapse stone material and the post-collapse stone material. [Effects of the Invention]

[0008] Following this disclosure has the effect of allowing stone material images to be matched while further reducing the workload on the worker. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing the system configuration of the stone material matching system of this embodiment. [Figure 2] This flowchart shows the control procedure for the stone material data generation control process. [Figure 3] This is a diagram showing examples of feature quantities related to the shape of stone that can be calculated from images of stone materials. [Figure 4] This is a diagram illustrating the contour radius. [Figure 5] This is a diagram illustrating the minimum irregularity index. [Figure 6] This flowchart shows the control procedure for the feature extraction process. [Figure 7] This flowchart shows the control procedure for the stone material matching and control process. [Figure 8] This figure shows an example of a display screen related to matching. [Modes for carrying out the invention]

[0010] Hereinafter, embodiments of the present invention will be described based on the drawings. FIG. 1 is a block diagram showing the system configuration of the stone material matching system 100 of the present embodiment. The stone material matching system 100 includes a processing device 1 as a stone material matching device and a storage device 2 that stores stone material images.

[0011] The processing device 1 may be an ordinary personal computer (PC, computer), and includes a CPU 11 (Central Processing Unit), a RAM 12 (Random Access Memory), a storage unit 13, a communication unit 14, a display unit 15 (display means), an operation reception unit 16, and the like.

[0012] The CPU 11 performs arithmetic processing and comprehensively controls the operation of the processing device 1. The CPU 11 may be a single processor, or a plurality of processors may operate in parallel or independently according to applications and the like.

[0013] The RAM 12 provides a working memory space for the CPU 11 and stores temporary data. The RAM 12 may be, for example, but not particularly limited to, a DRAM or the like. When the CPU 11 has a plurality of processors, each may have a corresponding RAM, or a RAM that is commonly used by the plurality of processors may be used.

[0014] The storage unit 13 is an auxiliary storage device that stores a program 131 related to the data generation control process of the stone material and the stone material matching control process of the stone material, setting data, and the like. The storage unit 13 may be, for example, a non-volatile memory such as a flash memory or an HDD (Hard Disk Drive). The storage unit 13 may be a configuration externally attached to the processing device 1 (a configuration outside the processing device 1), or may be a network drive or a cloud server that exists on the network and can be accessed. The setting data includes, for example, screen settings and display order settings related to the display of the matching result on the display screen.

[0015] The communication unit 14 controls the transmission and reception (communication) of data with external devices according to a predetermined communication standard. The predetermined communication standard includes, for example, TCP / IP related to a LAN (Local Area Network). Further, the communication unit 14 may have connection terminals such as USB (Universal Serial Bus) and be capable of controlling one-to-one communication with a connected device via USB.

[0016] The display unit 15 has a digital display screen and performs display on the digital display screen based on the control of the CPU 11. The digital display screen is, for example, a liquid crystal display screen (LCD), although it is not particularly limited. The operation reception unit 16 includes a pointing device such as a mouse and a keyboard, etc., receives an input operation, and outputs an operation signal corresponding to the received content to the CPU 11. Note that the display unit 15 and the operation reception unit 16 may be peripheral devices (components outside the processing device 1) connected to the connection terminals of the communication unit 14.

[0017] The storage device 2 stores stone images, as well as identification information and characteristic information associated with the stone images. The storage device 2 may be a dedicated database device such as a network drive or a cloud server, or a PC with a built-in or externally attached auxiliary storage device (HDD) of an appropriate capacity, etc. The stone images include a pre-collapse stone wall image photographed by an optical camera before the collapse and images (pre-collapse stone material images) classified into each stone material (pre-collapse stone material) included in the stone wall image, as well as post-collapse stone material images of the stone material after the collapse of the stone wall photographed by the optical camera. The identification information is, for example, a uniquely determined identification number, symbol, etc. The characteristic information will be described later.

[0018] Next, the collation of stone materials will be described. When a stone wall collapses, individual stones are scattered, shifting from their original positions and orientations. To restore the stone wall by returning the collapsed stones to their original positions, it is necessary to compare images of each stone before and after the collapse, detect matching combinations, and identify the position of the stone in the stone wall corresponding to the matching pre-collapse stone image (pre-collapse stone).

[0019] The processing unit 1 compares one stone from either the pre-collapse stone image or the post-collapse stone image (for example, the post-collapse stone image) with all possible stone images from the other (pre-collapse stone image), extracts the stone image from the other with a high degree of similarity, and displays the matching result. The comparison is performed based on a quantitative evaluation of characteristic information such as feature quantities that indicate the shape characteristics of each stone, that is, based on the degree of difference between the stones being compared. The degree of difference can be calculated, for example, as a value obtained by subtracting the cross-correlation coefficient between feature quantities from 1, the magnitude of the difference between feature quantities, or the distance between feature quantities.

[0020] For this matching process, images of one of the stone materials (stone material before collapse) and the other stone material (stone material after collapse) are acquired. Furthermore, characteristic information such as the contour, location, and shape of one of the stone materials (stone material before collapse) is identified in advance and stored in the storage device 2. Images of each stone material before collapse are obtained by identifying the range of each stone material (contour and the stone material area inside it) from within the image of the stone wall, and then dividing the image according to the identified contour. Feature quantities related to the shape of the contour of each identified stone material are extracted and associated with the pre-collapse stone material image along with the stone material identification information (e.g., identification number) and location information, and stored in the stone material data 21 of the storage device 2.

[0021] It is preferable that stone walls be photographed in advance and comprehensively, each at the optimal orientation and angle of view. However, images taken for other purposes, such as advertising, brochures, or guides, or simple snapshots, may also be used. Since stone walls often have a large exposed area and may face multiple directions, they may be photographed in multiple sections. In this case, identification information (stone wall number) may be assigned to each photographic area.

[0022] On the other hand, images of the collapsed stonework (post-collapse stonework images) are obtained by individually photographing each collapsed stonework by workers or managers at the site. The post-collapse stonework images are also input to the processing unit 1 via the communication unit 14. Here, each collapsed stonework may exist in any orientation depending on how it collapsed. The surface that was exposed on the stone wall before the collapse is significantly different from other surfaces in terms of the amount of soil and moss attached to it, so in most cases it can be easily identified by visual inspection at the time of photography. However, it is difficult to determine the position in terms of the rotational direction of the surface from the post-collapse stonework images alone. There are no particular limitations to the acquisition of feature quantities related to the post-collapse stonework, but it is sufficient to perform this each time a post-collapse stonework image is acquired.

[0023] Images of each stone are acquired so that they are viewed from the front. As mentioned above, the surface that was exposed in the stone wall before the collapse is easily visible, so it is easy for workers to photograph the stones from the front. Alternatively, each stone may be photographed from multiple directions, and, as with the stone wall images described later, the three-dimensional images obtained by applying SfM and MVS processing to the images taken from multiple directions may be orthogonalized to generate a post-collapse stone image viewed from the front. Note that the resolution and scale relative to the actual size of each image used for comparison may differ from one another.

[0024] Figure 2 is a flowchart showing the control procedure for the stone material data generation control process executed by the processing unit 1 of this embodiment. This stone material data generation control process, which includes the stone material matching method of this embodiment, is started and executed by the CPU 11 as part of the program 131, for example, when appropriate image data of a stone wall has been acquired and an input operation related to a start request by the user has been acquired.

[0025] The CPU 11 acquires images of the stone wall from the storage device 2 (step S101). Based on multiple images taken from multiple directions of the common area, the CPU 11 identifies the three-dimensional position of each point exposed on the surface of the stone wall and generates an image of the stone wall as seen from the front (step S102).

[0026] In generating this stone wall image, the CPU 11 selects multiple images of the stone wall that include a specific reference point (reference point: a characteristic point identifiable from the image, whose geographical location has been previously determined by satellite positioning or laser surveying), and calculates the shooting position (which may be relative to the reference point) and shooting direction of each image using SfM (Structure from Motion) or the like. Using the multiple images for which the shooting position and shooting direction have been calculated, the CPU 11 performs stereo matching of the images using, for example, MVS (Multi-View Stereo) processing to reconstruct the three-dimensional shape of the stone wall. The CPU 11 approximates the reconstructed three-dimensional shape of the stone wall surface using TIN (Triangulated Irregular Network) or the like, and generates a three-dimensional image of the stone wall by applying image textures to the range corresponding to each face. By orthogonally projecting this three-dimensional image from the front of the stone wall in the horizontal direction, the stone wall image is obtained.

[0027] The CPU 11 (first contour acquisition means (first contour acquisition step) and first region acquisition means) divides the surface area (stone area) for each pre-collapsed stone by multiscale division processing, etc., and extracts (acquires) its contour (step S103). Specifically, the CPU 11 first initializes each pixel as a separate region. Based on the feature quantities of adjacent regions (pixels), such as color variation, the CPU 11 determines whether these adjacent regions can be merged based on the similarity of those adjacent regions, and repeats this process until there are no more regions that can be merged, thereby identifying a region for each stone. At this time, the upper limit of the number of repetitions of the process and the maximum area of ​​the regions to be merged by the repetitions may also be included in the conditions for determining whether or not to merge. The pixels located on the outer perimeter of each region identified in this way form the contour of each stone. By tracing the pixels on the outer perimeter of this region in order, the contour is represented as polygon data (vector data). Smoothing processing may be applied to this polygon data (vector data). The results of the multiscale division process are displayed on the display unit 15 for visual confirmation by users or administrators, and any inappropriate parts may be corrected as appropriate by accepting correction operations from the operation reception unit 16.

[0028] The CPU 11 generates an elevation view of the stone wall from the stone wall image, drawing a set of outlines of each pre-collapse stone, and stores it in the storage device 2 in association with the stone wall identification information (stone wall number) (step S104). The CPU 11 sets each pre-collapse stone image, which is an image of the area of ​​each stone surface divided by the above outline information, and stores it in the storage device 2 along with the identification information (step S105).

[0029] The CPU 11 selects one of the pre-collapse stone images stored in the memory device 2 (step S106). In this step S106 process, any image that has already been selected is excluded. That is, the CPU 11 selects an unselected pre-collapse stone image from the memory device 2. The CPU 11 (feature extraction means (feature extraction step)) performs a stone feature calculation process using the selected pre-collapse stone image (step S107). The details of the feature calculation process will be described later. The CPU 11 determines whether or not it has selected all of the stored pre-collapse stone images (step S108). If it is determined that there are unselected pre-collapse stone images ("NO" in step S108), the CPU 11 returns to step S106.

[0030] If it is determined that all stored images of the stone materials before the collapse have been selected ("YES" in step S108), the CPU 11 acquires extra-image information relating to each pre-collapse stone material image that cannot be obtained from the image itself, and stores this extra-image information in the storage device 2 in association with the pre-collapse stone material image that it corresponds to (step S109). Extra-image information includes, for example, weight, stone quality, and processing status.

[0031] Next, we will explain the characteristic information of each stone material. The characteristic information of the stone material includes the feature quantities obtained in step S107 above and the extra-image information obtained in step S109.

[0032] Figure 3 is a diagram showing examples of the main feature quantities related to the shape of stone that can be calculated from stone images. The features calculated (extracted) in step S107 of Figure 2 should ideally be robust to rotation of the stone surface during collapse and minor damage. Examples of such features include contour radius, Hu moment, Fourier descriptor, semi-major-minor radius ratio, circularity, and minimum irregularity index.

[0033] The contour radius indicates the relationship between the distance between the center position along a given direction and the contour for each direction (angle) defined at predetermined angular intervals (e.g., 1-degree intervals) from the center position of the stone surface.

[0034] Figure 4 is a diagram illustrating the contour radius. For example, as shown in Figure 4(a), the center position Rg of the stone material region Rc, which is the area of ​​the stone material surface, is defined as the average value of the coordinates of each pixel on the contour Rb surrounding the stone material region Rc, or the average value of the coordinates of all pixels within the contour Rb. From this center position Rg, the distance (radius r) from the center position Rg to the contour Rb is determined for each angle θ. The reference direction r0 of the angle θ may be, for example, the direction in which the maximum value of the radial r obtained in each direction is obtained (the direction in which the distance from the center position Rg to the contour Rb in each direction is determined). The relationship between the angle θ (relative angle to the reference direction r0; deflection) obtained in this way and the radial r is equal for all angles θ if the contour Rb is circular, and if the contour Rb has irregularities, the radial r changes along those irregularities. For example, in the case of a square, a tendency is obtained in which the radial r takes a maximum value every 90 degrees. In the radial r of the stone material region Rc, as shown in Figure 4(b), multiple maxima occur corresponding to the convex parts. The similarity of the trend of change of the radial diameter r with respect to the relative angle can be quantitatively evaluated, for example, by the cross-correlation coefficient (as described later, when evaluating the degree of similarity by the degree of difference (the larger the difference, the greater the difference), you can subtract the cross-correlation coefficient from 1). For example, the degree of difference I1 of the contour radial diameter r can be calculated by the following formula 1.

number

[0035] The Hu moment consists of seven coefficients h0 to h6, which are analytically known as invariants for translation, rotation, and size. The explicit description and detailed explanation of the coefficients h0 to h6 are omitted here. The difference in Hu moments is also known. When calculating the difference as the degree of difference I2 between stone materials, the reciprocal of the product of the logarithm of each coefficient and the sign function value is calculated, the absolute value of the difference between the calculated values ​​between stone materials is found, and then the sum of these absolute values ​​for the seven coefficients is used.

[0036] A Fourier descriptor is a Fourier series representing the changes in the x-component (e.g., the reference direction r0) and y-component (the direction perpendicular to the reference direction r0) when moving along a contour. (By representing the x-component as a real number and the y-component as an imaginary number, the contour can be integrally represented as a trajectory on the complex plane.) Points on the contour may be discrete, and coefficients (Fourier descriptors) c(k) are determined for a number of orders k corresponding to the number of sampling points n. The more sampling points n there are, the more accurate the representation becomes. The degree of difference I3 of the Fourier descriptors can also be calculated using the same procedure as I2 for the Hu moment described above. In this case, instead of adding the seven coefficients, an addition is made based on the number of k.

[0037] The semi-major-semi-minor-maj

[0038] Circularity is the value obtained by dividing the area of ​​the stone surface enclosed by the contour by the square of the contour length of the stone. If the surface is circular, it is a constant of 4π regardless of the radius, and if the stone surface deviates from a circle, it will be a value greater than this constant. Since this value is a scalar value, the degree of difference I5 between stone images can be simply expressed as the absolute value of the difference in circularity.

[0039] The Minimum Irregularity Index is an index that indicates the degree of deviation (degree of irregularity) of the outline from a rectangle, which is a geometrically simple shape suitable for a stone wall. For example, the Minimum Irregularity Index is obtained by subtracting the area of ​​the surface of the stone R (stone area) from the area of ​​the smallest rectangle circumscribing the stone R (minimum area difference), and then normalizing this value by dividing it by the area of ​​the circumscribing rectangle. When stacking stones in multiple layers for a stone wall, it is easier to stack them stably if the stone surface is rectangular. Therefore, by evaluating the degree of deviation using this rectangle as a reference, a value that quantitatively and appropriately characterizes the stones of the stone wall can be obtained.

[0040] Figure 5 illustrates the minimum irregularity index. The rectangle circumscribing the stone area Rc can be defined at any angle, but as shown in Figure 5(a), the area of ​​the circumscribing rectangle Osr1, Osr2, etc., at other angles increases compared to the circumscribing rectangle Os. By setting the circumscribing rectangle while changing the orientation of the rectangle at appropriate angular intervals and identifying the one with the smallest area, the circumscribing rectangle Os with the minimum area can be determined.

[0041] As shown in Figure 5(b), the area inside the circumscribing rectangle Os, indicated by the hatched lines, and outside the stone material region Rc is the part that shows the irregularity of the stone material region Rc. By dividing the area of ​​this part by the area of ​​the circumscribing rectangle Os, the degree of deviation with respect to the size of the circumscribing rectangle Os is normalized, and the minimum irregularity index is obtained. Since the size of the circumscribing rectangle Os is always larger than the stone material region, the minimum irregularity index is positive. The closer the stone material region Rc is to a rectangle, the closer the value of the minimum irregularity index is to 0. The closer it is to a triangle or the larger the irregularities at five or more corners, the larger the positive value of the minimum irregularity index becomes. Since the minimum irregularity index is a scalar value, the degree of difference I6 between stone material images can be expressed, for example, as the absolute value of the difference in the minimum irregularity index. Furthermore, prior to or in conjunction with the use of the minimum irregularity index, criteria may be included to exclude candidates that match if the aspect ratio of the circumscribing rectangle Os of the pre-collapsed stone material differs significantly from the aspect ratio of the circumscribing rectangle Os of the post-collapsed stone material.

[0042] Alternatively, the minimum irregularity index may use the largest inscribed rectangle around the contour of the stone material R instead of the circumscribed rectangle Os around the contour of the stone material R. In this case, the minimum irregularity index would be, for example, the value obtained by subtracting the largest inscribed rectangle area from the area of ​​the stone material region of stone material R (the smallest area difference), and then normalizing this value by dividing it by the area of ​​the inscribed rectangle.

[0043] While it is not necessary for all of the above-described feature quantities to be calculated and acquired, the stone material matching device of this embodiment is capable of calculating (extracting) at least the contour diameter or the minimum irregularity index. Furthermore, other feature quantities not mentioned above may be used in combination.

[0044] Figure 6 is a flowchart showing the control procedure for the feature calculation process, which is step S107 in Figure 3. In the feature extraction process, the CPU 11 calculates the average position of the surface of the selected stone image (step S171). For example, the CPU 11 calculates the average position of the coordinates of all pixels (stone region) located inside the identified contour, or the average position of the coordinates of pixels on the contour.

[0045] The CPU 11 (feature extraction means (feature extraction step)) calculates the distance between the average position and the contour along this direction, while changing the direction from the obtained average position at predetermined angular intervals (for example, at 1-degree intervals). The CPU 11 sets the direction corresponding to the largest of the calculated distances as the reference direction (step S172).

[0046] The feature extraction means calculates the necessary features (step S173). If the features to be calculated include the contour radius, the distance in each direction has already been obtained in step S172, so the CPU 11 converts the obtained directions of each distance into relative angles from the defined reference direction.

[0047] The CPU 11 extracts feature points from each of the selected stone material images (step S174). These feature points are distinctive points that differ from other adjacent stone material regions and are preferably robust against rotation and brightness fluctuations. The method for extracting such feature points may be selected and used from known techniques. Known techniques for extracting feature points include, for example, SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Feature), FAST (Features from Accelerated Segment Test), and Affine-SIFT. Feature points extracted mechanically often include, for example, edges and corners on contours. The CPU 11 then finishes the feature calculation process and returns the process to the stone material data generation control process.

[0048] As described above, after data related to each stone before the collapse is obtained, if image data related to the stone after the collapse is obtained, the image data is used as input, and a process is performed to compare the image of the stone after the collapse with the image of the stone before the collapse to determine which of the pre-collapse stones matches (is the most similar).

[0049] Figure 7 is a flowchart showing the control procedure by the CPU 11 for the stone material matching control process executed by the processing unit 1. This stone material matching control process, which includes the stone material matching method of this embodiment, is included in program 131 and is started and executed by the CPU 11 when, for example, the operation reception unit 16 receives an operation related to a start request by the user.

[0050] The CPU 11 acquires an image of the collapsed stone (step S201). This image of the collapsed stone may have identification information (such as an identification number) attached to it beforehand, or the CPU 11 may add the identification information at the time of acquisition. The CPU 11 may also acquire additional characteristic information such as the weight, type of stone, location of fall, and temporary storage location of the collapsed stone, based on metadata attached to the image of the collapsed stone, and / or input operations received by the operation reception unit 16 or data received by the communication unit 14. The CPU 11 may also store and associate information other than data already associated with the image of the collapsed stone, such as metadata, with the image of the collapsed stone.

[0051] The CPU 11 (second contour acquisition means (second contour acquisition step) and second region acquisition means) divides the region of the collapsed stone from the post-collapsed stone image and extracts (acquires) its contour (step S202). Contour extraction is performed, for example, by multiscale division processing. In this case, since the image originally contains only one stone, the processing divides the area into inside and outside the stone region. In the post-collapsed stone image, the image is taken so that the collapsed stone is located approximately in the center, so the CPU 11 may simply determine that the region including the center of the image is inside the stone region.

[0052] The feature extraction means performs a feature calculation process for the collapsed stone materials (step S203). This feature calculation process is the same as the feature calculation process in Figure 6, which was called in step S107 in Figure 2. In this case, the image range (stone material region) of the collapsed stone materials determined in step S202 becomes the stone material region of the selected stone material image in steps S171 and S174. The other processing details are the same, so their explanation is omitted.

[0053] The CPU 11 (matching means (matching step)) selects any unselected pre-collapse stone material stored in the storage device 2 (step S204). Pre-collapse stone material images that have already been identified as matching other post-collapse stone material images may be excluded from the selection. The matching means calculates the degree of difference between the selected pre-collapse stone material and the post-collapse stone material (step S205). If multiple of the above-mentioned feature quantities or feature points are used to calculate the degree of difference, the matching means may, for example, calculate the degree of difference for each as described above, and then calculate an overall degree of difference by weighting these values ​​appropriately and performing a weighted addition (or weighted average). The degree of difference I7 of the feature points may be, for example, the sum of the distances between feature points that are assumed to correspond. Furthermore, in order to bring the feature points closer together by more appropriately matching the size and rotation direction within each image of the pre-collapse stone material and the post-collapse stone material, a projection transformation may be applied to one of the images (the image of the post-collapse stone material). The coefficients of the projection transformation can be explored, for example, using the RANSAC (Random Sample Consensus) method. For example, the overall difference I calculated by weighted addition is I = Σ (i=1~7) It is represented by Wi·Ii, where Wi is a predetermined positive weight corresponding to the degree of difference I1.

[0054] The matching means determines whether all pre-collapse stone images have been selected (step S206). If it is determined that not all pre-collapse stone images have been selected (there are pre-collapse stone images that have not been selected) ("NO" in step S206), the matching means returns to step S204.

[0055] If it is determined that all pre-collapse stone images have been selected (YES in step S206), the CPU 11 (display control means) displays information on the pre-collapse stone images that are determined to be similar to the post-collapse stone images based on the calculated difference degree using the display unit 15 (step S207). Alternatively, the display control means may output and transmit data related to the displayed image to an external terminal device or the like using the communication unit 14.

[0056] Similarity can be determined, for example, by whether the obtained overall difference is below a predetermined standard value (whether the degree of similarity meets a predetermined standard). The display control means causes the display unit 15 to display a list of pre-collapse stone images that have been determined to have an overall difference below a predetermined standard value, arranged in order of increasing overall difference. Preferably, each stone in the pre-collapse stone image displayed as a matching candidate is rotated to match the orientation of the stone in the post-collapse stone image, where the orientation of the reference direction is displayed.

[0057] The CPU 11 waits for an input operation from the operation reception unit 16 and obtains the content of the operation in which the user selects one of the pre-collapse stone material images from the above list display as matching the post-collapse stone material image (step S208). The CPU 11 identifies the pre-collapse stone material image that matches the post-collapse stone material image according to the obtained operation content (step S209). Then, the CPU 11 terminates the stone material matching control process.

[0058] Figure 8 shows an example of a display screen related to matching. In this example, screen SW1, which displays a selected image of the stone after the collapse, is shown in the upper left, and screen SW4, which displays a list of images of the stone before the collapse that are highly similar to the stone included in the image after the collapse, is shown in the lower right. As described above, each image of the stone before the collapse displayed on screen SW4 is displayed with its reference direction matching the reference direction of the image of the stone after the collapse displayed on screen SW1.

[0059] For example, the pre-collapse stonework located on the left side of screen SW4 has a lower degree of difference from the post-collapse stonework, meaning it is more similar. If there are many pre-collapse stonework pieces with a high degree of similarity, scrolling may be possible. Alternatively, even if the maximum number of pre-collapse stonework pieces that can be displayed is set (for example, 5), and the degree of difference is below a certain threshold, pre-collapse stonework images exceeding the maximum number may be excluded in order of highest degree of difference. The pre-collapse stonework image is displayed on screen SW2 in the upper right so that the selected stonework piece (in this case, the leftmost one) is centered.

[0060] In screens SW1 and SW2, the front view contours of each stone are displayed superimposed on the post-collapse stone image and the stone wall image, based on the elevation drawing of the stone wall. The contours are displayed by discretely identified points on the contour and contour lines that connect these points. The contours displayed in screen SW2 may be a different color from other stones if they correspond to the pre-collapse stones displayed in screen SW4 or the pre-collapse stones selected in screen SW4. In addition, screens SW1 and SW2 also show a line segment Lr0 that passes through the center position Rg of each stone and extends in the reference direction. Each stone is assigned an identification number.

[0061] The lower left screen SW3 is a feature display screen, showing the contour radius for the post-collapsed stone materials (top row) shown on screen SW1 and the pre-collapsed stone materials shown on screen SW4, respectively, with respect to the angle θ on the horizontal axis. In this example, the contour radius of each stone material is displayed vertically at predetermined intervals, but for example, the contour radius of the post-collapsed stone materials and the contour radius of the pre-collapsed stone materials selected on screen SW4 could be overlaid and displayed with different line types or colors.

[0062] In step S208 of Figure 7, the user makes the final judgment on the matching of the pre-collapse and post-collapse stone images via this display screen. For example, when one of the pre-collapse stone images displayed on screen SW4 is selected and confirmed, the processing unit 1 determines the matching combination of pre-collapse and post-collapse stone images. As described above, by displaying the pre-collapse and post-collapse stone images with the same reference direction, the user's judgment can be made easier.

[0063] [Differentiation] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are possible. For example, in the above embodiment, similarity was determined by whether the overall degree of difference met a predetermined criterion, but this is not limited to that. A criterion may be set for each of the degree of difference of multiple features, and similarity may be determined only when all criteria are met (i.e., all degree of difference is small).

[0064] Furthermore, the criteria for determining similarity and the weighting of differences for each feature may be variable. For example, if the difference of a particular feature does not decrease due to the conditions under which the photograph was taken, even if the differences of other features are small, the comparison stone image may not be appropriately selected as a candidate that matches the original stone image. In such cases, especially if the user determines that there are no stone images among the displayed candidates that seem to match the original stone image, the criteria for the feature whose difference does not decrease may be relaxed, or the weighting coefficient may be reduced, and the stone matching may be performed again.

[0065] Furthermore, in the above embodiment, similarity is determined using both the difference in feature quantities and the difference in feature points, but similarity may also be determined without using the difference in feature points.

[0066] Furthermore, although the above embodiment describes displaying a list of similar pre-collapse stone materials in order of decreasing overall difference, it is not limited to this. Among the pre-collapse stone materials identified as similar, they may be displayed in order of decreasing difference in specific feature quantities or feature points. Moreover, the feature quantities or feature points that determine this display order may be changeable by user operation settings received by the operation reception unit 16. Alternatively, the display order may be simply displayed in the order in which they are identified as similar, without sorting by difference.

[0067] Furthermore, in the above embodiment, the degree of difference was calculated to determine similarity, but instead of the degree of difference, the degree of similarity may be calculated to determine the degree of similarity. In this case, the pre-collapse stone materials may be displayed in order of decreasing similarity.

[0068] Furthermore, although the above embodiment was described as detecting a pre-collapse stone image similar to a given post-collapse stone image, the processes may also be performed in the opposite direction to detect a post-collapse stone image corresponding to a pre-collapse stone image at a certain location. In this case, it is necessary that stone data for all post-collapse stone images has been generated and stored in advance.

[0069] The stone material data generation and control processing does not necessarily have to be performed by the processing unit 1. The results of pre-collapse stone material classification, contour extraction, and feature extraction obtained by another computer may be stored in the storage device 2, and the processing unit 1 may retrieve these from the storage device 2. Furthermore, even if the stone material data generation control processing is performed by the processing unit 1, this stone material data generation control processing and the stone material matching control processing may be started when the communication unit 14 receives a start request from an external terminal device.

[0070] Furthermore, if it is difficult to determine the geographic coordinates (absolute coordinates) of the positional information of the stones before the collapse, the position of the stones before the collapse may be determined using relative coordinates based on a predetermined position in the stone wall.

[0071] Furthermore, in the above embodiment, the reference direction r0 was defined as the direction in which the radial length r is maximum, but this is not limited to this. The reference direction r0 may be the direction in which the radial length r is minimum, the average direction of the maximum and minimum directions, or the direction rotated by a predetermined angle from the direction in which the maximum length is maximum.

[0072] Furthermore, if the feature includes the minimum irregularity index but does not include the contour radius, the reference direction r0 may be determined separately based on the contour radius, or it may be determined based on some reference position related to the minimum irregularity index. For example, the point where a line segment extended from the center position Rg to a vertex of the circumscribing rectangle Os intersects the contour and has the largest radius r, or the point where a line passing through the center position Rg and parallel to a side of the circumscribing rectangle Os (e.g., the longer side) intersects the contour and has the larger radius r, may be determined as a point located on the reference direction side from the center position Rg.

[0073] Furthermore, in the above embodiment, the stone material data generation control process and the stone material matching control process used algorithms such as multiscale division processing to identify the contour of the stone material surface, but the embodiment is not limited to this. For example, the contour may be identified manually by the user using dedicated hardware or image editing software. Also, if the contour data is obtained separately from the image data, the coordinates and other elements may be transformed and adjusted to integrate them.

[0074] Furthermore, the above display example is just one example, and the display may be shown in other arrangements. Also, the position and size of each screen SW1 to SW4 may be changed as appropriate. In addition, screens SW1 to SW4 do not have to be displayed in the same window. For example, screen SW1 and screen SW4 may be displayed in different windows, and the user may compare them in a positional relationship as desired.

[0075] Furthermore, while the above description has used a storage unit 13 consisting of non-volatile memory such as an HDD or flash memory as an example of a computer-readable medium for storing the program 131 related to the control of stone data generation and stone material matching of the present invention, the invention is not limited to these. Other computer-readable media can include other non-volatile memories such as MRAM, and portable recording media such as CD-ROMs and DVD discs. In addition, a carrier wave can also be used as a medium for providing the program data of the present invention via a communication line.

[0076] As described above, the processing device 1 as a stone material matching device of this embodiment includes a CPU 11. The CPU 11, as a first contour acquisition means, acquires the contours of the pre-collapse stones included in images of the stone wall taken before the collapse, as a second contour acquisition means, acquires the contours of the post-collapse stones included in images taken after the collapse of the stone wall, as a feature quantity extraction means, extracts feature quantities from each image of the pre-collapse stones and the post-collapse stones that include at least the relationship between the angle θ with respect to the reference direction r0 and the radial r, which is determined based on the distance (radial r) from the center position Rg to the contour in each direction centered on the center position Rg of the stone material region Rc represented as the contour or inside the contour, as a matching means, compares the feature quantities of the pre-collapse stones and the feature quantities of the post-collapse stones to match the pre-collapse stones and the post-collapse stones. Thus, in the processing device 1, when comparing stone materials before and after collapse and extracting similar stone materials, the processing device 1 uses a feature quantity used for quantitative evaluation, which is the change pattern of the contour diameter according to the angle θ with respect to the reference direction determined according to the value of the contour diameter of the stone material. This makes it possible to easily and reliably match the orientation of the stone material after collapse, whose orientation becomes unknown after collapse, with the orientation of the stone material before collapse. Furthermore, since the degree of similarity of the change pattern of the diameter r according to such contour shape is evaluated as a whole, it is less affected by minor damage, scratches, or the resolution of the acquired image, while at the same time being able to easily exclude both partially similar shapes and shapes that are slightly different overall, thereby reducing the workload on the operator and enabling highly accurate matching.

[0077] Furthermore, the CPU 11, as a display control means, displays images of pre-collapsed and post-collapsed stone materials on the display unit 15 in orientations where the reference direction r0 matches, provided that the comparison results meet predetermined criteria related to the degree of similarity. By displaying the pre-collapsed and post-collapsed stone materials side by side in this way, aligned to the most plausible orientation, it becomes easier for workers to compare the two. Therefore, with this processing device 1, it becomes possible for workers to easily and reliably determine whether the comparison results from the processing device 1 are correct or not, with less effort.

[0078] Alternatively, in the processing device 1 of this embodiment, the CPU 11, as a first region acquisition means, acquires the stone region Rc of multiple pre-collapse stones included in an image of the stone wall before collapse, as a second region acquisition means, acquires the stone region Rc of post-collapse stones included in an image taken after the collapse of the stone wall, as a feature extraction means, determines the one (for example, the circumscribed rectangle Os) from the circumscribed rectangle and inscribed rectangle of the stone region Rc that has the smallest area difference from the stone region among the predetermined ones, for both the pre-collapse stones and the post-collapse stones, extracts a feature that includes at least an index related to the degree of irregularity corresponding to the smallest area difference (minimum irregularity index), and as a matching means, compares the feature of the pre-collapse stones with the feature of the post-collapse stones to match the pre-collapse stones and the post-collapse stones. The minimum irregularity index is a feature that does not depend on the orientation of the stone, and the circumscribing rectangle Os with the minimum area itself is determined according to the stone region Rc (contour). Therefore, for example, in cases where the stones appear somewhat similar overall, but it is difficult to determine their identity due to surface stains or scratches, it is easier to appropriately evaluate the degree of similarity while reducing the workload for the worker. Furthermore, the phrase "either the circumscribed rectangle or the inscribed rectangle, as predetermined" here does not mean that the processing unit must perform a process to select one of them. It is sufficient if the processing unit can use only one of them to calculate the minimum irregularity index.

[0079] Furthermore, the stone material matching method of this embodiment includes: a first contour acquisition step of acquiring the contours of the pre-collapse stones included in images of the stone wall taken before the collapse; a second contour acquisition step of acquiring the contours of the post-collapse stones included in images taken after the collapse of the stone wall; a feature quantity extraction step of extracting feature quantities that include at least the relationship between the angle θ with respect to a reference direction r0 and the radial r, which is determined based on the distance (radial r) from the center position Rg to the contour in each direction centered on the center position Rg of the stone material region represented as the contour or the inside of the contour; and a matching step of comparing the feature quantities of the pre-collapse stones and the feature quantities of the post-collapse stones to match the pre-collapse stones and the post-collapse stones. This method of matching stones allows for easy identification of the degree of similarity between stones that have lost their orientation after a stone wall collapse, by matching them to the orientation of the stones before the collapse. In particular, since the comparison does not utilize minute patterns in the stone area, it is less affected by differences in the resolution of the captured images. Therefore, this method of matching stones allows for accurate matching of stones before and after collapse while further reducing the workload of the workers.

[0080] Furthermore, the program 131 of this embodiment causes the computer (at least the CPU 11, RAM 12, and communication unit 14 (input / output unit) of the processing unit 1) to function as a first contour acquisition means for acquiring the contours of pre-collapse stones included in images of the stone wall before collapse, a second contour acquisition means for acquiring the contours of post-collapse stones included in images taken after the collapse of the stone wall, a feature quantity extraction means for extracting feature quantities that at least include the relationship between the angle θ with respect to a reference direction r0 and the radial r, which is determined based on the distance (radial r) from the center position Rg to the contour in each direction centered on the center position Rg of the stone material region Rc represented as the contour or the inside of the contour, for each image of the pre-collapse stones and the post-collapse stones, and a matching means for comparing the feature quantities of the pre-collapse stones and the feature quantities of the post-collapse stones to match the pre-collapse stones and the post-collapse stones. This program 131 makes it possible to easily perform the task of matching stone materials before and after a collapse, which tends to be excessively time-consuming on-site, using standard computer software processing, while also reducing the burden.

[0081] Furthermore, the specific configurations, processing operations, and procedures shown in the above embodiments can be modified as appropriate without departing from the spirit of the present invention. The scope of the present invention includes the scope of the invention described in the claims and its equivalents. [Explanation of Symbols]

[0082] 1 Processing Unit 2 Storage device 11 CPU 12 RAM 13 Storage section 131 Programs 14 Communications Department 15 Display 16 Operation reception section 21 Stone Data 100 Stone Material Matching System Os circumscribed rectangle R stone Rb contour Rc stone area Rg center position r0 Reference direction

Claims

1. A first region acquisition means for acquiring the stone material regions of multiple pre-collapse stone materials contained in an image of the stone wall before it collapsed, A second region acquisition means for acquiring the stone material area of ​​the collapsed stone material included in the image taken after the collapse of the aforementioned stone wall, A feature extraction means that determines, for both the pre-collapse stone and the post-collapse stone, which of the predetermined circumscribing rectangle and inscribed rectangle of the stone area has the smallest area difference with respect to the stone area, and extracts a feature quantity that includes at least an index relating to the degree of irregularity corresponding to the smallest area difference. A comparison means for comparing the features of the pre-collapse stone material and the post-collapse stone material, A stone material matching device characterized by comprising the following features.

2. Computers, A first region acquisition means that acquires the stone material regions of multiple pre-collapse stone materials contained in an image of the stone wall before it collapsed. A second region acquisition means for acquiring the stone material area of ​​the collapsed stone material included in the image taken after the collapse of the aforementioned stone wall, A feature extraction means that determines, for both the pre-collapse stone and the post-collapse stone, which of the predetermined circumscribing rectangle and inscribed rectangle of the stone area has the smallest area difference with respect to the stone area, and extracts a feature quantity that includes at least an index relating to the degree of irregularity corresponding to the smallest area difference. A matching means for comparing the pre-collapse stone material with the post-collapse stone material, A program designed to function as such.

Images