Region extraction apparatus, method, and program

Abstract

To provide a region extraction technique which reduces costs and time required for an operation of a machine learning model by, instead of re-learning using teacher data acquired by an application destination, segmenting an image to a plurality of regions with using an existing database different from that of the application destination, and applying a label to each of these regions.SOLUTION: A region extraction apparatus 10 includes: a database 14 for previously storing each of reference regions 15 obtained by segmenting second images 12 (12a, 12b, 12c) different from a first image 11 together with a label 16 for use in identifying an attribute; an input unit 18 for inputting the first image 11; a segmenting unit 20 for segmenting the first image 11 to some attention regions 17; a collation unit 35 for collating the attention region 17 with the reference region 15 based on attributes; and a giving unit 36 for giving the attention region 17 the same label 16 as that of the reference region 15 having the common attribute.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] An embodiment of the present invention relates to a region extraction technique that assigns a label to each of a plurality of regions obtained by segmenting an image. [Background technology]

[0002] Due to factors such as a shortage of workers, there is a growing need for automated inspections of infrastructure facilities, rather than manual visual inspections. As a result, segmentation technology, which recognizes and extracts areas corresponding to specific conditions or abnormalities from captured images, is attracting attention. One such segmentation technology is semantic segmentation, which classifies small regions, such as pixels, into classes for each type of subject present in an image, thereby dividing the image into regions according to the type of subject.

[0003] Generally, segmentation technology relies on the use of machine learning models, which are built through a process of visiting infrastructure facilities and other sites to collect image data, then creating a large amount of training data by labeling the collected image data.

[0004] In this way, the constructed machine learning model requires a lot of time and cost to operate, and the estimation accuracy decreases when conditions change over time or due to weather, etc., from the time the training data was created. Therefore, in order to maintain the estimation accuracy of the machine learning model, prior art has been disclosed that employs a method of tracking changes in conditions during operation and re-assigning appropriate labels to the training data to be applied. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2023 / 119664 Summary of the Invention [Problem to be solved by the invention]

[0006] In the prior art, the quality of a classification result is determined based on the classification score of the classification result obtained when a machine learning model performs semantic segmentation on images, which are operational data. The machine learning device also generates training data in which each pixel in an image that has been classified as "bad" is labeled with the classification result of each pixel in the corresponding image that has been classified as "good," and trains the machine learning model based on the generated training data. However, training data must be collected prior to the initial operation, and issues remain unresolved in terms of reducing costs and time.

[0007] The embodiments of the present invention have been made taking these circumstances into consideration, and aim to provide a region extraction technology that reduces the cost and time required to operate a machine learning model by using an existing database different from the target application, segmenting an image into multiple regions, and assigning labels to each of these regions, rather than pre-training using teaching data acquired at the target application. [Means for solving the problem]

[0008] The region extraction device according to the embodiment includes a database that stores in advance each of the reference regions obtained by segmenting a second image different from a first image together with a label that identifies its attribute, an input unit that inputs the first image, a segmentation unit that segments the first image into several regions of interest, a matching unit that matches the regions of interest with the reference regions based on the attributes, and an assignment unit that assigns the same label to the regions of interest as the reference regions that share the attributes. [Effects of the Invention]

[0009] According to an embodiment of the present invention, a region extraction technology is provided that reduces the cost and time required to operate a machine learning model by using an existing database different from the target of application, segmenting an image into multiple regions, and assigning labels to each of these regions, rather than pre-training using teaching data acquired at the target of application. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a region extraction device according to an embodiment of the present invention; [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of a segmentation unit of the region extraction device according to the embodiment. [Figure 3] 3 is a flowchart illustrating the steps of a region extraction method and an algorithm of a region extraction program according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] (First embodiment) Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Fig. 1 is a block diagram showing a region extraction device 10 according to a first embodiment of the present invention. As described above, the region extraction device 10 includes a database 14 that stores in advance reference regions 15 obtained by segmenting a second image 12 (12a, 12b, 12c) different from a first image 11, each of which is stored together with a label 16 that identifies its attribute; an input unit 18 that inputs the first image 11; a segmentation unit 20 that segments the first image 11 into several regions of interest 17; a matching unit 35 that matches the regions of interest 17 with the reference regions 15 based on the attributes; and an assignment unit 36 ​​that assigns the same label 16 to the regions of interest 17 as that of the reference regions 15 that share the same attribute.

[0012] The first image 11 is a photograph of an infrastructure facility or other site to which the present embodiment is applied, and includes still images and videos, and is externally input to the region extraction device 10 from an input unit 18. The region of interest 17 refers to a region in the first image 11 in which, for example, a device or an abnormality that is of interest during inspection work or the like is captured.

[0013] The image capturing device (not shown) for the first image 11 is, for example, at least one surveillance camera installed on the roof of a house or at a power plant. The image capturing device may also be installed on a mobile object such as a robot or a drone. In this case, the image capturing device and the region extraction device 10 may be electrically connected by wireless connection.

[0014] The image capturing device may be, for example, a monocular camera or a fisheye camera equipped with a lens and an image sensor. The image capturing device may also be, for example, a visible light camera that captures reflected visible light, an infrared camera, a camera that can acquire a depth map, or a camera that captures images using a distance sensor.

[0015] When a moving image is captured by a camera, subsequent processing may be performed for each frame, or may be performed based on at least one arbitrary frame out of multiple frames. Further, subsequent processing may be performed based on an image obtained by performing arithmetic processing such as averaging on multiple frames, or on an image obtained by performing arithmetic processing such as panoramic stitching on multiple images acquired from multiple surveillance cameras.

[0016] Furthermore, the image captured by the image capture device may be a three-channel color image composed of R, G, and B images, or may be converted to a color system other than the RGB color system through color system conversion processing. The different color system may be, for example, the HLS color system. Alternatively, the image may be a monochrome image obtained by multiplying each channel of the color image by a predetermined coefficient to create a single channel. The pixel Y constituting the monochrome image may be calculated, for example, by Y=0.2126*R+0.7512*G+0·0722*B.

[0017] The second images 12 (12a, 12b, 12c) are images of locations different from the location where the first image 11 was taken, including the location where the first image 11 was taken, and have already been segmented. Reference regions 15, to which labels 16 identifying the attributes of the equipment and structures captured in these second images 12 have been assigned, are stored in a database 14. The reference regions 15 can be obtained from the second image 12 in the same manner as the attention regions 17 described below, and are assigned labels 16 in advance, including by artificial methods. Furthermore, existing training data can be used for the attention regions 17 to which labels 16 have been assigned in this manner.

[0018] 2 is a block diagram showing an example configuration of the segmentation unit 20 of the region extraction device 10 according to the embodiment. As described above, the segmentation unit 20 includes an extraction unit 23 that extracts first feature amounts 21 from the first image 11, a calculation unit 25 that calculates one or more eigenvectors 26 from the first feature amounts 21, a selection unit 27 that selects a combination of the eigenvectors 26 that corresponds to a region of interest 17, and a definition unit 28 that defines the region of interest 17 from the first image 11 based on the selected eigenvectors 26. Note that the segmentation unit 20 is not limited to this configuration, and any unit may be used without particular limitation as long as it has the function of segmenting the first image 11 into several regions of interest 17.

[0019] The first feature amount 21 is an element for identifying an object or the image itself in the first image 11. Specific examples of this element include brightness distribution, color appearance rate, distribution, and type, positional relationship of objects, edges, etc. The extraction unit 23 employs a general method used to extract the first feature amount 21 from the first image 11.

[0020] The calculation unit 25 calculates one or more eigenvectors 26 together with eigenvalues ​​from the first feature 21. The eigenvectors 26 can be obtained by matrix transformation of the first feature 21 extracted from the values ​​of the intermediate layer of the deep neural network based on the spectral graph theory. The matrix to be transformed may be, for example, a Laplacian matrix, where L is, for example, L = DW or L = D^(-1 / 2)(DW)D^(-1 / 2). Note that the method of calculating the eigenvectors 26 is not particularly limited to the above-mentioned method, although it is an example.

[0021] The selection unit 27 selects a combination of eigenvectors 26 that corresponds to the region of interest 17. The eigenvectors 26 to be selected may be a predetermined number of the top eigenvectors 26 selected from the matrix converted from the first feature amount 21, or may be selected arbitrarily by the user. Alternatively, a combination of eigenvectors 26 that are close in direction or far from each other may be automatically selected based on the direction of the eigenvectors 26. Based on the eigenvectors 26 selected in this manner, the definition unit 28 can define the region of interest 17 from the first image 11.

[0022] Returning to Figure 1, the explanation continues. The matching unit 35 matches the attention area 17 with the reference area 15 based on the attributes and determines whether they are identical. The matching unit 35 can also determine the commonality of attributes between the attention area 17 and the reference area 15 by matching second feature amounts (not shown) extracted from each of the attention area 17 and the reference area 15. In this case, the determination of whether the attention area 17 and the reference area 15 match can be made by calculating the similarity of the respective second feature amounts (not shown) and judging whether the similarity is equal to or greater than a predetermined threshold value set in advance.

[0023] The second feature amount (not shown) of each of the attention region 17 and the reference region 15 may be acquired by filtering processing or a local feature descriptor, or may be encoded from a deep neural network model pre-trained with a public dataset, etc. Also, the second feature amount (not shown) of each of the reference regions 15 may be stored in the database 14 in advance.

[0024] The assigning unit 36 ​​assigns the same label 16 to the attention area 17 as the reference area 15 with which the attribute is common. Here, the label 16 may be defined based on, for example, a proper noun representing the subject in the first image 11, or an ID such as a number or alphabet may be assigned based on some rule. If the unknown feature does not match any known feature, a label 16 indicating, for example, "Unknown" may be issued.

[0025] Furthermore, the region extraction device 10 includes a pixel conversion unit 37 that performs pixel conversion so that the regions of interest 17 to which the common label 16 has been assigned can be distinguishably displayed in the first image 11. This allows the display unit 38 to display each of the multiple regions of interest 17 in a format in which they are extracted from the first image 11.

[0026] It should be noted that the pixel conversion unit 37 and the display unit 38 are not essential components. The region extraction device 10 can update the region of interest 17 to which the label 16 has been assigned as new teaching data 19 to be applied to machine learning. This allows the estimation accuracy of the machine learning model to be maintained by following changes in the situation during operation. The region of interest 17 to which the label 16 has been assigned can also be stored in the database 14 as a reference region 15 to which the label 16 has been assigned.

[0027] (Second embodiment) Next, a second embodiment of the present invention will be described with reference to Fig. 1. In the region extraction device 10 of the second embodiment, when the collation unit 35 extracts the second feature amount (not shown) from the region of interest 17, a representative pixel of the region of interest 17 is extracted as the second feature amount (not shown). Similarly, the second feature amount of the reference region 15 is extracted from the representative pixel. Then, the assignment unit 36 ​​assigns the corresponding label 16 to all pixels constituting the region of interest 17.

[0028] In the second embodiment, the representative pixel extracted as the second feature may be the average pixel value, or the second feature may be obtained by filtering a set of representative pixels or by using a local feature descriptor. Alternatively, the second feature may be encoded from a deep neural network model pre-trained on a public dataset or the like.

[0029] The display unit 38 displays the first image 11 in a format in which each of the multiple regions of interest 17 is extracted. Each of the regions of interest 17 extracted in the first image 11 may be configured with color information representing a label 16 (e.g., red: R=255, G=0, B=0), or may be accompanied by indexed information (e.g., if a serial ID=1 is assigned to a certain object, the pixel information of the indexed image may be R=1, G=1, B=1).

[0030] (Third embodiment) Next, a third embodiment of the present invention will be described with reference to Fig. 1. In the region extraction device 10 of the third embodiment, when a second feature (not shown) is extracted from the region of interest 17 in the matching unit 35, the second feature (not shown) is extracted from a rectangle that encompasses the region of interest 17. Similarly, the second feature of the reference region 15 is preferably extracted from a rectangle that encompasses the reference region 15. Then, the assigning unit 36 ​​assigns corresponding labels 16 to all pixels that make up the region of interest 17.

[0031] In the third embodiment, the extracted second feature may be obtained from a rectangular region by filtering or a local feature descriptor, or may be encoded from a deep neural network model that has been pre-trained using a public dataset or the like.

[0032] Next, the steps of the region extraction method and the algorithm of the region extraction program according to the embodiment will be described with reference to the flowchart in Figure 3 (see Figure 1 where appropriate). First, each of the reference regions 15 obtained by segmenting the second image 12 (12a, 12b, 12c) is created in the stored database 14 together with a label 16 (S11).

[0033] Next, a first image 11 captured at the site to which the system is to be applied is input (S12). The first image 11 is then segmented into several regions of interest 17 (S13). Next, the regions of interest 17 are compared with the reference region 15 based on their attributes (S14). If the attributes are common (S15, Yes), the same label 16 as the reference region 15 is assigned to the region of interest 17 (S16). On the other hand, if the attributes are not common (S15, No), a label 16 indicating "Unknown" is assigned to the region of interest 17 (S17).

[0034] Then, the first image 11, which has been pixel-converted so that the attention area 17 to which the common label 16 has been assigned is identifiable, is displayed on the display unit 38 (S18, END).

[0035] According to at least one of the embodiments of the region extraction device 10 described above, instead of pre-learning using teaching data acquired at the application destination, a database 14 of existing second images 12 that are different from the first image 11 at the application destination is used to segment the first image 11 into multiple regions of interest 17, and labels 16 are assigned to each of these regions of interest 17, thereby making it possible to reduce the cost and time required to operate a machine learning model.

[0036] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as the inventions described in the claims and their equivalents.

[0037] The above-described region extraction device includes a control device with a highly integrated processor such as a dedicated chip, FPGA (Field Programmable Gate Array), GPU (Graphics Processing Unit), or CPU (Central Processing Unit), a storage device such as ROM (Read Only Memory) or RAM (Random Access Memory), an external storage device such as HDD (Hard Disk Drive) or SSD (Solid State Drive), a display device such as a monitor, an input device such as a mouse or keyboard, and a communication I / F, and can be realized with a hardware configuration using a normal computer. Therefore, the components of the region extraction device can also be realized by a computer processor and can be operated by a region extraction program.

[0038] The region extraction program may be provided by being pre-installed in a ROM or the like. Alternatively, the program may be provided by being stored in an installable or executable file format on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD, or flexible disk (FD).

[0039] The area extraction program according to this embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. The area extraction device may also be configured by combining separate modules that independently perform the functions of the components and are interconnected via a network or dedicated lines. [Explanation of symbols]

[0040] 10...region extraction device, 11...first image, 12...second image, 14...database, 15...reference region, 16...label, 17...region of interest, 18...input unit, 19...teaching data, 20...segmentation unit, 21...feature, 23...extraction unit, 25...calculation unit, 26...eigenvector, 27...selection unit, 28...definition unit, 35...matching unit, 36...assignment unit, 37...pixel conversion unit, 38...display unit.

Claims

1. a database that stores in advance each of the reference regions obtained by segmenting a second image different from the first image, together with a label that identifies the attribute of the reference region; an input unit for inputting the first image; a segmentation unit for segmenting the first image into a number of regions of interest; a matching unit that matches the region of interest with the reference region based on the attribute; an assigning unit that assigns the same label to the region of interest as the reference region having the common attribute.

2. 2. The area extraction device according to claim 1, The segmentation unit an extraction unit that extracts a first feature amount from the first image; a calculation unit that calculates one or more eigenvectors from the first feature amount; a selection unit that selects a combination corresponding to the region of interest from among the eigenvectors; a defining unit that defines the region of interest from the first image based on the selected eigenvector.

3. 3. The area extraction device according to claim 1, A region extraction device that determines the commonality of the attributes by comparing second feature amounts extracted from the region of interest and the reference region.

4. 3. The area extraction device according to claim 1, A region extraction device comprising: a pixel conversion unit that converts pixels so that the region of interest to which the common label is assigned can be identified and displayed from the first image.

5. 4. The area extraction device according to claim 3, A region extraction device that extracts a representative pixel of the region of interest as the second feature amount.

6. 4. The area extraction device according to claim 3, A region extraction device that extracts the second feature amount from a rectangle that includes the region of interest.

7. storing in advance in a database each of reference regions obtained by segmenting a second image different from the first image together with a label identifying its attribute; inputting the first image; Segmenting the first image into several regions of interest; matching the region of interest to the reference region based on the attributes; and assigning the same label to the region of interest as to the reference region having the common attribute.

8. On the computer, a step of storing in advance in a database each of reference regions obtained by segmenting a second image different from the first image, together with a label identifying its attribute; inputting the first image; Segmenting the first image into several regions of interest; matching the region of interest to the reference region based on the attributes; assigning the same label to the region of interest as to the reference region having the common attribute.

Images