Image search system, image search method, and image search program

Abstract

To provide an image search system, an image search method, and an image search program that efficiently search for a similar image.SOLUTION: A support server 20 includes a control unit 21 connected to a user terminal 10. The control unit 21 acquires a query image in a query region designated by the user terminal 10 in an input image, and identifies a similar image candidate in the input image that is similar to the query image. Furthermore, the control unit 21 uses a pre-learning model to calculate a feature of the query image and a feature of the similar image candidate, and identifies, on the basis of a comparison result of the features, a similar image corresponding to the query image and outputs it to the user terminal 10.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an image search system, an image search method, and an image search program for searching for similar images in drawings.

Background Art

[0002] When extracting similar images, feature amounts of images may be used (see, for example, Patent Document 1). The information processing apparatus described in this patent document includes an acquisition unit, an extraction unit, and a learning unit. The acquisition unit acquires a feature amount model that outputs a feature amount of an image with the image as an input, and search targets that are a plurality of images each including a transaction target. The extraction unit extracts a group of similar images that are similar among the search targets, using a plurality of feature amounts corresponding to each of the plurality of images generated using the feature amount model. The learning unit updates parameters of the feature amount model using a dataset in which the group of similar images is an image including the same transaction target.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For example, in architectural design drawings, the arrangement of each facility is represented by respective symbols and the like. Here, when searching for the same facility in the drawing, the feature amount of the notation can be used. However, the notation of the facility may differ depending on the drawing. For this reason, when the feature amounts of the notations are different, they cannot be extracted. Further, when a symbol is added as a new notation, re-learning is required, so they cannot be extracted with the existing feature amounts.

Means for Solving the Problems

[0005] The image search system that solves the above problems includes a control unit connected to a user device. Then, in the input image, the control unit acquires a query image in a query area specified by the user device, identifies candidate similar images similar to the query image in the input image, calculates the feature amounts of the query image and the candidate similar images using a pre-trained model, identifies a similar image corresponding to the query image according to the comparison result of the feature amounts, and outputs it to the user device.

Effect of the Invention

[0006] According to the present disclosure, similar images can be efficiently retrieved.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Mode for Carrying Out the Invention

[0008] An embodiment in which an image search system, an image search method, and an image search program are embodied will be described with reference to FIGS. 1 to 4. In this embodiment, for example, it is assumed that the same equipment is searched for in a design drawing. As shown in FIG. 1, the image search system of this embodiment uses a user terminal 10 (user device) and a support server 20 that are interconnected via a network.

[0009] (Example of Hardware Configuration) FIG. 2 is an example of the hardware configuration of the information processing apparatus H10 that functions as the user terminal 10, the support server 20, etc.

[0010] The information processing apparatus H10 includes a communication device H11, an input device H12, a display device H13, a storage device H14, and a processor H15. Note that this hardware configuration is an example, and it may have other hardware.

[0011] The communication device H11 is an interface that establishes a communication path with other devices and performs data transmission and reception, and is, for example, a network interface, a wireless interface, or the like.

[0012] The input device H12 is a device that receives input from a user or the like, and is, for example, a mouse, a keyboard, or the like. The display device H13 is a display, a touch panel, or the like that displays various information.

[0013] The storage device H14 is a storage device that stores data and various programs for executing various functions of the user terminal 10 and the support server 20. Examples of the storage device H14 include a ROM, a RAM, a hard disk, and the like.

[0014] The processor H15 controls each process in the user terminal 10 and the support server 20 (for example, the process in the control unit 21 described later) using the programs and data stored in the storage device H14. Examples of the processor H15 include a CPU, an MPU, and the like. This processor H15 expands the program stored in the ROM or the like into the RAM and executes various processes corresponding to various processes. For example, when the application program of the user terminal 10 and the support server 20 is started, the processor H15 operates a process that executes each process described later.

[0015] Processor H15 is not limited to performing software processing for all processes it executes. For example, processor H15 may include a dedicated hardware circuit (e.g., an application-specific integrated circuit: ASIC) that performs hardware processing for at least a part of the processes it executes. That is, processor H15 may be configured as follows.

[0016] (1) One or more processors that operate according to a computer program (software) (2) One or more dedicated hardware circuits that execute at least a part of various processes (3) A circuitry including a combination thereof The processor includes a CPU and memories such as a RAM and a ROM. The memories store program codes or instructions configured to cause the CPU to execute processes. The memories, i.e., computer-readable media, include any available media accessible by a general-purpose or dedicated computer.

[0017] (Functions of user terminal 10 and support server 20) Using FIG. 1, the functions of user terminal 10 and support server 20 will be described. User terminal 10 is a computer terminal used by a user who uses this system.

[0018] Support server 20 is a computer system that performs image search. This support server 20 includes a control unit 21 and a learning model storage unit 22. Control unit 21 performs processes (processes including an acquisition stage, a candidate extraction stage, a feature extraction stage, etc.) described later. By executing an image search program for this purpose, control unit 21 functions as a management unit 210, a candidate extraction unit 211, a feature extraction unit 212, etc.

[0019] Management unit 210 executes a process of acquiring drawings and information regarding a query area from user terminal 10. The candidate extraction unit 211 executes a process of extracting similar image candidates by template matching in the drawing. The feature extraction unit 212 executes a process of comparing the feature amounts of the similar image candidates with the feature amounts of the query image and specifying similar image candidates similar to the query image.

[0020] The pre-trained model storage unit 22 stores a pre-trained model for calculating the feature amounts of images. This pre-trained model is recorded when machine learning for calculating feature amounts is performed using a large-scale dataset including various images. As this pre-trained model, one for calculating feature amounts obtained by vectorizing general-purpose images is used. In the present embodiment, it is used for calculating the feature amounts of similar image candidates and query images.

[0021] (Search process) Next, the search process will be described with reference to FIG. 3. Here, the control unit 21 of the support server 20 executes a process of acquiring the input drawing (step S11). Specifically, the management unit 210 of the control unit 21 acquires the input drawing from the user terminal 10 and temporarily stores it in the memory. In the present embodiment, an input drawing (input image) obtained by capturing a paper drawing with a scanner device is acquired. Here, as shown in FIG. 4(a), a case of using the input drawing 500 is assumed.

[0022] Next, the control unit 21 of the support server 20 executes a process of designating a query area in the input drawing (step S12). Specifically, the management unit 210 of the control unit 21 outputs the input image temporarily stored in the memory to the display device H13 of the user terminal 10. The user designates an area (query area) including the facility (object) to be searched in the output input drawing. In this case, the management unit 210 acquires the query image included in the query area designated in the user terminal 10. Here, as shown in FIG. 4(a), a case of designating the query area 501 in the input drawing 500 is assumed.

[0023] Next, the control unit 21 of the support server 20 executes a calculation process for the first feature amount of the query image (step S13). Specifically, the feature extraction unit 212 of the control unit 21 calculates the feature amount (first feature amount) of the query image using the pre-trained model recorded in the learning model storage unit 22.

[0024] Also, the control unit 21 of the support server 20 executes an extraction process for candidate similar images of the query image (step S14). Specifically, the candidate extraction unit 211 of the control unit 21 searches for candidate similar images in the input drawing by template matching using the query image as a template. In this case, resizing to change the size of the template, rotation at a predetermined angle, inversion, etc. are performed. Also, the contour features of the template image may be extracted, and a contour search (geometric shape search) using a model in which the features are vectorized may be used. In this case, the features of the template are extracted and vectorized, and candidate similar images are extracted based on the number of matching vectors. Then, the candidate extraction unit 211 temporarily stores the area position where the candidate similar images are extracted on the drawing.

[0025] Next, the control unit 21 of the support server 20 executes a calculation process for the second feature amount of the candidate similar images (step S15). Specifically, the feature extraction unit 212 of the control unit 21 calculates the feature amount (second feature amount) of the extracted candidate similar images using the pre-trained model recorded in the learning model storage unit 22.

[0026] Next, the control unit 21 of the support server 20 executes a comparison process for the first and second feature amounts (step S16). Specifically, the feature extraction unit 212 of the control unit 21 compares the first feature amount of the query image with the second feature amount of each candidate similar image.

[0027] Next, the control unit 21 of the support server 20 executes a narrowing-down process of similar image candidates according to the comparison result (step S17). Specifically, the feature extraction unit 212 of the control unit 21 identifies the region position of a similar region including similar image candidates with a second feature amount whose difference from the first feature amount is within a predetermined range with respect to the first feature amount. Then, the management unit 210 identifies the specified similar region on the input drawing and outputs it to the display device H13 of the user terminal 10.

[0028] As shown in FIG. 4(b), in the input drawing 500, a similar region 502 is displayed. In this case, the user uses the user terminal 10 to check the similar region 502 and select the facility to be specified.

[0029] (Operation of the Embodiment) Since the feature amounts of the query image and the similar image candidates are calculated using the pre-trained model, a similar region similar to the query region is extracted.

[0030] (Effect of the Embodiment) According to the present embodiment, the following effects can be obtained. (1-1) In the present embodiment, the control unit 21 of the support server 20 executes a calculation process of the first feature amount of the query image (step S13). When performing machine learning, a large amount of teacher information is used for the image to be searched. Since the pre-trained model is used, the feature amount of the query image can be calculated without performing machine learning using the image to be searched as teacher information.

[0031] (1-2) In the present embodiment, the control unit 21 of the support server 20 executes an extraction process of similar image candidates for the query image (step S14). Thereby, similar image candidates can be efficiently identified by pattern matching with a small computational load.

[0032] (1-3) In this embodiment, the control unit 21 of the support server 20 executes a calculation process for the second feature amount of the similar image candidates (step S15). Thereby, the feature amount of the similar image candidates can be calculated without performing machine learning using the image to be searched as teacher information.

[0033] (1-4) In this embodiment, the control unit 21 of the support server 20 executes a comparison process for the first and second feature amounts (step S16) and a narrowing-down process for the similar image candidates according to the comparison result (step S17). Thereby, using pattern matching and a pre-learning model, similar images corresponding to the query region can be efficiently extracted in the input drawing.

[0034] (Second Embodiment) Next, a second embodiment in which an image search system, an image search method, and an image search program are embodied will be described with reference to FIG. 5. In the above first embodiment, the control unit 21 of the support server 20 executes a designation process for the query region in the input drawing (step S12). In this embodiment, related images of the query image are output. In this case, images that may be used for the search are recorded in the related image information storage unit 23. For example, facility symbols, facility illustrations, etc. that may be used in the drawing are registered in the related image information storage unit 23. For example, symbols of JIS standards and images that have obtained good user feedback among the user's search results are registered as related images. In the following embodiments, parts that are the same as those in the above first embodiment are denoted by the same reference numerals, and detailed descriptions thereof are omitted.

[0035] As shown in FIG. 5, the control unit 21 of the support server 20 executes an acquisition process for the input drawing (step S11) and a designation process for the query region in the input drawing (step S12). Next, the control unit 21 of the support server 20 executes a process of acquiring related images of the query image (step S21). Specifically, the management unit 210 of the control unit 21 extracts related images similar to the query image included in the query area from the related image information storage unit 23. In this case, it is possible to use the pre-trained model recorded in the learning model storage unit 22. Also, a pattern matching method may be used.

[0036] Next, the control unit 21 of the support server 20 executes a process of proposing related images (step S22). Specifically, the management unit 210 of the control unit 21 outputs the extracted related images to the display device H13 of the user terminal 10 and makes a proposal to prompt the selection of related images. Then, when a related image is selected in the user terminal 10, the management unit 210 adds the related image as a query image.

[0037] Next, the control unit 21 of the support server 20 executes a process of calculating the first feature amount of the query image (step S13). Here, the feature extraction unit 212 calculates the respective first feature amounts for the related images together with the query image in the designated query area. Then, hereinafter, processing is performed using all the calculated first feature amounts.

[0038] Also, the control unit 21 of the support server 20 executes an extraction process of similar image candidates for the query image (step S14) and a calculation process of the second feature amount of the similar image candidates (step S15).

[0039] Next, the control unit 21 of the support server 20 executes a comparison process of the first and second feature amounts (step S16) and a process of narrowing down similar image candidates according to the comparison result (step S17). When using a plurality of first feature amounts, the area position of the similar area including the similar image candidates of the second feature amount whose difference from any one of the first feature amounts is within a predetermined range is specified.

[0040] According to this embodiment, in addition to the effects of the first embodiment, the following effects can be obtained. (2-1) In this embodiment, the control unit 21 of the support server 20 executes the acquisition process of related images of the query image (step S21) and the proposal process of related images (step S22). Thereby, not only the query image of the query area specified by the user, but also related images can be used to comprehensively identify the desired facilities.

[0041] This embodiment can be implemented with the following modifications. This embodiment and the following modification examples can be implemented in combination with each other within a technically non - conflicting range. · In each of the above embodiments, the user terminal 10 and the support server 20 are used, but the hardware configuration is not limited thereto. For example, the user terminal 10 and the support server 20 may be realized using a single piece of hardware or by cloud computing. · In each of the above embodiments, it is assumed that the same facilities are searched in the design drawings. The search target is not limited to the design drawings. For example, in a photographed image of materials or parts, a query area may be specified to search for similar materials or parts.

[0042] · In each of the above embodiments, the control unit 21 of the support server 20 executes the designation process of the query area in the input drawing (step S12). Here, the query image may be corrected. For example, if there are hidden parts in the query area by characters or other objects, etc., the image may be corrected to complement it. In this case, an image generation model is used to complement the image. Also, only the main objects displayed in the query area may be extracted. Thereby, the objects desired by the user can be identified.

[0043] · In each of the above embodiments, a pre-trained model for calculating the feature amount of an image is recorded in the learning model storage unit 22. Here, a plurality of different pre-trained models may be prepared and recorded in the learning model storage unit 22. In this case, the first and second feature amounts are calculated using each pre-trained model, and the difference is calculated for each pre-trained model. Then, the management unit 210 may narrow down the similar image candidates by a majority vote of agreement and disagreement.

[0044] · In each of the above embodiments, the control unit 21 of the support server 20 executes the acquisition process of the input drawing (step S11). Here, the input drawing is acquired by an image obtained by scanning a paper drawing. The input drawing is not limited to a scanned image of a paper drawing. For example, a drawing created by CAD (Computer Aided Design) may be used. Also, a PDF exported from a CAD drawing may be used.

[0045] Further, the input drawing may be composed of a plurality of layers. In the case of input drawing data composed of a plurality of layers, similar image candidates may be extracted in the layer in which the query area is specified. Also, when the query area is specified, the information of the layers that are not specified may be deleted.

[0046] · In each of the above embodiments, the control unit 21 of the support server 20 executes the designation process of the query area in the input drawing (step S12). In addition to this, query conditions may be acquired from the user terminal 10 by text. In this case, the text included in the drawing is extracted by character recognition. Then, the similar image candidates may be narrowed down based on the agreement or disagreement between the text extracted within a predetermined range of the similar image candidates and the query conditions (text).

[0047] In addition to the similar image candidates narrowed down according to the comparison result, the area extracted by searching with the query conditions (text) may be output to the display device H13 of the user terminal 10.

[0048] · In the above-described second embodiment, the control unit 21 of the support server 20 executes a process of acquiring related images of the query image (step S21). In this case, the related images to be proposed may be learned according to the user's confirmation result. In this case, as the confirmation result, the time required until the user determines the search result and the confirmation order are used. In this case, the control unit 21 of the support server 20 records the time required until the user determines the related image and the confirmation order. Then, the control unit 21 gives priority to the related images with a short required time and the related images with an early confirmation order and makes a proposal.

Description of Reference Numerals

[0049] 10… User terminal, 20… Support server, 21… Control unit, 210… Management unit, 211… Candidate extraction unit, 212… Feature extraction unit, 22… Learning model storage unit, 501… Query area.

Claims

1. An image search system comprising a control unit connected to a user device, wherein the control unit obtains a query image in a query area specified by the user device in an input image, identifies candidate similar images similar to the query image in the input image, calculates feature amounts of the query image and the candidate similar images using a pre-trained model, and identifies a similar image corresponding to the query image according to a comparison result of the feature amounts and outputs the similar image to the user device. An image search system characterized by this.

2. wherein the control unit obtains related images of the query image, and adds the related images to the query image. The image search system according to claim 1, characterized by this.

3. wherein the control unit outputs the related images to the user device, and adds the related images selected by the user device to the query image. The image search system according to claim 2, characterized by this.

4. wherein the control unit obtains a confirmation result of the similar images output to the user device, and identifies the related images according to the confirmation result. The image search system according to claim 2, characterized by this.

5. A method for searching for similar images using an image search system comprising a control unit connected to a user device, wherein the control unit obtains a query image in a query area specified by the user device in an input image, identifies candidate similar images similar to the query image in the input image, Using a pre-trained model, calculate the feature amounts of the query image and the candidate similar images, An image search method characterized by identifying a similar image corresponding to the query image according to the comparison result of the feature amounts and outputting the image to the user device.

6. A program for searching for similar images using an image search system including a control unit connected to a user device, The control unit, In an input image, obtain a query image in a query region designated by the user device, In the input image, identify candidate similar images similar to the query image, Using a pre-trained model, calculate the feature amounts of the query image and the candidate similar images, An image search program for functioning as means for identifying a similar image corresponding to the query image according to the comparison result of the feature amounts and outputting the image to the user device.

Images