Image search system, image search method, and image search program

Abstract

To provide an image search system, an image search method, and an image search program that efficiently search for a similar image.SOLUTION: A support server 20 includes a control unit 21 connected to a user terminal 10. The control unit 21 acquires a query image in a query region designated by the user terminal 10 in an input image, and identifies a similar image candidate in the input image that is similar to the query image. Furthermore, the control unit 21 uses a pre-learning model to calculate a feature of the query image and a feature of the similar image candidate, and identifies, on the basis of a comparison result of the features, a similar image corresponding to the query image and outputs it to the user terminal 10.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image search system, an image search method, and an image search program for searching for similar images in drawings. [Background technology]

[0002] When extracting similar images, image features may be used (see, for example, Patent Document 1). The information processing device described in this patent document has an acquisition unit, an extraction unit, and a learning unit. The acquisition unit acquires a feature model that receives an image as input and outputs image features, and a search target that is a plurality of images each containing a transaction object. The extraction unit extracts a group of similar images from the search target using a plurality of feature values corresponding to each of the multiple images generated using the feature model. The learning unit updates the parameters of the feature model using a dataset in which the similar image group is images containing the same transaction object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-000313 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, in architectural design drawings, the location of each piece of equipment is indicated by a symbol or other symbol. When searching for the same piece of equipment in a drawing, the notation feature can be used. However, the notation of the equipment may differ depending on the drawing. For this reason, extraction is not possible when the notation feature is different. Furthermore, when a symbol is added as a new notation, re-learning is required, and extraction is not possible using existing feature values. [Means for solving the problem]

[0005] An image search system that solves the above problem includes a control unit connected to a user device, which acquires a query image in a query region specified by the user device in an input image, identifies candidate similar images in the input image that are similar to the query image, calculates feature amounts of the query image and the candidate similar images using a pre-trained model, and identifies similar images corresponding to the query image based on a comparison result of the feature amounts, and outputs the identified similar images to the user device. [Effects of the Invention]

[0006] According to the present disclosure, similar images can be efficiently searched for. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is an explanatory diagram of an image search system according to a first embodiment. [Figure 2] FIG. 2 is an explanatory diagram of a hardware configuration of the first embodiment. [Figure 3] FIG. 2 is an explanatory diagram of a processing procedure according to the first embodiment. [Figure 4] 1A and 1B are explanatory diagrams of the drawings of the first embodiment, in which FIG. 1A is an explanatory diagram of specifying a query region, and FIG. 1B is an explanatory diagram of displaying a similar image region. [Figure 5] FIG. 10 is an explanatory diagram of a processing procedure according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] An embodiment of an image search system, an image search method, and an image search program will be described with reference to Figures 1 to 4. In this embodiment, it is assumed that the same equipment is searched for in design drawings, for example. As shown in FIG. 1, the image search system of this embodiment uses a user terminal 10 (user device) and a support server 20 that are interconnected via a network.

[0009] (Example of hardware configuration) FIG. 2 shows an example of the hardware configuration of an information processing device H10 that functions as the user terminal 10, the support server 20, and the like.

[0010] The information processing device H10 includes a communication device H11, an input device H12, a display device H13, a storage device H14, and a processor H15. Note that this hardware configuration is an example, and the information processing device H10 may include other hardware.

[0011] The communication device H11 is an interface that establishes a communication path with another device and executes data transmission and reception, and is, for example, a network interface or a wireless interface.

[0012] The input device H12 is a device that accepts input from a user, etc., and is, for example, a mouse, a keyboard, etc. The display device H13 is a display, a touch panel, etc. that displays various information.

[0013] The storage device H14 is a storage device that stores data and various programs for executing various functions of the user terminal 10 and the support server 20. Examples of the storage device H14 include a ROM, a RAM, and a hard disk.

[0014] The processor H15 controls each process in the user terminal 10 and the support server 20 (for example, the process in the control unit 21, which will be described later) using the programs and data stored in the storage device H14. Examples of the processor H15 include a CPU and an MPU. The processor H15 loads a program stored in a ROM or the like into a RAM and executes various processes corresponding to the various processes. For example, when an application program in the user terminal 10 or the support server 20 is started, the processor H15 operates a process that executes each process, which will be described later.

[0015] The processor H15 is not limited to a processor that performs all of its processing using software. For example, the processor H15 may include a dedicated hardware circuit (e.g., an application-specific integrated circuit (ASIC)) that performs hardware processing for at least some of the processing it performs. That is, the processor H15 may be configured with the following:

[0016] (1) One or more processors operating according to a computer program (software) (2) One or more dedicated hardware circuits that perform at least some of the various processes (3) Circuits, including combinations thereof The processor includes a CPU and memory, such as RAM and ROM, that stores program code or instructions configured to cause the CPU to perform processes. Memory, or computer-readable media, includes any available media that can be accessed by a general-purpose or special-purpose computer.

[0017] (Functions of user terminal 10 and support server 20) The functions of the user terminal 10 and the support server 20 will be described with reference to FIG. The user terminal 10 is a computer terminal used by a user who uses this system.

[0018] The support server 20 is a computer system that performs image search. The support server 20 includes a control unit 21 and a learning model storage unit 22. The control unit 21 performs the processes described below (processes including an acquisition stage, a candidate extraction stage, a feature extraction stage, etc.) By executing an image search program for this purpose, the control unit 21 functions as a management unit 210, a candidate extraction unit 211, a feature extraction unit 212, etc.

[0019] The management unit 210 executes a process of acquiring information about drawings and query regions from the user terminal 10. In the drawing, the candidate extraction unit 211 executes a process of extracting similar image candidates by template matching. The feature extraction unit 212 performs a process of comparing the feature amounts of the similar image candidates with the feature amounts of the query image, and identifying similar image candidates that are similar to the query image.

[0020] The learning model storage unit 22 stores a pre-learning model for calculating image features. This pre-learning model is stored when machine learning is performed to calculate features using a large-scale dataset containing a variety of images. This pre-learning model calculates features obtained by vectorizing a general-purpose image. In this embodiment, this model is used to calculate the features of similar image candidates and a query image.

[0021] (Search processing) Next, the search process will be described with reference to FIG. Here, the control unit 21 of the support server 20 executes an input drawing acquisition process (step S11). Specifically, the management unit 210 of the control unit 21 acquires the input drawing from the user terminal 10 and temporarily stores it in memory. In this embodiment, the input drawing (input image) is acquired by capturing a paper drawing with a scanner device. Here, it is assumed that an input drawing 500 is used as shown in FIG. 4(a).

[0022] Next, the control unit 21 of the support server 20 executes a process of specifying a query region in the input drawing (step S12). Specifically, the management unit 210 of the control unit 21 outputs the input image temporarily stored in the memory to the display device H13 of the user terminal 10. The user specifies a region (query region) in the output input drawing that includes the facility (object) to be searched for. In this case, the management unit 210 acquires the query image included in the query region specified in the user terminal 10. Here, it is assumed that a query region 501 is designated in an input drawing 500 as shown in FIG. 4(a).

[0023] Next, the control unit 21 of the support server 20 executes a calculation process of the first feature amount of the query image (step S13). Specifically, the feature extraction unit 212 of the control unit 21 calculates the feature amount (first feature amount) of the query image using the pre-learning model recorded in the learning model storage unit 22.

[0024] The control unit 21 of the support server 20 also executes a process for extracting similar image candidates for the query image (step S14). Specifically, the candidate extraction unit 211 of the control unit 21 searches for similar image candidates in the input drawing by template matching using the query image as a template. In this case, the template may be resized, rotated at a predetermined angle, or inverted. Alternatively, contour features of the template image may be extracted and a contour search (geometric shape search) using a model in which the features are vectorized may be performed. In this case, the template features are extracted and vectorized, and similar image candidates are extracted based on the number of matching vectors. The candidate extraction unit 211 then temporarily stores the position of the area in the drawing from which the similar image candidates are extracted.

[0025] Next, the control unit 21 of the support server 20 executes a calculation process of the second feature amount of the similar image candidate (step S15). Specifically, the feature extraction unit 212 of the control unit 21 calculates the feature amount (second feature amount) for the extracted similar image candidate using the pre-learning model recorded in the learning model storage unit 22.

[0026] Next, the control unit 21 of the support server 20 executes a comparison process of the first and second feature amounts (step S16). Specifically, the feature extraction unit 212 of the control unit 21 compares the first feature amount of the query image with the second feature amount of each similar image candidate.

[0027] Next, the control unit 21 of the support server 20 executes a process of narrowing down similar image candidates according to the comparison result (step S17). Specifically, the feature extraction unit 212 of the control unit 21 identifies the region position of a similar region that includes similar image candidates having a second feature amount whose difference with respect to the first feature amount is within a predetermined range. Then, the management unit 210 identifies the identified similar region on the input drawing and outputs it to the display device H13 of the user terminal 10.

[0028] 4(b), a similar area 502 is displayed on the input drawing 500. In this case, the user uses the user terminal 10 to check the similar area 502 and select the facility to be designated.

[0029] (Operation of the embodiment) The feature quantities of the query image and similar image candidates are calculated using a pre-learning model, so similar regions that are similar to the query region are extracted.

[0030] (Effects of the embodiment) According to this embodiment, the following effects can be obtained. (1-1) In this embodiment, the control unit 21 of the support server 20 executes a calculation process of a first feature amount of a query image (step S13). When performing machine learning, a large amount of training information is used for the image to be searched. Because a pre-learning model is used, the feature amount of the query image can be calculated without performing machine learning using the image to be searched as training information.

[0031] (1-2) In this embodiment, the control unit 21 of the support server 20 executes a process of extracting similar image candidates for the query image (step S14). This allows for efficient identification of similar image candidates through pattern matching, which has a low computational load.

[0032] (1-3) In this embodiment, the control unit 21 of the support server 20 executes a process of calculating the second feature amount of the similar image candidate (step S15). This allows the feature amount of the similar image candidate to be calculated without performing machine learning using the image to be searched as training information.

[0033] (1-4) In this embodiment, the control unit 21 of the support server 20 performs a comparison process between the first and second feature amounts (step S16) and a process of narrowing down similar image candidates based on the comparison result (step S17). This makes it possible to efficiently extract similar images corresponding to the query region in the input drawing using pattern matching and a pre-learned model.

[0034] (Second embodiment) Next, a second embodiment of an image search system, an image search method, and an image search program will be described with reference to FIG. 5. In the first embodiment, the control unit 21 of the support server 20 executes a query area designation process in the input drawing (step S12). In this embodiment, related images of the query image are output. In this case, images that may be used in the search are recorded in the related image information storage unit 23. For example, the related image information storage unit 23 is registered with equipment symbols and illustrations that may be used in the drawing. For example, JIS standard symbols and images that have received favorable user feedback from user search results are registered as related images. In the following embodiments, parts similar to those in the first embodiment are designated by the same reference numerals, and detailed descriptions thereof will be omitted.

[0035] As shown in FIG. 5, the control unit 21 of the support server 20 executes an input drawing acquisition process (step S11) and a query region designation process (step S12) in the input drawing. Next, the control unit 21 of the support server 20 executes a process of acquiring related images of the query image (step S21). Specifically, the management unit 210 of the control unit 21 extracts related images similar to the query image included in the query region from the related image information storage unit 23. In this case, it is possible to use a pre-learning model recorded in the learning model storage unit 22. Alternatively, a pattern matching method may be used.

[0036] Next, the control unit 21 of the support server 20 executes a related image suggestion process (step S22). Specifically, the management unit 210 of the control unit 21 outputs the extracted related image to the display device H13 of the user terminal 10 and makes a suggestion to prompt the user to select a related image. Then, when a related image is selected on the user terminal 10, the management unit 210 adds the related image as a query image.

[0037] Next, the control unit 21 of the support server 20 executes a calculation process of the first feature amount of the query image (step S13). Here, the feature extraction unit 212 calculates the first feature amount of each of the query image and related images in the specified query region. Then, in the following process, all the calculated first feature amounts are used.

[0038] Furthermore, the control unit 21 of the support server 20 executes a process of extracting similar image candidates to the query image (step S14) and a process of calculating second feature amounts of the similar image candidates (step S15).

[0039] Next, the control unit 21 of the support server 20 performs a comparison process between the first and second feature amounts (step S16), and a process of narrowing down similar image candidates based on the comparison result (step S17). When multiple first feature amounts are used, the control unit 21 identifies the region position of a similar region that includes similar image candidates having second feature amounts whose difference from any of the first feature amounts is within a predetermined range.

[0040] According to this embodiment, in addition to the effects of the first embodiment, the following effects can be obtained. (2-1) In this embodiment, the control unit 21 of the support server 20 executes a process of acquiring related images of a query image (step S21) and a process of proposing related images (step S22). This allows the desired facilities to be comprehensively identified using not only the query image in the query area specified by the user but also the related images.

[0041] This embodiment can be modified as follows: This embodiment and the following modifications can be combined and implemented within the scope of technical compatibility. In each of the above embodiments, the user terminal 10 and the support server 20 are used, but the hardware configuration is not limited to this. For example, the user terminal 10 and the support server 20 may be realized using a single piece of hardware, or may be realized by cloud computing. In the above embodiments, it is assumed that the same equipment is searched for in a design drawing. The search target is not limited to the design drawing. For example, a query area may be specified in a photographed image of a material or part to search for similar material or part.

[0042] In each of the above embodiments, the control unit 21 of the support server 20 executes a query region specification process in the input drawing (step S12). Here, the query image may be corrected. For example, if there is a part of the query region that is hidden by text or other objects, the image may be corrected to complement the image. In this case, an image generation model is used to complement the image. Also, only the object that is primarily displayed in the query region may be extracted. This allows the user to identify the object desired.

[0043] In each of the above embodiments, a pre-learning model for calculating image features is stored in the learning model storage unit 22. A plurality of different pre-learning models may be prepared and stored in the learning model storage unit 22. In this case, the first and second features are calculated using each pre-learning model, and the difference is calculated for each pre-learning model. The management unit 210 may then narrow down the similar image candidates by majority vote between matches and mismatches.

[0044] In each of the above embodiments, the control unit 21 of the support server 20 executes an input drawing acquisition process (step S11). Here, the input drawing is acquired as an image obtained by scanning a paper drawing. The input drawing is not limited to a scanned image of a paper drawing. For example, a drawing created by CAD (Computer Aided Design) may be used. Also, a PDF exported from a CAD drawing may be used.

[0045] The input drawing may also be composed of multiple layers. In the case of input drawing data composed of multiple layers, similar image candidates may be extracted in the layer in which the query region is specified. Furthermore, when a query region is specified, information in layers that are not specified may be deleted.

[0046] In each of the above embodiments, the control unit 21 of the support server 20 executes a process of specifying a query region in the input drawing (step S12). In addition, a query condition may be acquired as text from the user terminal 10. In this case, the text included in the drawing is extracted by character recognition. Then, similar image candidates may be narrowed down based on whether the text extracted within a predetermined range of the similar image candidates matches or does not match the query condition (text).

[0047] In addition to the similar image candidates narrowed down based on the comparison result, areas extracted by searching using a query condition (text) may also be output to the display device H13 of the user terminal 10.

[0048] In the second embodiment, the control unit 21 of the support server 20 executes a process for acquiring related images of the query image (step S21). In this case, the control unit 21 may learn the related images to be suggested depending on the confirmation result of the user. In this case, the confirmation result is the time required for the user to confirm the search results and the confirmation order. In this case, the control unit 21 of the support server 20 records the time required for the user to confirm the related images and the confirmation order. Then, the control unit 21 gives priority to suggesting related images that require a shorter time or that are confirmed earlier. [Explanation of symbols]

[0049] 10...user terminal, 20...support server, 21...control unit, 210...management unit, 211...candidate extraction unit, 212...feature extraction unit, 22...learning model storage unit, 501...query area.

Claims

1. An image retrieval system comprising a control unit connected to a user device, The control unit acquiring a query image in a query region specified by the user device in an input image; Identifying similar image candidates similar to the query image in the input image by template matching; Calculating the feature amounts of the query image and the feature amounts of the similar image candidates using a pre-learning model generated by machine learning to calculate the feature amounts of vectorized general-purpose images; An image search system characterized in that, according to the comparison result of the feature amounts, a similar image corresponding to the query image is identified and output to the user device.

2. The control unit obtaining related images of the query image; The image retrieval system of claim 1 , wherein the related images are added to the query image.

3. The control unit outputting the related image to the user device; The image retrieval system of claim 2 , wherein related images selected on the user device are added to the query image.

4. The control unit Obtaining a confirmation result of the similar image output to the user device; The image search system according to claim 2 , wherein the related images are identified in accordance with the confirmation result.

5. 1. A method for retrieving similar images using an image retrieval system having a controller connected to a user device, comprising: The control unit acquiring a query image in a query region specified by the user device in an input image; Identifying similar image candidates similar to the query image in the input image by template matching; Calculating the feature amounts of the query image and the feature amounts of the similar image candidates using a pre-learning model generated by machine learning to calculate the feature amounts of vectorized general-purpose images; An image search method comprising: identifying a similar image corresponding to the query image according to a comparison result of the feature amounts; and outputting the identified similar image to the user device.

6. A program for searching for similar images using an image search system having a control unit connected to a user device, The control unit acquiring a query image in a query region specified by the user device in an input image; Identifying similar image candidates similar to the query image in the input image by template matching; Calculating the feature amounts of the query image and the feature amounts of the similar image candidates using a pre-learning model generated by machine learning to calculate the feature amounts of vectorized general-purpose images; an image search program for causing the program to function as a means for identifying similar images corresponding to the query image according to the comparison result of the feature amounts and outputting the images to the user device;

Images