Information processing system, control method for information processing system, program, recording medium

Abstract

There was a concern that it would be impossible to search for similar images if the contour shape was drawn with lines. [Solution] A first image generated from the search image and a second image, which is a silhouette image generated from the search image, are obtained. The similarity between the search image and the multiple registered images is calculated by comparing the obtained first image and the second image with each of the multiple registered images. By calculating in this way, similar images can be searched with high reliability even for images with outline shapes drawn with lines.

Description

Technical Field

[0001] The present invention relates to an information processing system, a control method of an information processing system, and the like.

Background Art

[0002] Image search for retrieving similar images from images accumulated in the past is carried out in various fields. Such image search can be considered applicable to fields such as design and manufacture of members such as molds. In the fields of design and manufacture of members such as molds, when estimating the cost and labor involved in mold manufacturing, the actual results of past similar cases are referred to, but the determination of similarity is often carried out manually using drawings of past members. However, since high skills and experience are required, it is difficult for those who are not skilled.

[0003] Therefore, various mechanisms have been proposed for accurately retrieving similar images from the accumulated image group using an image as a search query. In Patent Document 1, first hash values and second hash values having different lengths are calculated for registered images and query images, respectively, and similar images are retrieved by comparing the hash values of the registered images and the search query images.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, in design drawings of members such as molds, a contour shape having concavities and convexities drawn with lines is used, but there is a concern that an image having such a contour shape cannot be correctly identified as a similar image by the method disclosed in Patent Document 1.

[0006] Therefore, the present invention aims to provide a highly reliable similar image search method that can search for similar images even if the image has a contour shape drawn with lines. [Means for solving the problem]

[0007] To solve the above problems, one aspect of the information processing system according to the present invention is characterized by having an acquisition unit that acquires a first image generated from a search image and a second image which is a silhouette image generated from the search image, and a calculation unit that calculates the similarity between the search image and the plurality of registered images by comparing the first image and the second image acquired by the acquisition unit with each of the plurality of registered images. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a highly reliable similar image search method that can search for similar images even if the image has a contour shape drawn with lines. [Brief explanation of the drawing]

[0009] [Figure 1] (a) A diagram showing the configuration of an information processing system equipped with an image search server according to an embodiment of the present invention. (b) A diagram showing the hardware configuration according to an embodiment of the present invention. [Figure 2] This is a flowchart illustrating the registration process flow of the information processing system according to the first embodiment. [Figure 3] (a) A diagram illustrating the necessity of silhouette images. (b) A diagram illustrating image processing that creates resized images and silhouette images from input images. [Figure 4] (a) This diagram illustrates the method for extracting images used for image registration. (b) This diagram illustrates the registration process. [Figure 5] (a) An example of a data table for registering feature quantities of resized images. (b) An example of a data table for registering feature quantities of silhouette images. [Figure 6]This figure shows the search process flow for similar figures in the information processing system according to the first embodiment. [Figure 7] This figure shows the search process flow for similar figures in the information processing system according to the first embodiment. [Figure 8] (a) This figure shows a case where the resized image is more important than the silhouette image. (b) This is an example of a data table used for weighting. [Figure 9] (a) A diagram illustrating the extraction of images used in the search process. (b) A diagram illustrating the search process. [Figure 10] This is an example of a data table in which the distance to the searched image is registered. [Figure 11] This is an example of a data table sorted by similarity to the searched drawings for registered drawings. [Figure 12] (a) An example of a search result drawing. (b) An example of a method for displaying search results identified in the search process. [Figure 13] (a) A diagram illustrating the method for determining the weight coefficients according to the second embodiment. (b) An example of a data table used to determine the weight coefficients according to the second embodiment. [Modes for carrying out the invention]

[0010] With reference to the drawings, an information processing apparatus, an information processing method, etc., according to an embodiment of the present invention will be described. The embodiments shown below are illustrative, and for example, those skilled in the art can modify the detailed configuration as appropriate without departing from the spirit of the present invention.

[0011] In the drawings referenced in the following descriptions of embodiments and examples, elements denoted by the same reference numeral have the same function unless otherwise specified. In cases where multiple identical elements are shown in a drawing, the designation and description of the element may be omitted.

[0012] In addition, for the sake of convenience in illustration and description, the drawings may be represented schematically. Therefore, the shapes, sizes, arrangements, etc. of the elements shown in the drawings may not necessarily exactly match the actual objects.

[0013] [First Embodiment] (Information Processing System) Fig. 1(a) shows the configuration of an information processing system including an information processing apparatus according to this embodiment. The information processing system includes a client terminal 10, an image search server 100, a storage unit 110, and a drawing DB 120.

[0014] The drawing DB 120 can store drawings of members such as molds manufactured in the past, design information corresponding to the drawings, cost information, etc. In this embodiment, the drawings stored in the drawing DB 120 are used as registered drawings. In this embodiment, a drawing of a mold used for plastic injection molding is stored in the drawing DB, and a case of extracting a similar drawing similar to a search drawing from the drawing DB will be described as an example. The drawing includes a plan view of the mold viewed from a plurality of directions, and image search is performed using such a plan view.

[0015] The storage unit 110 is provided with a resized image feature amount storage unit 111 and a silhouette image feature amount storage unit 112 that are referred to during search processing.

[0016] The client terminal 10 is a terminal operated by a user, and can perform operations such as a drawing display instruction, an image cut-out instruction, a result display, and a similar drawing display.

[0017] The image search server 100 functions as a registration processing unit and a search processing unit, and includes functional block units such as an image acquisition unit 101, an image processing unit 102, a feature amount extraction unit 103, a feature amount registration unit 104, and a similarity calculation unit 105.

[0018] The image acquisition unit 101 is an acquisition unit that acquires the image cut-out instructed by the client terminal 10.

[0019] The image processing unit 102 is a processing unit that generates a resized image and a silhouette image from the acquired image.

[0020] The feature extraction unit 103 extracts features from the generated resized image and the silhouette image, respectively.

[0021] The feature registration unit 104 stores each extracted feature as index information in the resized image feature storage unit 111 and the silhouette image feature storage unit 112 of the storage unit 110.

[0022] The similarity calculation unit 105 uses the information stored in the storage unit 110 and the information extracted from the search image by the feature extraction unit 103 to calculate (determine the similarity) the similarity between the search image / drawing and the registered image / drawing.

[0023] Figure 1(b) shows an example of the hardware configuration of the client terminal 10 and the image search server 100. The CPU 201 comprehensively controls each device and controller connected to the system bus 204.

[0024] Furthermore, the ROM or external memory stores the BIOS (Basic Input / Output System), which is the control program for the CPU201, the operating system program (hereinafter referred to as OS), and various programs necessary to realize each function. RAM203 functions as the main memory and work area of ​​the CPU201.

[0025] The CPU 201 loads the necessary programs and other data into the RAM 203 for processing and then executes those programs to perform various operations.

[0026] Furthermore, the input controller (input C) 205 controls input from input devices 209, such as a keyboard or a pointing device like a mouse (not shown).

[0027] The video controller (VC) 206 controls the display on display terminals such as the display 210. The display terminals are assumed to be CRTs or LCDs, but are not limited to these.

[0028] The memory controller (MC) 207 controls access to external memory 211, which is connected via an adapter to a hard disk (HD), flexible disk (FD), or PCMCIA card slot, and stores boot programs, browser software, various applications, font data, user files, editing files, and other data.

[0029] The communication interface controller (communication interface) 208 connects to and communicates with external devices via the network and performs communication control processing over the network. For example, it can handle internet communication using TCP / IP.

[0030] Furthermore, the CPU 201 enables display on the display 210 by, for example, performing the process of expanding (rasterizing) outline fonts into the display information area in RAM 203.

[0031] Furthermore, the CPU 201 enables user input via a mouse cursor (not shown) on the display 210.

[0032] In the client terminal 10 and image search server 100 of the present invention, various programs used to perform various processes are recorded in external memory 211 and, when necessary, loaded into RAM 203 and executed by CPU 201. Furthermore, definition files and various information tables used by the programs related to the present invention can also be stored in external memory 211.

[0033] (Feature registration process) Next, the flowchart in Figure 2 will be used to explain the flow of feature registration processing by the information processing system. Each process shown in the flowchart is achieved by reading and executing the control program stored in the CUP201.

[0034] In S201, the CPU 201 of the client terminal 10 displays a drawing (registered drawing) stored in the drawing database. In S202, the CPU 201 of the client terminal 10 receives an image cropping instruction from the user on the displayed drawing, and then sends the information of the cropped image to the image search server 100. Here, the user can receive instructions to crop multiple images from a single drawing. Regarding image cropping, the user may manually specify a range as described above, or automatic shape extraction may be performed using machine learning. In this embodiment, an example of performing the extraction work on the client terminal 10 is used for explanation, but the extraction work may also be performed on the server.

[0035] In S203, the CPU 201 of the image search server 100 retrieves the cropped image (registered image) obtained from the client terminal 10.

[0036] In S204, CPU 201 of image search server 100 creates resized images and silhouette images used to extract features from cropped images (registered images). Since the aspect ratio of the cropped images is not constant, a resizing process is performed to achieve predetermined vertical and horizontal dimensions so that the vertical and horizontal resolutions are constant, thereby generating resized images. In other words, depending on the size of the cropped image, resizing processes with different magnifications for vertical and horizontal dimensions may be performed. The reason for performing this resizing process is to extract features of the same size for all images in the subsequent feature extraction process. That is, by adjusting the vertical and horizontal dimensions of the images in advance so that the features are of the same size, the similarity of the images can be determined by comparing the features. Note that if the cropped images have been cropped to the predetermined vertical and horizontal dimensions, resizing is not necessary. The silhouette image is an image generated by filling in the inside of the outermost contour of the cropped image.

[0037] Here, we will explain the necessity of silhouette images using Figure 3(a). Images 31 and 33 are examples of contour shapes drawn only with lines, and at first glance they may not appear similar. However, depending on the type of similarity determination method, it may not be possible to correctly determine the irregularities of the contour shapes of images 31 and 33, and they may be judged as similar. On the other hand, the silhouette image 32 generated from image 31 and the silhouette image 34 generated from image 33 clearly show the inside of the contour shape, and the irregularities of the overall shape can be correctly determined, so they can be judged as not similar even with the above method. However, in the case of images with distinctive shapes on the inside, similarity cannot be correctly determined using only silhouette images. Therefore, in this embodiment, similar drawings are correctly identified by using both a resized image and a silhouette image generated from the same image.

[0038] Figure 3(b) is a diagram illustrating the image generation process performed in S204. When the image acquisition unit 101 acquires the registered image 35, the image processing unit 102 can generate a resized image 36, which is obtained by image processing the registered image 35 and resizing it to a predetermined size, and a silhouette image 37, which is a silhouette of the registered image 35.

[0039] In S205, the CPU 201 of the image search server 100 extracts features from the resized image and the silhouette image, respectively (creates features). Features are numerical representations of the characteristics of the target image, and in this embodiment, a one-dimensional numerical array can be used.

[0040] In S206, the CPU 201 of the image search server 100 stores the feature quantities generated from the resized images extracted in S205 in the resized image feature quantity storage unit 111, and further stores the feature quantities generated from the silhouette images in the silhouette image feature quantity storage unit 112. If multiple cropped images are obtained in S203, the processing from S204 to S206 is repeated until the feature quantities of all images are registered.

[0041] Figure 5(a) shows an example of a data table when stored in the resized image feature storage unit 111. The drawing ID 501, which is the source from which the registered image 35 was extracted, the image ID 502 of the registered image 35, and the resized image feature 503 extracted from the image with that image ID are registered.

[0042] Figure 5(b) shows an example of a data table when the data is stored in the silhouette image feature storage unit 112. The drawing ID 501, which is the source from which the registered image 35 was extracted, the image ID 502 of the registered image 35, and the silhouette image feature 513 extracted from the image with that image ID are registered.

[0043] Figure 4(a) illustrates the method for extracting images used for image registration performed in S202. For example, a user can extract a registered image 43 by displaying a drawing 41 stored in the drawing DB on the display terminal of the client terminal 10 and specifying a selection area 42. Next, Figure 4(b) illustrates the flow of the feature quantity registration process performed from S204 to S206. The resized image 401 is an image obtained by resizing the registered image 400 to a resolution of 224 x 224 pixels. The silhouette image 411 is a silhouette image obtained by resizing the registered image 400 to a resolution of 224 x 224 pixels while maintaining its aspect ratio and filling it with the outermost contour. Features can be extracted using the trained models 402 and 412. As the trained models used, known neural networks that are multi-layered CNNs (Convolutional Neural Networks) such as VGG and ResNet can be used.

[0044] The feature vectors 403 extracted from the resized image 401 and 413 extracted from the silhouette image are values ​​output from the trained models 402 and 412, and are output, for example, as one-dimensional numerical arrays. Feature vector values ​​can be registered separately for the resized image and the silhouette image. Since weighting the resized image generally leads to better similarity detection, the feature vector array for the resized image can be set to 1024, and the feature vector array for the silhouette image can be set to 512.

[0045] By pre-storing feature quantities corresponding to a large number of image IDs in the resized image feature quantity storage unit 111 and the silhouette image feature quantity storage unit 112, the search process for similar figures described later can be performed efficiently. In this embodiment, an example was described using a drawing that has been registered in the drawing DB in advance, but the feature quantities of images included in the drawing may be automatically registered in the storage unit 110 when the drawing is registered in the drawing DB.

[0046] (Search process for similar shapes) Next, using the flowcharts in Figures 6 and 7, we will explain the search process for finding registered figures similar to the search target drawing from among the drawings stored in the drawing database by the information processing system. Each process shown in the flowchart is realized by reading and executing the control program stored in the CUP201. Some processes are the same as those in the flowchart in Figure 2, so their explanation will be omitted.

[0047] In S601, the CPU 201 of the client terminal 10 displays the search diagram. In S602, the CPU 201 of the client terminal 10 receives an image cropping instruction from the user on the displayed diagram and then sends the information of the cropped image to the image search server 100. Here, the user can receive instructions to crop multiple images from a single diagram. Figure 12(a) is a diagram illustrating the method of cropping search images used in the search process. The user can extract search images 1204 and 1205 by displaying the search diagram 1201 on the screen of the client terminal 10 and specifying the selection ranges 1202 and 1203. In S603, the CPU 201 of the image search server 100 obtains the cropped images (search images) acquired from the client terminal 10.

[0048] In S604, the CPU 201 of the image search server 100 creates a resized image (first image) and a silhouette image (second image) to be used to extract features from the cropped image (search image). In S605, the CPU 201 of the image search server 100 extracts features from the resized image (first image) and the silhouette image (second image), respectively (creates features). The details of the generation of the resized image and silhouette image and the feature extraction process are the same as in the registration process in Figure 2, so the explanation is omitted, but the resolution of the resized image during the search must match the resolution of the resized image during the registration process.

[0049] In S606, the CPU 201 of the image search server 100 uses the feature quantities of the search image extracted in S605 and the feature quantities stored in the storage unit 110 to calculate the similarity between the search image and the registered image. Methods for calculating the similarity between feature quantities include Euclidean distance and cosine similarity. In this embodiment, the method for evaluating the similarity between feature quantities will be explained below using Euclidean distance. The smaller the Euclidean distance value obtained here, the more similar the images are considered to be; conversely, the larger the distance value, the less similar the images are considered to be.

[0050] Next, we will explain the details of S606 using the subflowchart in Figure 7.

[0051] In S701, the CPU 201 of the image search server 100 calculates a distance (distance calculation process) that indicates the similarity between the feature quantities of the resized images extracted in S605 and the feature quantities of the resized images stored in the resized image feature quantity storage unit 111. This distance is calculated for each feature quantity of the resized images registered in the storage unit 110, that is, for all resized images that have undergone the registration process in advance.

[0052] In S702, the CPU 201 of the image search server 100 performs weighting (weighting) processing on the distance calculated in S701.

[0053] In S711, the CPU 201 of the image search server 100 calculates the distance between the feature quantities of the silhouette images extracted in S605 and the feature quantities of the silhouette images stored in the silhouette image feature quantity storage unit 112. This distance is calculated for each silhouette image feature quantity registered in the storage unit 110, that is, for all silhouette images that have undergone prior registration processing.

[0054] In S712, the CPU 201 of the image search server 100 performs weighting (weighting) processing on the distance calculated in S711.

[0055] Here, we will explain the details of the weighting process performed in S702 and S712. In this embodiment, features are extracted from silhouette images so that even images drawn only with lines can be correctly judged for their contours. However, depending on the shape of the lines, there are cases where the features of the resized image need to be given more weight than the features of the silhouette image when determining similarity. An example of this is explained with reference to Figure 8(a). Figure 8(a) shows a resized image 81 generated from the registered image at the time of registration, a silhouette image 82 created from the same registered image, a resized image 83 generated from the search image, and a silhouette image 84 generated from the same search image. Resized image 81 and resized image 83 are similar in their internal shapes, but silhouette image 82 and silhouette image 84 have different external shapes. Therefore, if features from the silhouette image are given more weight, it is difficult to determine that the search image is similar to the registered image.

[0056] Therefore, it is preferable to focus on the area of ​​the silhouette image and the complexity of the resized image, and decide which of the features of the resized image or the silhouette image to prioritize depending on the situation. In other words, by multiplying the similarity calculated from the features by a weighting coefficient corresponding to the shape characteristics of the search image, adjustments are made so that images that should be given more weight have a greater influence on the similarity determination.

[0057] Specifically, the adjustment can be made by preparing a data table 800 for determining the weight characteristics as shown in Figure 8(b) in a storage unit such as the storage unit 110 beforehand. This data table 800 consists of the area ratio 801 occupied by the silhouette region, a weight coefficient 802 multiplied by the distance of the resized image, and a weight coefficient 803 multiplied by the distance of the silhouette image. Here, the silhouette region is the filled area of ​​the silhouette image generated from the search image in S607, and the silhouette area ratio indicates the ratio of the area of ​​the silhouette region to the total area.

[0058] The reason for determining weighting coefficients according to the silhouette area is that for silhouette images where the silhouette region occupies a large proportion, the features of the outermost contour of the silhouette become weaker, and the resized image is more likely to be important as a feature. Therefore, in the example of data table 800, the weighting coefficient of the resized image is set to be larger when the silhouette area proportion is small. On the other hand, the weighting coefficient of the silhouette image is set to be larger when the silhouette area proportion is large. In other words, the larger the silhouette area proportion, the smaller the weighting coefficient of the resized image becomes, and the larger the weighting coefficient of the silhouette image becomes.

[0059] When calculating distance using resized images and silhouette images in this way, the search accuracy can be improved by weighting the distance based on the area of ​​the silhouette image. When the search accuracy with and without silhouette images was evaluated using the average precision, it was found that using silhouette images improved the search accuracy by 10%, and using area-based weighting improved it by another 13%.

[0060] In other words, in S702, the CPU 201 of the image search server 100 calculates the silhouette area ratio based on the silhouette image of the search image, and then determines the respective weight coefficients by referring to the data table 800 based on that area ratio. Then, the distance obtained from the features of the resized image is multiplied by the weight coefficient 802 for the resized image to calculate the adjusted distance.

[0061] The calculated adjustment distance can be temporarily stored in the memory unit using a data table 1000 as shown in Figure 10. The data table 100 consists of a drawing ID 1001 for the registered drawing, an image ID 1002 for the registered image, and a distance 1003 between the registered image and the search image. The drawing ID 1001 indicates a unique number for each registered drawing, the image ID 1002 indicates a unique number for each registered image, and the distance 1003 to the search image is an adjustment distance obtained by weighting the distance calculated from the feature quantities of the search image and the feature quantities of the resized image using a weighting coefficient.

[0062] Similarly, in S712, the distance obtained from the features of the silhouette image is multiplied by a weighting coefficient of 803 for silhouette images to calculate the adjusted distance. The adjusted distance calculated for the silhouette image can also be temporarily stored using a data table similar to that in Figure 10. In S703, the CPU 201 of the image search server 100 adds up the adjusted distance for the resized image and the adjusted distance for the silhouette image to calculate the total distance for all registered images that have undergone registration processing. Note that the calculation method at this time may use multiplication as well as addition. The total distance can be said to indicate the similarity between the search image and the registered image, and the registered image with the shortest total distance is said to be the image that is most similar to the search image.

[0063] If multiple search images are obtained in S603, the processes from S604 to S606 are repeated for each search image.

[0064] In S607, the CPU 201 of the image search server 100 calculates the total drawing distance by summing the distances between each of the multiple search images included in the search drawing and the closest registered image included in that drawing ID for each registered drawing ID. It then transmits this calculation result to the client terminal 10. In other words, it calculates the similarity between each registered drawing ID and the search drawing. Figure 11 shows an example of a data table 1100 sorted in descending order of similarity (similarity order) based on the calculated similarity. The data table 1100 consists of the similarity order 1101, the registered drawing ID 1102, and the total drawing distance 1103.

[0065] In S609, the CPU 201 of the client terminal 10 receives similar search results from the image search server 100 and displays them on a display terminal such as the display 210. The display method may include displaying a data table 1100 capable of determining similarity, as shown in Figure 11, or displaying registered drawings in order of similarity.

[0066] In S610, the CPU 201 of the client terminal 10 receives a specification from the user on a display terminal that displays information capable of determining similarity, and displays detailed information of the registered drawing specified in S611 on the display terminal. Detailed information may also be obtained and displayed from the drawing DB 120. Alternatively, the display terminal of the client terminal 10 may initially display the registered drawing with the closest similarity without receiving a specification from the user. In this embodiment, an example of displaying similar drawings as search results has been used, but similar registered images may be displayed for each searched image.

[0067] Figure 12(b) shows an example of how registered drawings are displayed on the display screen 1210 in order of similarity. The data 1211 of the displayed registered drawing includes the image of the registered drawing 1212, the information 1213 from the data table 1100, and a display button 1214 for selecting the registered drawing 1212. By pressing the display button 121, the system accepts the user's selection and displays the registered drawing data 1211 and detailed information in an enlarged view.

[0068] Next, we will explain the flow of the similarity calculation process performed from S604 to S606 using Figure 9(b). The resized image 901 is an image obtained by resizing the search image 900 to a resolution of 224 x 224 pixels. The silhouette image 911 is a silhouette image obtained by resizing the search image 900 to a resolution of 224 x 224 pixels while maintaining the aspect ratio, and filling it with the outermost contour. Features can be extracted using the trained models 902 and 912. As for the trained models used, known neural networks that are multi-layered CNNs (Convolutional Neural Networks) such as VGG16 and ResNet can be used. Note that the trained model used when extracting features must be the same model used during the registration process.

[0069] The feature vectors 903 extracted from the resized image 901 and 913 extracted from the silhouette image are values ​​output from the trained models 902 and 912, and are output, for example, as one-dimensional numerical arrays. Feature vector values ​​can be registered separately for the resized image and the silhouette image. As with extracting features from registered images, the array of features for the resized image obtained from the search image must be set to 1024, and the array of features for the silhouette image must be set to 512.

[0070] Next, a distance 905 indicating the similarity between the resized image features 903 and the resized image features stored in the resized image feature storage unit 111 is calculated (S701). Furthermore, a weighting process is performed on the distance 905 to calculate an adjusted distance 907 (S702).

[0071] Furthermore, a distance 915 indicating the similarity between the silhouette image features 913 and the silhouette image features stored in the silhouette image feature storage unit 112 is calculated (S711). A weighting process is then applied to the distance 915 to calculate an adjusted distance 917 (S712).

[0072] Furthermore, since the distance calculation process using feature quantities takes more time as the number of registered images in the storage unit 111 increases, an approximate nearest neighbor search algorithm can be used to speed up the calculation.

[0073] Subsequently, the adjustment distance 907 for the resized image and the adjustment distance 917 for the silhouette image are added together to calculate the total distance 921 for all registered images that have undergone the registration process. In this embodiment, the weighting process was described in which weighting was applied to both the feature quantities of the silhouette image and the feature quantities of the resized image, but the weighting process may be performed on either the silhouette image or the resized image, or both.

[0074] By using the mechanism described above, it is possible to reliably search for drawings / images similar to the searched drawing / image from among the registered drawings / images already created and saved in the drawing database.

[0075] [Second Embodiment] In the first embodiment, a method was described in which a weighting coefficient was determined by the area of ​​the silhouette image, and the search accuracy was improved by weighting the distance with this weighting coefficient. In this embodiment, instead of the area of ​​the silhouette image, the complexity of the internal shape of the resized image is considered, a weighting coefficient is determined according to the internal shape, and the search accuracy is improved by weighting the distance with this weighting coefficient. The following description will focus on the differences from the first embodiment.

[0076] Figure 13(a) is an example of a resized image with a resolution of 224 x 224 pixels and a complex internal shape, generated from a search image. Image 1302 is a diagram of resized image 1301 divided into a 32 x 32 pixel grid. The weight coefficient can be determined by the number of grids containing the line shapes of the resized image. In the example of image 1302 in Figure 13(a), there are 3 blank grids that do not contain any line shapes from the resized image. Image 1312 is a diagram of resized image 1311 divided into a 32 x 32 pixel grid. In the example of image 1312, there are 31 blank grids that do not contain any line shapes from the resized image.

[0077] The fewer the number of blank grids, the more important the interior of the resized image becomes. Therefore, the fewer the number of blank grids, the larger the weight coefficient applied to the features of the resized image. Conversely, the fewer the number of blank grids, the smaller the weight coefficient applied to the features of the silhouette image. Such weight coefficients can be prepared in advance in a storage unit such as the storage unit 110 as a data table 1320, as shown in Figure 13(b). The data table 1320 consists of the number of blank grids 1321, a weight coefficient 1322 applied to the distance of the resized image, and a weight coefficient 1323 applied to the distance of the silhouette image.

[0078] In the example of Image 1301, there are 3 blank grids, so from Data Table 1320, we can see that the weighting coefficient multiplied by the distance of the resized image is 0.7 and the weighting coefficient multiplied by the distance of the silhouette image is 0.3. Also, in the example of Image 1311, there are 31 blank grids, so we can see that the weighting coefficient multiplied by the distance of the resized image is 0.5 and the weighting coefficient multiplied by the distance of the silhouette image is 0.5.

[0079] Using the weight coefficients calculated in this way, the weighting processes S702 and S712 in Figure 7 are performed. The other processes are the same as in the first embodiment, so their explanation is omitted.

[0080] By using the weighting coefficients of this embodiment, it is possible to reliably search for drawings / images similar to the searched drawing / image from among the registered drawings / images in the drawing database.

[0081] [Modified version of the second embodiment] In the second embodiment, a weighting coefficient was determined according to the number of grids containing the linear shape of the resized image. However, the weighting coefficient may also be determined by focusing on the length of the line segments of the linear shape of the resized image. The longer the line segments of the resized image, the more complex the interior of the image becomes, and therefore the interior of the resized image becomes more important. For this reason, the longer the line segments, the larger the weighting coefficient applied to the features of the resized image. On the other hand, the longer the line segments, the smaller the weighting coefficient applied to the features of the silhouette image. Using the weighting coefficients set in this way, the weighting processes S702 and S712 in Figure 7 are performed. The other processes are the same as in the first embodiment, so their explanation is omitted.

[0082] By using the weighting coefficients of this modified version, it is possible to reliably search for drawings / images similar to the searched drawing / image from among the registered drawings / images in the drawing database.

[0083] The present invention, using the embodiments described above, can also be implemented as, for example, a system, apparatus, method, program, or storage medium. Specifically, it may be applied to a system consisting of multiple devices, or to an apparatus consisting of a single device.

[0084] Furthermore, the present invention includes supplying software programs that realize the functions of the embodiments described above, either directly or remotely, to a system or device. The present invention also includes cases where the functions are achieved by an information processing device of the system or device reading and executing the supplied program code.

[0085] Therefore, in order to realize the functional processing of the present invention in an information processing device, the program code installed in the information processing device itself also realizes the present invention. In other words, the present invention also includes the computer program itself for realizing the functional processing of the present invention.

[0086] In that case, as long as it has the functionality of a program, it may take the form of object code, a program executed by an interpreter, script data supplied to the OS, etc.

[0087] Recording media for supplying programs include, for example, flexible disks, hard disks, optical disks, magneto-optical disks, MO disks, CD-ROMs, CD-Rs, and CD-RWs. Other options include magnetic tape, non-volatile memory cards, ROMs, and DVDs (DVD-ROMs, DVD-Rs).

[0088] In addition, the program can also be supplied by connecting to an internet homepage using a browser on a client computer, and then downloading the computer program itself, or a compressed file including an automatic installation function, from the homepage to a recording medium such as a hard disk.

[0089] Furthermore, this can also be achieved by dividing the program code constituting the program of the present invention into multiple files and having each file downloaded from a different homepage. In other words, a WWW server that allows multiple users to download program files for implementing the functional processing of the present invention on an information processing device is also included in the present invention.

[0090] Furthermore, the present invention can also be implemented by encrypting the program, storing it on a storage medium such as a CD-ROM, distributing it to users, and allowing users who meet predetermined conditions to download decryption key information from a website via the internet. Then, the encrypted program can be executed using the downloaded key information and installed on an information processing device.

[0091] Furthermore, the functions of the aforementioned embodiment are realized when the information processing device executes the program it has read. In addition, the OS or other operating system running on the information processing device may perform some or all of the actual processing based on the instructions of that program, and the functions of the aforementioned embodiment can also be realized through that processing.

[0092] Furthermore, the program read from the recording medium is written to the memory of a function expansion board inserted into the information processing device or a function expansion unit connected to the information processing device. Subsequently, based on the instructions of that program, the CPU and other components of the function expansion board or function expansion unit perform some or all of the actual processing, and the functions of the embodiment described above are realized through this processing as well.

[0093] The embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0094] <Summary of Embodiments> This embodiment includes the following configuration.

[0095] (Item 1) An acquisition unit that acquires a first image generated from a search image and a second image which is a silhouette image generated from the search image, An information processing system characterized by having a calculation unit that calculates the similarity between the searched image and the plurality of registered images by comparing the first image and the second image acquired by the acquisition unit with each of the plurality of registered images.

[0096] (Item 2) The information processing system according to item 1, characterized in that the similarity is calculated based on the similarity between the first image and each of the plurality of registered images, and the similarity between the second image and each of the plurality of registered images.

[0097] (Item 3) The information processing system according to item 1 or 2, characterized in that the first image is an image obtained by resizing the search image to a predetermined size.

[0098] (Item 4) The information processing system according to item 3, characterized in that the first image is an image resized by different magnifications in the vertical and horizontal directions.

[0099] (Item 5) The aforementioned search image is an image identified by the user from the search drawing. The information processing system according to any one of items 1 to 4, characterized in that the calculation unit calculates the similarity of registered drawings that are similar to the searched drawing.

[0100] (Item 6) The similarity between the first image and each of the multiple registered images is calculated based on the feature quantities extracted from the first image and the feature quantities corresponding to each of the multiple registered images that have been stored in advance. The information processing system according to any one of items 2 to 5, characterized in that the similarity between the second image and each of the plurality of registered images is calculated based on the feature quantities extracted from the second image and the feature quantities corresponding to each of the plurality of registered images that are stored in advance.

[0101] (Item 7) The information processing system according to item 6, characterized in that the similarity between the first image and each of the multiple registered images is calculated based on a weighted value obtained from the similarity derived from the feature quantities extracted from the first image and the feature quantities corresponding to each of the multiple registered images that are stored in advance.

[0102] (Item 8) The information processing system according to item 6 or 7, characterized in that the similarity between the second image and each of the plurality of registered images is determined based on a weighted value obtained from the similarity derived from the feature quantities extracted from the first image and the feature quantities corresponding to each of the plurality of registered images that are stored in advance.

[0103] (Item 9) The information processing system according to item 8, characterized in that the weighting is performed using weight coefficients determined by the second image.

[0104] (Item 10) The information processing system according to item 8, characterized in that the weighting is performed using weight coefficients determined by the first image.

[0105] (Item 11) The information processing system according to any one of items 1 to 10, characterized in that the silhouette image is an image generated by filling in the inside of the outermost contour included in the search image.

[0106] (Item 12) A method for controlling an information processing system, In the aforementioned information processing system, An acquisition step of obtaining a first image generated from a search image and a second image which is a silhouette image generated from the search image, A calculation step which involves comparing the first image and the second image obtained in the acquisition step with each of the multiple registered images to calculate the similarity between the search image and the multiple registered images, A control method for an information processing system, characterized by controlling the system so that the following occurs.

[0107] (Item 13) A program that can be executed on an information processing system, The aforementioned information processing system An acquisition unit that acquires a first image generated from a search image and a second image which is a silhouette image generated from the search image, A program that functions as a determination unit that determines the similarity between the searched image and the multiple registered images by comparing the first image and the second image acquired by the acquisition unit with each of the multiple registered images.

[0108] (Item 14) A recording medium readable by an information processing system, containing the program described in item 13. [Explanation of Symbols]

[0109] 10 client terminals 100 Image Search Servers 110 Storage Unit 120 Drawing Database

Claims

1. An acquisition unit that acquires a first image generated from a search image and a second image which is a silhouette image generated from the search image, An information processing system characterized by having a calculation unit that calculates the similarity between the searched image and the plurality of registered images by comparing the first image and the second image acquired by the acquisition unit with each of the plurality of registered images.

2. The information processing system according to claim 1, characterized in that the similarity is determined based on the similarity between the first image and each of the plurality of registered images, and the similarity between the second image and each of the plurality of registered images.

3. The information processing system according to claim 1, characterized in that the first image is an image obtained by resizing the search image to a predetermined size.

4. The information processing system according to claim 3, characterized in that the first image is an image resized by different magnifications in the vertical and horizontal directions.

5. The aforementioned search image is an image identified by the user from the search drawing. The information processing system according to claim 1, characterized in that the calculation unit calculates the similarity of registered drawings that are similar to the searched drawing.

6. The similarity between the first image and each of the multiple registered images is calculated based on the feature quantities extracted from the first image and the feature quantities corresponding to each of the multiple registered images that have been stored in advance. The information processing system according to claim 2, characterized in that the similarity between the second image and each of the plurality of registered images is calculated based on the feature quantities extracted from the second image and the feature quantities corresponding to each of the plurality of registered images that are stored in advance.

7. The information processing system according to claim 6, characterized in that the similarity between the first image and each of the plurality of registered images is calculated based on a weighted value obtained from the similarity derived from the feature quantities extracted from the first image and the feature quantities corresponding to each of the plurality of registered images that are stored in advance.

8. The information processing system according to claim 6 or 7, characterized in that the similarity between the second image and each of the plurality of registered images is determined based on a weighted value obtained from the similarity derived from the feature quantities extracted from the first image and the feature quantities corresponding to each of the plurality of registered images that are stored in advance.

9. The information processing system according to claim 8, characterized in that the weighting is performed using weight coefficients determined by the second image.

10. The information processing system according to claim 8, characterized in that the weighting is performed using weight coefficients determined by the first image.

11. The information processing system according to claim 1, characterized in that the silhouette image is an image generated by filling in the inside of the outermost contour included in the search image.

12. A method for controlling an information processing system, In the aforementioned information processing system, An acquisition step of obtaining a first image generated from the search image and a second image which is a silhouette image generated from the search image, A calculation step which involves comparing the first image and the second image obtained in the acquisition step with each of the multiple registered images to calculate the similarity between the search image and the multiple registered images, A control method for an information processing system, characterized by controlling the system so that the following occurs.

13. A program that can be executed on an information processing system, The aforementioned information processing system An acquisition unit that acquires a first image generated from a search image and a second image which is a silhouette image generated from the search image, A program that functions as a calculation unit that calculates the similarity between the searched image and the multiple registered images by comparing the first image and the second image acquired by the acquisition unit with each of the multiple registered images.

14. A recording medium that has the program described in claim 13 recorded on it and is readable by an information processing system.

Images