Drawing search system

Abstract

This system provides a drawing search function that takes handwritten sketches or simple drawings created with 2D CAD as input data and searches for and outputs drawings of the same or similar objects with greater accuracy than conventional methods. [Solution] The drawing search system 1 comprises a parametric CAD generation model that outputs geometric features 12 of an input image, and a vision transformer model that outputs visual features 22 of an input image. The system searches for drawings corresponding to the target object using a feature vector 52 which is a combination of the geometric features 12 and the visual features 22.

Description

[Technical Field]

[0001] The present invention relates to a drawing retrieval system. In particular, the present invention relates to a drawing retrieval system for retrieving a drawing depicting an object by using a hand-drawn drawing representing the object or an image of the object as input data. [Background Art]

[0002] In recent years, image retrieval technology using deep learning has become widespread. For example, by inputting the name of an object to be searched into a search engine on the Internet, a large number of images of the object or images related to the object can be displayed. Further, by inputting image data into a large-scale visual language model, a descriptive text about an object included in the image data can be output. These image retrieval methods use deep learning models trained on natural images such as photograph data.

[0003] In the field of image retrieval in recent years, Content-Based Image Retrieval (CBIR) technology, which retrieves similar or the same class images using an input image, has been developed. Non-Patent Document 1 discloses an image retrieval technology utilizing Convolutional Neural Networks (CNN) along with the development of deep learning. Non-Patent Document 2 further discloses an image retrieval model based on the design structure of Vision Transformer (hereinafter also referred to as ViT), which is an image recognition model. The Vision Transformer utilizes a self-attention mechanism, divides an image into patches, inputs each patch as a token (a unit of data) into an encoder to extract feature quantities, thereby enabling feature representation of the entire image better than conventional techniques.

[0004] On the other hand, in manufacturing and design settings, there is a demand to search for previously created drawings using handwritten sketches or simple two-dimensional drawings as input data. However, product drawings are created based on Japanese Industrial Standards (JIS) and International Organization for Standardization (ISO) standards, and the appearance of handwritten sketches and drawings often differs. Most content-based image search technologies target natural images, and the accuracy of identifying and searching for similar images that look different at first glance is not yet sufficient.

[0005] As a search technology specifically for similar images in drawings, machine learning models have been developed to extract and parameterize geometric components and their constraints from handwritten sketches and drawings created with 2D CAD. Non-patent document 3 discloses a technology that utilizes an encoder-decoder model of the Transformer architecture to generate drawings by learning as a sequence modeling task based on the order in which humans draw drawings. However, the technology in non-patent document 3 was mainly developed for application to parametric CAD, and its application to the field of searching handwritten drawings has not been realized. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Noh Hyeonwoo, Araujo Andre, Sim Jack, Weyand Tobias, Han Bohyung, "Large-Scale Image Retrieval with Attentive Deep Local Features", 2017, IEEE International Conference on Comput-er Vision (ICCV), IEEE, pp. 3476-3485. [Non-Patent Document 2] Alaaeldin El-Nouby, Natalia Neverova, Ivan Lap-tev, Herve Jegou, “Training Vision Transformers for Image Retrieval” 2021, arXiv:2102.05644. [Non-Patent Document 3] Ari Seff, Wenda Zhou, Nick Richardson, Ryan P Adams “Vitruvion: A Generative Model of Parametric CAD Sketches” 2022, International Conference on Learning Representations [Overview of the project] [Problems that the invention aims to solve]

[0007] Conventional image search models primarily targeted natural images such as photographs. Therefore, they sometimes failed to adequately extract features from artificial, custom-designed images like drawings, especially those with a lot of blank space. Insufficient feature extraction made it difficult to distinguish between products depicted in drawings. Furthermore, the models could overreact to differences in drawing styles and noise in lines and symbols, potentially leading to inaccurate distances between similar and dissimilar drawings in the feature space, making it difficult to find the correct image.

[0008] The present invention has been made in view of the above problems, and aims to provide a technology that uses handwritten sketches or simple drawings drawn with two-dimensional CAD as input data to search for drawings depicting the same or similar objects with greater accuracy than conventional methods. [Means for solving the problem]

[0009] This invention relates to a drawing search system. The drawing search system of the present invention takes an image representing the characteristics of an article as input data, searches for and outputs a drawing corresponding to the article. The drawing search system of the present invention comprises a parametric CAD generation model that outputs geometric features of an image and a vision transformer model that outputs visual features of an image, and is characterized by searching for a drawing corresponding to an article using a feature vector obtained by combining the geometric features and visual features of an image as input data.

[0010] In the drawing search system of the present invention, it is preferable that the parametric CAD generation model is an encoder that performs primitive recognition and constraint recognition of geometric features of images.

[0011] The drawing search system of the present invention further comprises an adapter, which is an additional layer for adjusting geometric features, wherein the adapter preferably performs feature extraction by dimensionality reduction and restoration, and feature aggregation by adding class tokens. [Effects of the Invention]

[0012] The drawing search system of the present invention can search for and output drawings of the target object with higher accuracy than conventional systems, even when inputting data such as handwritten sketches of the target object or simple drawings of the target object created with 2D CAD. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 is a schematic diagram of the feature extractor configuration of the drawing search system. [Figure 2] Figure 2 is a schematic diagram of the classifier in the drawing search system. [Figure 3] Figure 3 schematically illustrates the two-stage learning process of the drawing search system. [Figure 4] Figure 4 shows an example of two-dimensional CAD data used for training the drawing search system. [Figure 5] Figure 5 shows an example of a hand-drawn drawing that can be used as input data for a drawing search system. [Figure 6] FIG. 6 is a diagram comparing a search result of the drawing search system of an example and a search result of the drawing search system of a comparative example. [Figure 7] FIG. 7 is a diagram comparing a search result of the drawing search system of an example and a search result of the drawing search system of a comparative example. MODE FOR CARRYING OUT THE INVENTION

[0014] Hereinafter, the most preferred embodiment of the drawing search system of the present invention will be described with reference to the drawings. Next, a result of searching performed using the drawing search system of the present embodiment will be described. FIG. 1 shows a schematic diagram of a preferred configuration of the drawing search system of the present invention. The drawing search system of the present embodiment is embodied as a program executed on a single computer or a plurality of computers.

[0015] The drawing search system 1 of the present embodiment includes a feature extractor 2 and a discriminator 3. The feature extractor 2 includes a geometric model 11 and a visual model 21. When identical image data representing an article to be searched is input to the geometric model 11 and the visual model 21, the geometric model 11 outputs a geometric feature amount 12 of the image data, and the visual model 21 outputs a visual feature amount 22 of the image data. The drawing search system 1 combines the respective geometric feature amount 12 and visual feature amount 22 at a fully connected layer 51, and outputs the combined result as a feature vector 52. The discriminator 3 searches for drawings with high similarity using the feature vector 52, searches for a drawing corresponding to the article, and outputs the searched drawing.

[0016] The geometric model 11 in the present embodiment is an encoder of a parametric CAD generation model. The parametric CAD generation model performs primitive recognition and constraint recognition on an input image, and outputs geometric feature quantities 12. The geometric feature quantities 12 referred to herein are individual attributes and elements of data representing the size, shape, structure, etc. of a figure. Primitive recognition for outputting geometric feature quantities 12 refers to recognizing and extracting basic geometric elements constituting an object, and includes, for example, recognition and extraction of elements such as points, lines, circles, arcs, and polygons. Constraint recognition for outputting geometric feature quantities 12 refers to recognizing and extracting constraints for maintaining consistency of an object, such as position, angle, dimensions, and dependencies between components.

[0017] The parametric CAD generation model in the present embodiment is a model created based on the encoder of the image-conditional generation model of Vitruvion in Non-Patent Document 3, whose code is publicly available. The geometric model 11 of the present embodiment is configured to output intermediate feature quantities encoded by the encoder of the parametric CAD generation model. Primitives are represented by binary information indicating the type, coordinates, radius, and whether the drawn line type is a solid line, and these are handled as one set. Position embedding is applied to each primitive, the maximum token length is set to 100, and num_bins and the number of position embeddings are set to 128, thereby increasing the number of learnable primitives. Furthermore, the expressive power is improved by increasing the number of heads of the transformer to 8 and the number of layers to 12.

[0018] The visual model 21 in the present embodiment is a vision transformer model. The vision transformer model extracts and outputs visual feature quantities 22 from an input image. In the present embodiment, the model is created based on Vision Transformer (ViT) whose code is publicly available. By combining the retention of detailed information and feature extraction by the visual model 21 with the feature quantities of the geometric model 11, the accuracy of similarity calculation is improved.

[0019] The feature extractor 2 of this embodiment further includes an adapter 41, which is an additional layer for adjusting the geometric features 12 extracted by the geometric model 11. The adapter 41 performs feature extraction by dimensionality reduction and restoration, and feature aggregation by adding class tokens.

[0020] The geometric features 12 adjusted by the adapter 41 are combined with the visual features 22. Then, dimensionality reduction is performed in the fully connected layer 51 to obtain a feature vector 52 used for distance measurement. Next, the drawing retrieval system 1 inputs the obtained feature vector 52 into the classifier 3 and maps it to the embedding space 31 of the classifier 3. The embedding space 31 is trained so that similar images are placed closer together and different images are placed further apart, and the system outputs drawing data with feature vectors close in distance to the input feature vector 52, i.e., drawing data with high similarity, as search results. Cosine similarity or Euclidean distance can be used to evaluate similarity. An overview of the processing by the classifier 3 of the drawing retrieval system 1 is shown in Figure 2.

[0021] The drawing search system 1 of this embodiment improves search accuracy by performing optimal learning as described below. Figure 3 shows an overview of the learning process of the drawing search system.

[0022] In this embodiment, two types of input data were used for training: a filtered SketchGraphs dataset and a set of two-dimensional drawing data publicly available on the internet. Each sketch in the SketchGraphs dataset consists of basic primitives such as points, lines, circles, and arcs. The SketchGraphs dataset contains sketches of the same shape, resulting in a bias in the data. Furthermore, it tends to have many sketches with simple shapes and few data containing many primitives or complex shapes. Therefore, with the aim of improving the performance of the search system by training it with data containing complex shapes, a set of two-dimensional drawing data publicly available on the internet was created and additional training was performed. Hereinafter, the dataset based on two-dimensional drawing data from the internet will be referred to as the Drawing dataset.

[0023] The Drawing dataset uses 2D drawing data publicly available on the internet, specifically selected data in DXF format. Since DXF format 2D drawing data is stored in text format, it cannot be directly input into the search system. Therefore, the DXF data was converted into primitive information and formatted to a format suitable for the pipeline for training. Figure 4 shows an example of an image from the Drawing dataset used for training. The Drawing dataset contains 10,441 data points, selected from approximately 40,000 data points collected from the internet, with data containing an excessive number of primitives excluded, resulting in a total primitive count of 100 or less. By limiting the upper limit of the primitive count, assembly drawings containing multiple parts could be excluded from the training data, enabling more efficient training.

[0024] For training, we used the SketchGraphs dataset and the Drawing dataset, with annotations (labels) added to individual data points. The annotations are numbers that classify drawings of similar products based on product variety, and these were used as signals to determine similarity during training and testing. In this training, we used 3,210 data points as annotated training data.

[0025] As the first stage of learning, distance learning was performed on the encoder of the parametric CAD generation model, which is geometric model 11. Using the SketchGraphs dataset and the annotated Drawing dataset, training was performed to accurately predict primitives.

[0026] As the second stage of learning, distance learning was performed on the vision transformer model (visual model 21), the adapter 41 that adjusts geometric features, and the fully connected layer 51 that performs feature integration and dimensionality reduction. Contrast loss was used as the loss function for learning, and optimization was performed. During the second stage of learning, the weights of the geometric model 11 were frozen and reused from those obtained in the first stage of learning.

[0027] The following shows the effect of optimal training on the geometric model 11 of the drawing search system 1 of this embodiment, expressed as the accuracy of primitive prediction. There are three evaluation methods: Accuracy, bits per primitive, and bits per sketch.

[0028] Accuracy is a value that represents the prediction accuracy of the coordinate values ​​of each primitive; a higher value indicates higher accuracy.

[0029] Bits per primitive (bpp) is a value that represents the amount of information in bits required to represent one primitive, and is expressed by the following equation (1).

number

[0030] Bits per sketch (bps) is a value that represents the amount of information in bits required to represent the entire sketch, and is expressed by the following equation (2).

number

[0031] The geometric model 11 in this embodiment is trained using two types of datasets: the SketchGraphs dataset and the Drawing dataset. As a comparative example, Table 1 shows the evaluation results of a model trained on the SketchGraphs dataset using Vitruvision, which served as the basis for model creation, a model trained on the SketchGraphs dataset using only the geometric model 11, and a model trained on the Drawing dataset using only the geometric model 11. In Table 1, the SketchGraphs dataset is denoted by the symbol SG.

[0032] [Table 1]

[0033] The geometric model 11 of this embodiment was found to be able to extract geometric features with higher accuracy than the comparative example, which was trained on a single dataset. Furthermore, when training was performed using only the SketchGraphs dataset, there was a tendency for the predictions to be biased towards simple shapes. When training was performed using only the Drawing dataset, the loss may not have been sufficiently reduced due to the insufficient amount of data, resulting in decreased accuracy. The geometric model 11 of this embodiment, by performing two-stage training using two datasets, ensures a sufficient amount of data necessary for training while improving performance for complex shapes.

[0034] Table 2 below shows the differences in search accuracy when different training data are applied to the geometric model 11 of this embodiment. The recall metric is used to evaluate accuracy. Here, recall refers to the "percentage (TP) in which the model's predicted value is judged as positive when the actual correct value is positive." recall@k in Table 2 is calculated using the following formula. recall@k = Top k TPs / Total number of TPs in the entire dataset [Table 2]

[0035] It is clear that the training method for the geometric model 11 using two image datasets contributes to improving the accuracy of drawing searches in the entire drawing search system 1.

[0036] Table 3 shows the results of comparing the accuracy of drawing search system 1 with that of the drawing search system disclosed in Non-Patent Document 2. The comparison was performed by searching using drawing data from the Drawing dataset that was not used for training as input data, and evaluating the recall rate. The drawing search system in Non-Patent Document 2 is shown below as a "conventional example". [Table 3]

[0037] The drawing search system 1 was found to have virtually equivalent performance to the conventional drawing search system, with no significant difference. However, when the input data was realistic image data similar to assembly drawings, the search accuracy of the conventional search system was slightly higher. This is likely because the conventional drawing search system was pre-trained on natural images, and the visual features improved the accuracy of the search results. Furthermore, since the geometric model of the drawing search system 1 is an encoder of a parametric CAD generation model, and thus utilizes intermediate features of the generation model, the effect of parameter updates in distance learning may have been relatively low.

[0038] The drawing search system 1 of this embodiment is capable of searching for corresponding drawings with higher accuracy than conventional systems, even when using handwritten drawings as input data. The results of a search performed using handwritten drawings as input are described below.

[0039] The handwritten drawings used as input data were created using a tablet device to mimic 2D CAD drawings. An example of a handwritten drawing is shown in Figure 5. The 2D CAD drawings used as examples for the handwritten drawings were randomly selected for each class used in the learning process.

[0040] The drawings to be searched are image data based on 2D CAD drawings. The images to be searched are pre-selected so that each set of images always includes images of the same class as the image that served as the source for the input data.

[0041] Table 4 below shows the search accuracy when using handwritten drawings as input data to search for corresponding drawings from a set of 1960 images. [Table 4]

[0042] The drawing search system 1 was confirmed to have approximately 10% improved accuracy compared to the conventional example.

[0043] Figures 6 and 7 show the drawings output by the drawing search system 1, which ranked handwritten input data as corresponding drawings. The comparative example shows the results output by a conventional drawing search system. In the figures, the output drawings enclosed in black borders represent the correct class, and the output drawings enclosed in gray borders represent the incorrect class.

[0044] In the search results using the side view of the "screw" in Figure 6(a) as input data, the drawing search system 1 ranked the drawings in the correct class as 1st and 2nd place. However, in the previous example, the drawings output from 1st to 4th place were ranked as incorrect. The input drawing in this case contained three rectangles, and the drawing search system 1 recognized its outline. On the other hand, the drawings ranked higher in the previous example had their outlines described as two rectangular parts, making it highly probable that the outlines were not correctly recognized. The accuracy of outline recognition significantly affects the accuracy of the search results, as can be inferred from the results in Figure 6(b).

[0045] In the search results shown in Figure 6(c), the drawing search system was able to find images of the correct class for all of the top five positions, whereas in the comparative example, images of the incorrect class were output as the 3rd and 5th ranked drawings. One possible reason why incorrect class images were selected in the comparative example is that the distance between different classes is small, and the incorrect class images may have been judged as being as similar as the correct class images. This means that the variance between classes is small.

[0046] As shown in Figures 7(a) and 7(c), the drawing search system 1 performs searches while correcting for the center deviation inherent in handwritten drawings. Furthermore, it performs more accurate identification than conventional methods regarding whether lines in a drawing contain handwritten noise or distortion, or whether they represent the outline of the drawing being searched. Note that the black rectangular area at the top of the input data in Figure 7(c) is the data path name and label name added after the search, and does not affect the search results.

[0047] On the other hand, in the search results of drawing search system 1 and the conventional example, a tendency for search accuracy to decrease was observed when a handwritten drawing containing many dashed lines, as shown in Figure 7(b), was input. If the learning content can reflect the JIS or ISO regulations for drawing creation, search accuracy can be improved.

[0048] As described above, the drawing search system 1 of this embodiment extracts features necessary for primitive inference, such as lines and circles, from handwritten drawing images by considering geometric features, thereby reducing the domain gap with the drawing image being searched. As a result, the accuracy of identifying external shapes and structures is improved compared to conventional drawing search systems, providing a drawing search system with higher search accuracy. [Explanation of Symbols]

[0049] 1. Drawing Search System 2. Feature Extractor 3. Classifier 11 Geometric Models 21 Visual Models 31 Embedded space 41 Adapter 51 Fully connected layer 52 Feature vectors

Claims

1. A drawing search system that takes an image representing the characteristics of an item as input data, searches for and outputs a drawing corresponding to the said item, A parametric CAD generation model that outputs geometric features of the aforementioned image, A vision transform model that outputs the visual features of the aforementioned image, It is equipped with, A drawing search system characterized by searching for drawings corresponding to the aforementioned article using a feature vector obtained by combining the aforementioned geometric features and the aforementioned visual features, with the aforementioned image as input data.

2. The drawing search system according to claim 1, wherein the parametric CAD generation model is an encoder that performs primitive recognition and constraint recognition of geometric features of an image.

3. The drawing search system according to claim 2, further comprising an adapter which is an additional layer for adjusting the geometric features, wherein the adapter performs feature extraction by dimensionality reduction and restoration, and feature aggregation by adding class tokens.

Images