Processing method and processing device using same
Patent Information
- Application Number
- JP2021041612
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-03-15
- Publication Date
- 2025-04-25
- Estimated Expiration
- 2041-03-15
AI Technical Summary
Existing neural networks struggle to accurately classify and count diverse electrical symbols in construction drawings due to variations in symbol shapes and insufficient learning, leading to increased user effort and reduced accuracy.
A processing device and method that utilizes a legend image with a list of reference symbols, employing two neural networks to extract and classify electrical symbols from construction drawings, determining similarity through convolutional and pooling layers, and accumulating counts based on maximum similarity.
Accurately integrates and counts electrical symbols with high precision, reducing user effort and improving processing efficiency by leveraging a legend image and neural networks to enhance symbol recognition.
Smart Images
Figure 00000008_0000 
Figure 00000009_0000 
Figure 00000010_0000
Abstract
Description
Technical Field
[0001] The present disclosure relates to a processing technology, particularly a processing method for executing processing on an image and a processing apparatus using the same.
Background Art
[0002] In recent years, construction drawings are easily created using a CAD (Computer-Aided Design) system or the like. When creating an estimate based on a construction drawing, the user extracts symbols included in the construction drawing and derives the quantity of each symbol. When performing these manually, the user's effort increases. To reduce the user's effort, it is possible to use a neural network to extract symbols. However, since there are more than dozens of types of symbols and there are also symbols with similar shapes, it is difficult to train a neural network for all symbols. On the other hand, if the training is insufficient, the accuracy of the processing decreases. In contrast, in Patent Document 1, symbols are classified into a plurality of groups, a neural network is trained based on the groups, and symbols are extracted from the target image as groups (see, for example, Patent Document 1).
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] It is desired to more accurately accumulate the number of symbols included in an image.
[0005] The present disclosure has been made in view of such a situation, and an object thereof is to provide a technology for accurately accumulating the number of symbols included in an image.
Means for Solving the Problems
[0006] To solve the above problems, an apparatus according to one aspect of the present disclosure is an apparatus for accumulating multiple symbols contained in a target image to be processed for each type of symbol, comprising: an extraction unit that extracts each symbol as a reference symbol from a legend image showing a list of multiple types of symbols; a first processing unit that extracts each of the multiple symbols contained in the target image as a target symbol by executing processing of a first neural network trained with multiple types of reference symbols on the target image; a second processing unit that derives the similarity between each of the multiple target symbols and each reference symbol by executing processing of a second neural network trained with multiple types of reference symbols on each combination of each of the multiple target symbols and each reference symbol; and an accumulation unit that calculates the number of target symbols contained in each reference symbol by classifying each target symbol into the reference symbol with the greatest similarity.
[0007] Another aspect of the present disclosure is a processing method. This method is a processing method for accumulating multiple symbols contained in a target image to be processed for each type of symbol, and comprises the steps of: extracting each symbol as a reference symbol from a legend image showing a list of multiple types of symbols; extracting each of the multiple symbols contained in the target image as a target symbol by performing processing on the target image with a first neural network trained with multiple types of reference symbols; deriving the similarity between each of the multiple target symbols and each reference symbol by performing processing on each of the multiple target symbols and each reference symbol; and accumulating the number of target symbols contained in each reference symbol by classifying each target symbol into the reference symbol with the greatest similarity.
[0008] Furthermore, any combination of the above components, or any conversion of the expressions of this disclosure between methods, apparatus, systems, computer programs, or recording media on which computer programs are recorded, are also valid forms of this disclosure. [Effects of the Invention]
[0009] According to this disclosure, the number of symbols contained in an image can be calculated with high accuracy. [Brief explanation of the drawing]
[0010] [Figure 1] This figure shows the target image to be processed in the example. [Figure 2] This figure shows an example image of the embodiment. [Figure 3] This figure shows the configuration of the processing apparatus according to the embodiment. [Figure 4] This figure shows an overview of the processing in the second processing unit, as shown in Figure 3. [Figure 5] This figure shows an overview of the processing of the neural network for target symbols and the neural network for reference symbols in Figure 4. [Figure 6] This figure shows an overview of the processing in the calculation unit. [Figure 7] Figure 3 is a flowchart showing the processing procedure by the processing unit. [Modes for carrying out the invention]
[0011] Before specifically describing the embodiments of this disclosure, an overview of these embodiments will be provided. The embodiments relate to a processing device that accumulates the number of electrical symbols of each type from a construction drawing in which multiple electrical symbols are shown. Using a neural network is a promising way to improve the accuracy of extracting electrical symbols from construction drawings. However, since the same equipment can be represented by various electrical symbols, it is difficult to train a neural network on all electrical symbols. For example, even the same equipment may have different shapes of electrical symbols depending on the region, etc. Therefore, it is difficult to train a neural network on all electrical symbols. Insufficient training can also lead to misclassification. This embodiment uses a neural network to accurately accumulate the number of electrical symbols included in a construction drawing.
[0012] The processing device according to this embodiment uses a legend image attached to an architectural drawing, which is a legend image showing a list of electrical symbols used in the architectural drawing, as the neural network. The processing device extracts each electrical symbol from the legend image as a reference symbol and trains the first neural network using the reference symbols. The processing device uses the first neural network on the architectural drawing to extract each electrical symbol from the architectural drawing as a target symbol. The processing device trains the second neural network using the reference symbols. The processing device uses the second neural network on the target symbols and the reference symbols to derive the similarity between the target symbols and each reference symbol. Based on the similarity, the processing device calculates the number of target symbols included in each reference symbol.
[0013] Figure 1 shows the target image 10 to be processed. An example of the target image 10 is a construction drawing, as mentioned above. The target image 10 contains symbols collectively called symbols 20, from the 1st symbol 20a to the 26th symbol 20z, the 27th symbol 20aa, the 28th symbol 20ab, the 29th symbol 20ac, and the 30th symbol 20ad. Each symbol 20 is an electrical symbol. Here, the number of symbols 20 included in the target image 10 is set to "30," but it is not limited to this. Different shapes of symbols 20 indicate different electrical symbols and therefore correspond to different equipment. For example, the 5th symbol 20e and the 6th symbol 20f have the same shape, but the 5th symbol 20e and the 1st symbol 20a have different shapes. When creating an estimate based on such a target image 10, the user needs to count the number of symbols 20 for each type of symbol 20. As the size of the target image 10 increases and the number of symbols 20 included in the target image 10 increases, the amount of effort required for this task increases.
[0014] Figure 2 shows the legend image 30. The legend image 30 is a legend table that lists the symbols 20 used in the target image 10. The legend image 30 may be placed in a part of the target image 10, for example, in the lower left or lower right section, or it may be prepared separately from the target image 10. Hereinafter, the symbols 20 in the legend image 30 will be referred to as reference symbols 40. In Figure 2, the legend image 30 includes the first reference symbol 40a, the second reference symbol 40b, and the third reference symbol 40c, which are collectively referred to as reference symbols 40. The first reference symbol 40a indicates a fixed-temperature smoke detector, the second reference symbol 40b indicates an intercom, and the third reference symbol 40c indicates a photoelectric smoke detector. The reference symbols 40 included in the legend image 30 are not limited to these. In general, the reference symbols 40 are not standardized and differ depending on the target image 10. Also, the reference symbols 40 do not have to be exactly the same as the symbols 20 actually used in the target image 10. These reference symbols 40 are placed within a table composed of vertical and horizontal lines.
[0015] Figure 3 shows the configuration of the processing unit 100. The processing unit 100 accumulates multiple symbols contained in the target image 10 to be processed, categorized by symbol type. The processing unit 100 includes an extraction unit 110, a first processing unit 112, a second processing unit 114, and an accumulation unit 116. The target image 10 is an architectural drawing shown in Figure 1 and is also electronic information. The legend image 30 is a legend table shown in Figure 2. As mentioned above, the legend image 30 may be placed in a part of the target image 10.
[0016] The extraction unit 110 receives the legend image 30 and extracts each reference symbol 40 from the legend image 30. The extraction unit 110 has pre-stored the vertical and horizontal lines that make up the table, and identifies the position of the table contained in the legend image 30 by performing pattern matching on the vertical and horizontal lines. The extraction unit 110 extracts the reference symbols 40 that are placed in the identified table. Techniques other than pattern matching may be used for the reference symbols 40 in the extraction unit 110. The extraction unit 110 outputs the extracted multiple types of reference symbols 40 to the first processing unit 112 and the second processing unit 114.
[0017] The first processing unit 112 receives the target image 10 and also receives a plurality of types of reference symbols 40 from the extraction unit 110. The first processing unit 112 holds a convolutional neural network including a combination of a convolutional layer and a pooling layer as the first neural network. The first processing unit 112 trains the first neural network with a plurality of types of reference symbols 40. For the training of the first neural network, known techniques may be used, but the convolutional layer of the convolutional neural network is trained. The first processing unit 112 predicts the coordinates of the centroid of each position of the plurality of symbols 20 included in the target image 10, or a part of the coordinates of each position of the plurality of symbols 20, by executing the processing of the trained first neural network on the target image 10. The part of the coordinates of the position is, for example, the coordinates of the upper left end and the lower right end of the symbol 20. The first processing unit 112 extracts each of the plurality of symbols 20 arranged at the predicted coordinates from the target image 10 as a target symbol 50. The first processing unit 112 outputs the extracted plurality of target symbols 50 to the second processing unit 114.
[0018] The second processing unit 114 receives a plurality of types of reference symbols 40 from the extraction unit 110 and receives a plurality of target symbols 50 from the first processing unit 112. The second processing unit 114 holds a second neural network and derives the similarity between each of the plurality of types of reference symbols 40 and each target symbol 50 by using the second neural network.
[0019] FIG. 4 shows an overview of the processing in the second processing unit 114. The second neural network in the second processing unit 114 includes a neural network 200 for target symbols for inputting the target symbols 50 and a neural network 210 for reference symbols for inputting the reference symbols 40.
[0020] Figure 5 shows an overview of the processing of the neural network 200 for target symbols and the neural network 210 for reference symbols. The neural network 200 for target symbols and the neural network 210 for reference symbols are convolutional neural networks, and include a first convolutional layer 242a to a third convolutional layer 242c collectively referred to as a convolutional layer 242, a first pooling layer 244a and a second pooling layer 244b collectively referred to as a pooling layer 244, and a fully connected layer 246. Here, as an example, the first convolutional layer 242a, the first pooling layer 244a, the second convolutional layer 242b, the second pooling layer 244b, the third convolutional layer 242c, and the fully connected layer 246 are arranged in order. The configuration of the convolutional neural network is not limited to this.
[0021] Each convolutional layer 242 performs spatial filtering on the input image while shifting a spatial filter having a size smaller than the size of the image. Since spatial filtering is a well-known technique, the description thereof is omitted here, but this spatial filtering corresponds to a convolution process, and feature amounts of the image are extracted by the convolution process. Padding or the like may be performed in the convolutional layer 242.
[0022] Each pooling layer 244 reduces the size of the image by combining a plurality of pixels included in an arbitrary region in the image into one pixel. Here, in order to combine a plurality of pixels into one pixel, average pooling or max pooling is performed. In average pooling, the average value of the plurality of pixel values in the region is used for one pixel, and in max pooling, the maximum value among the plurality of pixel values in the region is used for one pixel. The pooling process is performed to enhance the robustness against translational movement of the representative value or the average value in the region of interest. Since the pooling layer 244 follows the convolutional layer 242, it also corresponds to the feature amounts of the image extracted in the convolutional layer 242.
[0023] The fully connected layer 246 is placed at the final stage of the target symbol neural network 200 and the reference symbol neural network 210. Each convolutional layer 242 of the target symbol neural network 200 and the reference symbol neural network 210 is trained using the reference symbol 40. Since known techniques are used for this training, a detailed explanation is omitted here.
[0024] After training is complete, the first convolutional layer 242a of the neural network 200 for target symbols is input with the target symbol 50 (Figure 4), and each convolutional layer 242 and each pooling layer 244 performs the processing described above. The third convolutional layer 242c is an intermediate layer, and the image of the processing result of the intermediate layer is output as a feature image. The feature image obtained from the intermediate layer of the neural network 200 for target symbols is shown as the target feature image 220 in Figure 4. The target feature image 220 has a size of, for example, 4 × 4 × 256.
[0025] After training is complete, the first convolutional layer 242a of the reference symbol neural network 210 is input with the reference symbol 40 (Figure 4), and each convolutional layer 242 and each pooling layer 244 performs the processing described above. The third convolutional layer 242c outputs the image of the processing result of the intermediate layer as a feature image. The feature image obtained from the intermediate layer of the reference symbol neural network 210 is shown as the reference feature image 230 in Figure 4. The reference feature image 230 also has a size of, for example, 4 × 4 × 256. Return to Figure 4.
[0026] The second processing unit 114 converts the target feature image 220, which has a size of 4 × 4 × 256, into a first feature vector 222, which is a one-dimensional vector. The size of the first feature vector 222 is, for example, 4096. The second processing unit 114 also converts the reference feature image 230, which has a size of 4 × 4 × 256, into a second feature vector 232, which is a one-dimensional vector. The size of the second feature vector 232 is also, for example, 4096. The second processing unit 114 derives the distance between the first feature vector 222 and the second feature vector 232 by using the squared error of the two vectors, etc. The smaller the distance between the first feature 222 and the second feature 232, the greater the similarity between the first feature 222 and the second feature 232. Conversely, the larger the distance between the first feature 222 and the second feature 232, the smaller the similarity between the first feature 222 and the second feature 232. Therefore, distance and similarity are related parameters; for example, the reciprocal of the distance corresponds to the similarity. The second processing unit 114 derives the similarity (distance) for each combination of all reference symbols 40 and all target symbols 50. In other words, the second processing unit 114 derives the similarity (distance) between each of the multiple types of target symbols 50 and each reference symbol 40. Returning to Figure 3, the second processing unit 114 outputs the multiple similarity (distance) values to the integration unit 116.
[0027] The accumulation unit 116 receives multiple similarity (distance) values from the second processing unit 114. The accumulation unit 116 classifies each target symbol 50 into the reference symbol 40 with the highest similarity. To explain the processing in the accumulation unit 116, Figure 6 is also used here. Figure 6 shows an overview of the processing in the accumulation unit 116. To clarify the explanation, some of the target symbols 50 are shown as the first target symbol 50a, the second target symbol 50b, and the third target symbol 50c, and some of the reference symbols 40 are shown as the first reference symbol 40a and the third reference symbol 40c. The distance between the first target symbol 50a and the first reference symbol 40a is shown as "0.1", and the distance between the first target symbol 50a and the second reference symbol 40b is shown as "0.6". Since the distance between the first target symbol 50a and the first reference symbol 40a is smaller than the distance between the first target symbol 50a and the second reference symbol 40b, the first target symbol 50a is classified as the first reference symbol 40a. A smaller distance corresponds to a higher degree of similarity. The same process is applied to the second target symbol 50b and the third target symbol 50c, so that the second target symbol 50b is classified as the first reference symbol 40a, and the third target symbol 50c is classified as the third reference symbol 40c.
[0028] The accumulation unit 116, following the classification, accumulates the number of target symbols 50 contained in each reference symbol 40. In Figure 6, the number of target symbols 50 contained in the first reference symbol 40a is "2", and the number of target symbols 50 contained in the second reference symbol 40b is "1". The accumulation unit 116 outputs the accumulation result.
[0029] The subject of the apparatus, system, or method in this disclosure comprises a computer. The functions of the subject of the apparatus, system, or method in this disclosure are realized by the computer executing a program. The computer comprises a processor as its main hardware component, which operates according to the program. The processor is of any type as long as it can realize its functions by executing the program. The processor consists of one or more electronic circuits, including semiconductor integrated circuits (ICs) or LSIs (Large Scale Integrations). Multiple electronic circuits may be integrated on one chip or provided on multiple chips. Multiple chips may be aggregated in one device or provided on multiple devices. The program is recorded on a non-temporary recording medium such as a ROM, optical disc, or hard disk drive that is readable by the computer. The program may be pre-stored on the recording medium or supplied to the recording medium via a wide-area communication network, including the Internet.
[0030] The operation of the processing device 100 with the above configuration will now be explained. Figure 7 is a flowchart of the processing procedure by the processing device 100. The extraction unit 110 extracts reference symbols 40 from the legend image 30 (S10). The first processing unit 112 extracts target symbols 50 from the target image 10 (S12). The second processing unit 114 derives the similarity between the target symbols 50 and the reference symbols 40 (S14). The accumulation unit 116 classifies the target symbols 50 into one of the reference symbols 40 based on the similarity (S16). The accumulation unit 116 accumulates the number of target symbols 50 for each reference symbol 40 (S18).
[0031] In this embodiment, the similarity between the target symbol 50 and the reference symbol 40 is derived by performing processing by the second neural network on the reference symbol 40 extracted from the legend image 30 and the target symbol 50 extracted from the target image 10 by processing by the first neural network, so that the target symbol 50 can be classified into one of the reference symbols 40. In addition, since the target symbol 50 is classified into one of the reference symbols 40 using the legend image 30, the number of target symbols 50 contained in each reference symbol 40 can be accumulated with high accuracy. Furthermore, the first neural network predicts the coordinates of the centroid of the position of the target symbol 50, or the coordinates of a part of the position of the target symbol 50, so that the target symbol 50 can be extracted from the target image 10. In addition, the similarity can be derived by deriving the distance between the feature quantities of the target symbol 50 and the reference symbol 40 obtained from the intermediate layer of the second neural network.
[0032] An outline of one aspect of this disclosure is as follows: A processing device (100) in one aspect of this disclosure is a processing device (100) that accumulates multiple symbols contained in a target image (10) to be processed for each type of symbol, comprising: an extraction unit (110) that extracts each symbol as a reference symbol (40) from a legend image (30) showing a list of multiple types of symbols, and a first neural network trained with multiple types of reference symbols (40) on the target image (10) to extract each of the multiple symbols contained in the target image (10) as a target symbol (50). The system comprises a first processing unit (112), a second processing unit (114) that derives the similarity between each of the multiple target symbols (50) and each reference symbol (40) by executing processing on a second neural network trained with multiple types of reference symbols (40) for each combination of multiple target symbols (50) and each reference symbol (40), and an integrating unit (116) that calculates the number of target symbols (50) included in each reference symbol (40) by classifying each target symbol (50) to the reference symbol (40) with the greatest similarity.
[0033] The first neural network used in the first processing unit (112) may predict the coordinates of the centroid of the position of the target symbol (50), or the coordinates of a part of the position of the target symbol (50).
[0034] The second processing unit (114) may derive the similarity score as the distance between the feature quantities of the target symbol (50) and the reference symbol (40) obtained from the intermediate layer of the second neural network.
[0035] Another aspect of this disclosure is a processing method. This method is a processing method for accumulating multiple symbols contained in a target image (10) to be processed for each type of symbol, and comprises the steps of: extracting each symbol as a reference symbol (40) from a legend image (30) showing a list of multiple types of symbols; extracting each of the multiple symbols contained in the target image (10) as a target symbol (50) by performing processing on the target image (10) with a first neural network trained with multiple types of reference symbols (40); deriving the similarity between each of the multiple target symbols (50) and each reference symbol (40) by performing processing on each of the multiple target symbols (50) and each reference symbol (40); and accumulating the number of target symbols (50) contained in each reference symbol (40) by classifying each target symbol (50) to the reference symbol (40) with the greatest similarity.
[0036] The present disclosure has been described above based on examples. These examples are illustrative, and it will be understood by those skilled in the art that various modifications are possible for each component or combination of processing steps, and that such modifications are also within the scope of the present disclosure. [Explanation of Symbols]
[0037] 10 Target image, 20 Symbol, 30 Legend image, 40 Reference symbol, 50 Target symbol, 100 Processing unit, 110 Extraction unit, 112 First processing unit, 114 Second processing unit, 116 Integration unit, 200 Neural network for target symbol, 210 Neural network for reference symbol, 220 Target feature image, 222 First feature quantity, 230 Reference feature image, 232 Second feature quantity, 242 Convolutional layer, 244 Pooling layer, 246 Fully connected layer.
Claims
1. A processing device that accumulates a plurality of symbols included in a target image to be processed for each symbol type, an extracting unit that extracts each symbol as a reference symbol from a legend image showing a list of multiple types of symbols; a first processing unit that extracts each of the plurality of symbols included in the target image as a target symbol by executing processing on the target image by a first neural network that has been trained using a plurality of types of the reference symbols; a second processing unit that derives a similarity between each of the plurality of target symbols and each of the reference symbols by executing processing of a second neural network that has been trained using a plurality of types of the reference symbols for each combination of the plurality of target symbols and each of the reference symbols; an accumulator for accumulating the number of the target symbols included in each reference symbol by classifying each target symbol into the reference symbol with the highest similarity; A processing device comprising:
2. 2. The processing device according to claim 1, wherein the first neural network used in the first processing unit predicts the coordinates of the center of gravity of the position of the target symbol or the coordinates of a part of the position of the target symbol.
3. The processing device according to claim 1 or 2, wherein the second processing unit derives, as the similarity, a distance between feature amounts of the target symbol and the reference symbol obtained from an intermediate layer of the second neural network.
4. A processing method for accumulating a plurality of symbols included in a target image to be processed for each symbol type, comprising: A step of extracting each symbol as a reference symbol from a legend image showing a list of multiple types of symbols; extracting each of the plurality of symbols included in the target image as a target symbol by executing processing of a first neural network trained by a plurality of types of the reference symbols on the target image; a step of deriving a similarity between each of the plurality of target symbols and each of the plurality of reference symbols by executing processing of a second neural network trained by a plurality of types of the reference symbols for each combination of the plurality of target symbols and each of the plurality of reference symbols; accumulating the number of target symbols contained in each reference symbol by grouping each target symbol with the reference symbol with which it has the greatest similarity; A processing method comprising: