Drawing recognition device and drawing recognition program

Abstract

To provide a drawing recognition device and drawing recognition program that recognize a correspondence relation between an attribute of a symbol and a connection line without requiring arrangement knowledge.SOLUTION: A drawing recognition device 100 comprises: a drawing acquisition unit 101 that acquires drawing data; and a recognition unit 102 that has a symbol recognition unit 1021 recognizing information on a symbol in the drawing data on the basis of the drawing data and a first machine learning model, an attribute information recognition unit 1022 recognizing information on an attribute in the drawing data on the basis of the drawing data and a second machine learning model, a connection line recognition unit 1023 recognizing information on a connection line on the drawing data on the basis of the drawing data and a third machine learning model, and an association recognition unit 1024 recognizing information on a correspondence relation between the attribute in the drawing data, the symbol therein and the connection line therein on the basis of the pieces of information recognized by the symbol recognition unit 1021, the attribute information recognition unit 1022 and the connection line recognition unit 1023 and a fourth machine learning model.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to a drawing recognition device and a drawing recognition program.

Background Art

[0002] In a drawing in which characters or symbols are mixed, there is known a drawing recognition technique for recognizing characters or symbols existing on the drawing and determining their correspondence. For example, in Patent Document 1, in drawing data in which characters, symbols, and pipes are mixed, the correspondence as to which symbol or pipe the individually recognized characters correspond to is determined based on the arrangement knowledge of the mutual positional relationship between the characters and the symbols or pipes prepared in advance. A drawing recognition technique is disclosed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the prior art as disclosed in Patent Document 1, in performing drawing recognition, in order to recognize the correspondence between a symbol (symbol), a character (attribute) associated with the symbol, and a connection line connecting the symbols, there is a problem that arrangement knowledge must be prepared in advance. [[ID=!37]]

[0005] The present disclosure has been made to solve the above problems, and an object thereof is to provide a drawing recognition device capable of recognizing the correspondence between a symbol, an attribute, and a connection line without requiring arrangement knowledge.

Means for Solving the Problems

[0006] The drawing recognition device relating to this disclosure includes: a drawing acquisition unit that acquires drawing data; a first recognition unit that recognizes information about symbols in the drawing data based on the drawing data acquired by the drawing acquisition unit and a first machine learning model that takes the drawing data as input and outputs recognition results about symbols on the drawing data; a second recognition unit that recognizes information about attributes in the drawing data based on the drawing data acquired by the drawing acquisition unit and a second machine learning model that takes the drawing data as input and outputs recognition results about attributes on the drawing data; a third recognition unit that recognizes information about connecting lines in the drawing data based on the drawing data acquired by the drawing acquisition unit and a third machine learning model that takes the drawing data as input and outputs recognition results about connecting lines that connect symbols on the drawing data; and the first recognition unit Information about symbols recognized using the first machine learning model. , second recognition part Information about attributes recognized using the second machine learning model. , and the third recognition unit Regarding the connection lines recognized using the third machine learning model The system includes a fourth recognition unit that recognizes information regarding the correspondence between attributes, symbols, and connecting lines in drawing data, based on a fourth machine learning model that takes information and information regarding attributes recognized by a second recognition unit as input and outputs information regarding the correspondence between attributes and symbols in drawing data, and information regarding the correspondence between attributes and connecting lines in drawing data. [Effects of the Invention]

[0007] According to this disclosure, the correspondence between symbols, attributes, and connecting lines can be recognized without requiring knowledge of placement. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the configuration of the drawing recognition device according to Embodiment 1. [Figure 2] This is a diagram illustrating an example of an image drawing in Embodiment 1. [Figure 3] This figure illustrates an example of a determination result output by the determination unit in Embodiment 1. [Figure 4] This is a diagram illustrating an example of component information in Embodiment 1. [Figure 5] This figure shows a detailed configuration example of the recognition unit in Embodiment 1. [Figure 6] In Embodiment 1, Figure 6A shows an example of the symbol recognition result acquired by the symbol recognition unit, the attribute recognition result acquired by the attribute information recognition unit, and the connection line recognition result acquired by the connection line recognition unit. Figure 6B shows an image in which the symbol recognition result, attribute recognition result, and connection line recognition result are mapped onto an image drawing acquired by the drawing acquisition unit, and Figure 6B shows an image in which the contents of the symbol recognition result, attribute recognition result, and connection line recognition result are shown in text. [Figure 7] This figure illustrates a specific example of inference performed by the association recognition unit in Embodiment 1. [Figure 8] This figure illustrates another specific example of inference by the association recognition unit in Embodiment 1. [Figure 9] This figure illustrates an example of how the association recognition unit in Embodiment 1 absorbs variations in attribute notation. [Figure 10] This figure illustrates a specific example of the determination process performed by the determination unit in Embodiment 1. [Figure 11] This is a flowchart illustrating the operation of the drawing recognition device according to Embodiment 1. [Figure 12] This flowchart illustrates the details of the correspondence recognition process performed by the association recognition unit in step ST5 of Figure 11. [Figure 13] Figures 13A and 13B show an example of the hardware configuration of the drawing recognition device according to Embodiment 1. [Figure 14] This figure shows an example of the configuration of a learning device according to Embodiment 1. [Figure 15] This figure shows a specific example of learning when the association model generation unit generates the fourth machine learning model in Embodiment 1. [Figure 16] This is a flowchart illustrating the operation of the learning device according to Embodiment 1. [Figure 17]It is a flowchart for explaining the details of the generation process of the fourth machine learning model performed by the association model generation unit in step ST25 of FIG. 16. [Figure 18] FIGS. 18A and 18B are diagrams showing an example of the hardware configuration of the learning device 400 according to Embodiment 1.

Mode for Carrying Out the Invention

[0014] The user, for example, operates the keyboard 201 or mouse 202 to cause the drawing recognition device 100 to display a list of multiple image drawings 1 recorded in the drawing recording unit on the display 301. Then, the user, for example, operates the keyboard 201 or mouse 202 to select a desired image drawing 1 from the displayed list of image drawings 1.

[0015] The operation input device 200 receives the image drawing 1 selected by the user. The image drawing 1 received by the operation input device 200 is input to the drawing recognition device 100.

[0016] Here, Figure 2 is a diagram illustrating an example of Image Drawing 1 in Embodiment 1.

[0017] Image drawing 1 is, for example, a digital drawing of a paper drawing with a circuit diagram printed on it, such as the one shown in Figure 2, which has been scanned or otherwise digitized. Image drawing 1 may also be digital data created on a computer from a paper drawing with a circuit diagram printed on it, such as the one shown in Figure 2. The format of image drawing 1 is any image format, such as jpg or tiff. In Embodiment 1, as an example, image drawing 1 is digital drawing data of a circuit diagram.

[0018] The drawing recognition device 100 performs drawing recognition of the image drawing 1 based on the image drawing 1 input from the operation input device 200.

[0019] In Embodiment 1, drawing recognition performed by the drawing recognition device 100 refers to recognizing information about one or more symbols on the image drawing 1 (hereinafter referred to as "symbol information"), information about attributes associated with those symbols (hereinafter referred to as "attribute information"), information about connecting lines between those symbols (hereinafter referred to as "connection line information"), and the correspondence between symbols, attributes, and connecting lines.

[0020] Symbol information includes, for example, the number of symbols, the position information of each symbol, the type of each symbol, and the probability information of each symbol. Attribute information includes, for example, the number of attributes, the position information of each attribute, the type of each attribute, and the probability information of each attribute. Connecting line information includes the number of connecting lines, the position information of each connecting line, the type of each connecting line, and the probability information of each connecting line. In Embodiment 1, "probability" refers to the probability of certainty. For example, the probability of a symbol refers to the probability of it being a symbol, the probability of an attribute refers to the probability of it being an attribute, and the probability of a connecting line refers to the probability of it being a connecting line.

[0021] In Embodiment 1, the symbol in Image Drawing 1 refers to, for example, a part shown on Image Drawing 1. The attribute in Image Drawing 1 refers to, for example, a character associated with a symbol that identifies that symbol. Specifically, in Embodiment 1, the attribute refers to a character indicating the part's device number or part's name, etc. The connecting line in Image Drawing 1 refers to, for example, a line connecting symbols. In Embodiment 1, the connecting line also includes intersections.

[0022] The drawing recognition device 100 includes a drawing acquisition unit 101, a recognition unit 102, an inference unit storage unit 103, a determination unit 104, a parts recording unit 105, a confirmed data recording unit 106, an unconfirmed data recording unit 107, and a drawing display unit 108.

[0023] The inference unit 103 stores the first machine learning model 131, the second machine learning model 132, the third machine learning model 133, and the fourth machine learning model 134. The inference unit 103 also stores the transformation table 135.

[0024] The first machine learning model 131, the second machine learning model 132, and the third machine learning model 133 are models that have been trained to take image drawing 1 as input and output information related to the results of drawing recognition of image drawing 1.

[0025] Furthermore, the fourth machine learning model 134 is a model that has been trained to take information regarding the results of drawing recognition of image drawing 1 performed by the second machine learning model 132 as input, and to output information regarding the results of inferring the correspondence between attributes and symbols on image drawing 1, and the correspondence between attributes and connecting lines.

[0026] In Embodiment 1, "correspondence between attributes and symbols" means the association between attributes and symbols. In other words, in Embodiment 1, "correspondence between attributes and symbols" means which attributes are associated with which symbols.

[0027] Furthermore, in Embodiment 1, "correspondence between attributes and connecting lines" means the association between attributes and connecting lines. In other words, in Embodiment 1, "correspondence between attributes and connecting lines" means which attributes are associated with which connecting lines.

[0028] Furthermore, in Embodiment 1, the first machine learning model 131, the second machine learning model 132, the third machine learning model 133, and the fourth machine learning model 134 are collectively referred to simply as "machine learning models."

[0029] More specifically, in Embodiment 1, the first machine learning model 131 is a machine learning model that takes an image drawing 1 as input and outputs the result of recognizing symbol information on the image drawing 1 (hereinafter referred to as "symbol recognition result"). In other words, in Embodiment 1, the first machine learning model 131 is a machine learning model for symbol information recognition.

[0030] The second machine learning model 132 is a machine learning model that takes an image drawing 1 as input and outputs the result of recognizing attribute information associated with symbols on the image drawing 1 (hereinafter referred to as "attribute recognition result"). In other words, in Embodiment 1, the second machine learning model 132 is a machine learning model for attribute information recognition.

[0031] The third machine learning model 133 is a machine learning model that takes image drawing 1 as input and outputs the result of recognizing the connection line information between symbols on image drawing 1 (hereinafter referred to as "connection line recognition result"). In other words, in Embodiment 1, the third machine learning model 133 is a machine learning model for recognizing connection line information.

[0032] The fourth machine learning model 134 is a machine learning model that takes the attribute recognition results output by the second machine learning model 132 as input and outputs information regarding the results of inferring the correspondence between attributes and symbols on image drawing 1, and the correspondence between attributes and connecting lines. In other words, in Embodiment 1, the fourth machine learning model 134 is a machine learning model used for inferring the correspondence between attributes and symbols, and the correspondence between attributes and connecting lines.

[0033] The fourth machine learning model 134 may take numerical features obtained by transforming the attribute recognition results output by the second machine learning model 132 based on the transformation table 135 as input, and output information regarding the results of inferring the correspondence between attributes and symbols on the image drawing 1, and the correspondence between attributes and connecting lines. In the following embodiment 1, the fourth machine learning model 134 takes the above numerical features as input and outputs information regarding the results of inferring the correspondence between attributes and symbols on the image drawing 1, and the correspondence between attributes and connecting lines.

[0034] The conversion table 135 is a table configured to convert all attributes included in the attribute recognition results output by the second machine learning model 132 into numerical values.

[0035] The machine learning model is generated by the learning device 400 (see Figure 14 below) performing machine learning training. Details of the learning device 400 will be described later.

[0036] The drawing acquisition unit 101 acquires the image drawing 1 input from the user via the operation input device 200. The drawing acquisition unit 101 also performs predetermined processing on the acquired image drawing 1 to generate inference data.

[0037] As described above, the drawing recognition device 100 inputs the image drawing 1 into a machine learning model to obtain a drawing recognition result for the image drawing 1. However, there are limitations on the image size that can be used as input for the machine learning model. Specifically, for example, if the image drawing 1 is a Full High Definition (FHD) image with a size of 1920 x 1080 pixels, it exceeds the upper limit of the image size that can be used as input for the machine learning model.

[0038] Therefore, the drawing acquisition unit 101 divides the acquired image drawing 1 into data of, for example, 224 x 224 pixels in size, and uses the divided data as inference data. The intervals for dividing the image drawing 1 can be equal to 224 x 224 pixels in all directions (top, bottom, left, and right), and there may be overlapping parts when dividing the image drawing 1.

[0039] Furthermore, if the acquired image drawing 1 is within the size limits that allow it to be used as input for a machine learning model, the drawing acquisition unit 101 does not need to perform the processing described above, and can simply use the acquired image drawing 1 as inference data.

[0040] The drawing acquisition unit 101 outputs the inference data generated based on the image drawing 1 to the recognition unit 102.

[0041] In this example, the division of image drawing 1 is performed by the drawing acquisition unit 101, but this is only one example. For example, the division of image drawing 1 may be performed by the recognition unit 102, which will be described later. In this case, the drawing acquisition unit 101 outputs the acquired image drawing 1 as is to the recognition unit 102, which will then divide image drawing 1 as necessary when performing inference using a machine learning model.

[0042] The recognition unit 102 performs inference using the first machine learning model 131, the second machine learning model 132, or the third machine learning model 133 recorded in the inference unit storage unit 103, based on the inference data output from the drawing acquisition unit 101, thereby acquiring symbol recognition results, attribute recognition results, or connection line recognition results. As a result, the recognition unit 102 recognizes symbol information, attribute information, or connection line information. The recognition unit 102 outputs each acquired recognition result to the determination unit 104.

[0043] Furthermore, the recognition unit 102, based on the acquired symbol recognition results, attribute recognition results, and connection line recognition results, and the component information recorded in the component recording unit 105, uses the fourth machine learning model 134 and conversion table 135 recorded in the inference unit storage unit 103 to infer the correspondence between attributes and symbols, and the correspondence between attributes and connection lines, and obtains the inference results. Based on these inference results, the recognition unit 102 recognizes information regarding the correspondence between attributes, symbols, and connection lines, and generates a recognition result of the correspondence. The recognition unit 102 outputs the generated recognition result of the correspondence to the determination unit 104.

[0044] Furthermore, if unconfirmed data is recorded in the unconfirmed data recording unit 107, the recognition unit 102 retrieves the unconfirmed data from the unconfirmed data recording unit 107, and uses the fourth machine learning model 134 and the conversion table 135 to re-infer the correspondence between attributes and symbols, and the correspondence between attributes and connecting lines, for the image drawing 1 that was the source of the unconfirmed data, and retrieves the inference results again. Based on the re-retrieved inference results, the recognition unit 102 generates a recognition result again and outputs the generated recognition result to the determination unit 104.

[0045] The determination unit 104 comprehensively determines the appropriateness of the information regarding the correspondence between attributes, symbols, and connection lines in the inference data, based on the symbol recognition results, attribute recognition results, connection line recognition results, and correspondence relationship recognition results output by the recognition unit 102, and the component information recorded in the component recording unit 105.

[0046] If the determination unit 104 determines that the result is suitable, it generates a final determination result based on the symbol recognition result, attribute recognition result, connection line recognition result, and correspondence relationship inference result, and records the generated determination result as confirmed data in the confirmed data recording unit 106.

[0047] An example of a judgment result is shown in Figure 3. The judgment result shown in Figure 3 is information set for each symbol, including the symbol, the position of the symbol, the attribute associated with the symbol (device number), the symbol to which the symbol is connected, and the position of the connected symbol (hereinafter referred to as "associated judgment result information"). If there are multiple connected symbols, all of them are associated in the associated judgment result information. Furthermore, in the associated judgment result information, the position of the symbol and the position of the connected symbol are represented, for example, by the coordinates on Image Diagram 1 of the center of the rectangle in which the symbol and the connected symbol are recognized, respectively.

[0048] On the other hand, if the determination unit 104 determines that the result is negative, it records the symbol recognition result, attribute recognition result, connection line recognition result, and correspondence relationship inference result as undetermined data in the undetermined data recording unit 107.

[0049] The parts recording unit 105 records parts information. The parts information is information about the parts shown in the image diagram 1.

[0050] An example of part information is shown in Figure 4. Part information is part list data that includes the part's fixture number and part code, as shown in Figure 4. In Embodiment 1, parts are shown as symbols on Image Drawing 1. Also in Embodiment 1, the fixture number is shown as an attribute on Image Drawing 1.

[0051] Furthermore, in Embodiment 1, symbols (parts) are classified into categories, and a corresponding number is defined for each category. The corresponding number is, for example, the number of device numbers that can be associated with a symbol (part), or the number of pins that the symbol (part) has. For example, a symbol belonging to a certain category is defined as having one device number that can be associated with it, and two pins that this symbol has.

[0052] The confirmed data recording unit 106 records the confirmed data output by the determination unit 104.

[0053] The unconfirmed data recording unit 107 records the unconfirmed data output by the determination unit 104.

[0054] The drawing display unit 108 displays the confirmed data recorded in the confirmed data recording unit 106 on the display 301 of the display device 300.

[0055] Specifically, the drawing display unit 108 converts the finalized data into CAD data using, for example, a CAD application, and displays it on the display 301 of the display device 300.

[0056] Furthermore, the drawing display unit 108 displays the finalized data when the number of times the recognition unit 102 has performed inference on the correspondence between attributes and symbols, and the correspondence between attributes and connecting lines, exceeds the threshold number.

[0057] For example, if the number of times the recognition unit 102 performs the above-mentioned inference exceeds the threshold number, the control unit (not shown) of the drawing recognition device 100 notifies the drawing display unit 108 of this fact. When the drawing display unit 108 receives such notification from the control unit, it refers to the confirmed data recording unit 106 and displays the confirmed data.

[0058] Although the above example describes the control unit making the notification, the notification is not limited to the control unit; for example, the recognition unit 102 may also make the notification.

[0059] Next, a detailed example of the configuration of the recognition unit 102 of the drawing recognition device 100 will be explained with reference to Figure 5. For the sake of simplicity, Figure 5 shows only the drawing acquisition unit 101, the recognition unit 102, and the inference unit storage unit 103 from the example configuration of the drawing recognition device 100 shown in Figure 1.

[0060] As shown in Figure 5, the recognition unit 102 includes a symbol recognition unit 1021, an attribute information recognition unit 1022, a connection line recognition unit 1023, and an association recognition unit 1024.

[0061] The symbol recognition unit 1021 acquires symbol recognition results based on the inference data output from the drawing acquisition unit 101 and the first machine learning model 131.

[0062] Specifically, the symbol recognition unit 1021 analyzes the inference data and extracts image features from the inference data. The symbol recognition unit 1021 inputs the image features extracted from the inference data into the first machine learning model 131 to obtain the symbol recognition result.

[0063] In Embodiment 1, the symbol recognition unit 1021 acquires the number of symbols on the inference data, the position information of each symbol, the type of each symbol, and the accuracy of each symbol as symbol recognition results based on the inference data and the first machine learning model 131. The symbol recognition unit 1021 outputs the acquired symbol recognition results to the association recognition unit 1024 and the determination unit 104.

[0064] The attribute information recognition unit 1022 acquires attribute recognition results based on the inference data output from the drawing acquisition unit 101 and the second machine learning model 132.

[0065] Specifically, the attribute information recognition unit 1022 analyzes the inference data and extracts image features from it. The attribute information recognition unit 1022 inputs the image features extracted from the inference data into the second machine learning model 132 to obtain attribute recognition results.

[0066] In Embodiment 1, the attribute information recognition unit 1022 acquires the number of attributes on the inference data, the position information of each attribute, the type of each attribute, and the accuracy of each attribute as attribute recognition results based on the inference data and the second machine learning model 132. The attribute information recognition unit 1022 outputs the acquired attribute recognition results to the association recognition unit 1024 and the determination unit 104.

[0067] The connection line recognition unit 1023 acquires connection line recognition results based on the inference data output from the drawing acquisition unit 101 and the third machine learning model 133.

[0068] Specifically, the connection line recognition unit 1023 analyzes the inference data and extracts image features from the inference data. The connection line recognition unit 1023 inputs the image features extracted from the inference data into the third machine learning model 133 to obtain the connection line recognition result.

[0069] In Embodiment 1, the connection line recognition unit 1023 obtains connection line recognition results based on the inference data and the third machine learning model 133, including the number of connection lines on the inference data, the position information of each connection line, the type of each connection line, and the accuracy of each connection line. The connection line recognition unit 1023 outputs the obtained connection line recognition results to the association recognition unit 1024 and the determination unit 104.

[0070] Figure 6 shows an example image of the symbol recognition result acquired by the symbol recognition unit 1021, the attribute recognition result acquired by the attribute information recognition unit 1022, and the connection line recognition result acquired by the connection line recognition unit 1023 in Embodiment 1.

[0071] Figure 6A shows an image in which the symbol recognition results, attribute recognition results, and connection line recognition results are mapped to image drawing 1 acquired by the drawing acquisition unit 101. Figure 6B shows an image in which the contents of the symbol recognition results, attribute recognition results, and connection line recognition results are shown in text.

[0072] Note that Figures 6A and 6B are linked. Specifically, for example, the symbols shown as 611, 612, and 613 in Figure 6A correspond to symbol No. 1 "relay (a)", symbol No. 2 "relay (b)", and symbol No. 3 "fuse" in Figure 6B, respectively. Also, for example, the attributes shown as 621, 622, 623, and 624 in Figure 6A correspond to attribute No. 1 "30X004", attribute No. 2 "30X003", attribute No. 3 "FL3", and attribute No. 4 "FL4" in Figure 6B, respectively. Furthermore, for example, the connection line shown as 631 in Figure 6A corresponds to the connection line No. 2 in Figure 6B.

[0073] In Image Diagram 1, or in other words, the symbols, attributes, and connecting lines in the inference data are recognized as rectangular areas on Image Diagram 1.

[0074] Figure 6A shows an image in which rectangular regions representing recognized symbols, rectangular regions representing recognized attributes, and rectangular regions representing recognized connecting lines are mapped onto image drawing 1 acquired by the drawing acquisition unit 101.

[0075] In Figure 6B, for example, the symbol recognition result includes the symbol name and the position information of the recognized symbol. The position of the symbol is represented by the minimum and maximum X coordinates and the minimum and maximum Y coordinates of the rectangular area representing the symbol on Image Diagram 1. Note that the coordinates shown in Figure 6B are coordinates with the top left corner of the diagram as the origin in Figure 6A.

[0076] Furthermore, in Figure 6B, for example, the attribute recognition result is assumed to include the attribute, in other words, the character, and the position information of the recognized character. The position of the character is represented by the minimum and maximum X coordinates and the minimum and maximum Y coordinates of the rectangular area representing the character on Image Diagram 1.

[0077] Furthermore, in Figure 6B, for example, the connection line recognition result includes the type of connection line and the position information of the recognized connection line. The position of the connection line is represented by the minimum and maximum X coordinates and the minimum and maximum Y coordinates of the rectangular area indicating the connection line on Image Diagram 1.

[0078] Note that the symbol recognition results, attribute recognition results, and connection line recognition results shown in Figures 6A and 6B are merely examples. The symbol recognition results, attribute recognition results, and connection line recognition results may include information other than that shown in Figures 6A and 6B, such as IDs.

[0079] The association recognition unit 1024 uses the symbol recognition result output from the symbol recognition unit 1021, the attribute recognition result output from the attribute information recognition unit 1022, and the connection line recognition result output from the connection line recognition unit 1023, along with the component information recorded in the component recording unit 105, to infer the correspondence between attributes and symbols, and the correspondence between attributes and connection lines, using the fourth machine learning model 134 and the conversion table 135 recorded in the inference unit storage unit 103, and obtains the inference result.

[0080] Here, a specific example of inference by the association recognition unit 1024 will be explained with reference to Figure 7.

[0081] In Figure 7, reference numeral 701 indicates a symbol and attribute based on the symbol recognition result output from the symbol recognition unit 1021 and the attribute recognition result output from the attribute information recognition unit 1022.

[0082] For the sake of simplicity, this explanation will use the case of inferring the correspondence between attributes and symbols as an example, and the diagram shown in reference numeral 701 omits the illustration of the connection lines based on the connection line recognition results. However, the inference of the correspondence between attributes and connection lines can be performed in the same manner as the inference of the correspondence between attributes and symbols shown below.

[0083] The association recognition unit 1024 identifies one attribute to be used for inferring a correspondence relationship with a symbol from the diagram shown by reference numeral 701. The association recognition unit 1024 then acquires the identified attribute (hereinafter also referred to as the "main attribute") and the attributes that exist within a specific range centered on this main attribute (hereinafter also referred to as the "peripheral attributes") as a set of attributes. The specific range is determined, for example, based on the distance between the position information included in the attribute recognition result of the main attribute and the position information included in the attribute recognition result of the peripheral attributes.

[0084] For example, in the example shown in Figure 7, the association recognition unit 1024 sets the main attribute to "A1," and the peripheral attributes that exist within a specific range centered on this main attribute "A1" to "T001" and "V02A," and acquires these attributes as a set of attributes, as shown by reference numeral 702. Note that the main attribute "A1" corresponds to the device number in the part information shown in Figure 4.

[0085] Next, the association recognition unit 1024 sorts the peripheral attributes included in the acquired attribute group in order of their relative relevance to the main attribute, concatenates them to the main attribute, and generates a column (string) of attributes.

[0086] Here, the relationship to the main attribute is considered to be stronger the closer the attribute is to the coordinate position of the main attribute. For example, in the above example, of the two peripheral attributes, "T001" is closer to the main attribute "A1" than "V02A". Therefore, in the above example, "T001" has a stronger relationship to the main attribute "A1" than "V02A". Thus, the association recognition unit 1024 concatenates the peripheral attributes "T001" and "V02A" in that order to the main attribute "A1" and generates a column of attributes (string). When generating the column of attributes, the association recognition unit 1024 considers the strength of the relationships between the attributes and the strength of the relationships between the individual characters that make up each attribute, and generates a column of attributes that includes information on the strength of these relationships.

[0087] In the following, the attribute columns generated in this way will be denoted as, for example, [[A][1]][[T][0][0][1]][[V][0][2][A]]. Here, the space between adjacent parentheses contains information about the strength of the relationship between adjacent attributes and between adjacent characters that make up each attribute. For example, the notation [[A][1]][[T][0][0][1]] contains information about the strength of the relationship between the attribute column "A1" which indicates the primary attribute and the attribute column "T001" which indicates the peripheral attribute, as well as information about the strength of the relationships between the characters "A" and "1", "T" and "0", "0" and "0", and "0" and "1".

[0088] Next, the association recognition unit 1024 refers to the part information recorded in the part recording unit 105 and, based on the category to which the part (symbol) belongs and the corresponding number defined for each category, deletes from the generated attribute column any peripheral attributes that are determined to have a discrepancy in the corresponding number or peripheral attributes that are determined to have a low correlation with the main attribute.

[0089] For example, in the above example, suppose the device number "B1" is included in the peripheral attributes, and the association recognition unit 1024 generates a column (string) of attributes [[A][1]][[T][0][0][1]][[V][0][2][A]][[B][1]]. Also, suppose that in the part information recorded in the part recording unit 105, the category to which the part associated with the main attribute "A1" belongs is defined to have only one device number.

[0090] In this case, the attribute column [[A][1]][[T][0][0][1]][[V][0][2][A]][[B][1]] contains two fixture numbers, "A1" and "B1". This does not match the number "1" of fixture numbers defined in the category to which the part associated with the main attribute "A1" belongs. Therefore, the association recognition unit 1024 removes "B1" from the attribute column, making the attribute column [[A][1]][[T][0][0][1]][[V][0][2][A]].

[0091] In this way, the association recognition unit 1024 refers to the part information recorded in the part recording unit 105 and appropriately modifies the generated attribute column to match the number of corresponding elements defined in the part information. This prevents the association recognition unit 1024 from using an inappropriate attribute column in the inference process, thereby suppressing a decrease in inference accuracy.

[0092] Next, the association recognition unit 1024 decomposes the attribute column into character units and converts each decomposed character into a numerical feature based on the conversion table 135. In this way, the association recognition unit 1024 converts the generated attribute column into a numerical feature. This numerical feature includes information about the strength of the relationship between adjacent attributes and between adjacent characters that make up each attribute.

[0093] Next, the association recognition unit 1024 inputs the numerical features obtained by the above conversion into the fourth machine learning model 134 and obtains the result of inferring the correspondence between the principal attribute and the symbol, which is output from the fourth machine learning model 134.

[0094] The inference results include, for example, the names of symbols to which the primary attribute may be associated, the probability that the primary attribute is associated with that symbol, the distance between that symbol and the primary attribute, and the acceptance / rejection result, as shown in the figure shown in the figure 703. In the acceptance / rejection result, for example, in the figure shown in the figure 701, only the symbol with the closest distance to the primary attribute will be marked with "○", and all other symbols will be marked with "×". This is based on the idea that primary attributes tend to be associated with symbols that are closer to them than with symbols that are farther away, and that they tend to be associated with symbols that are closest to the primary attribute in particular.

[0095] Based on the inference result 703, the association recognition unit 1024 associates the main attribute with the symbol that is closest to the main attribute, i.e., the symbol whose acceptance result is "○". For example, in the example shown in Figure 7, the association recognition unit 1024 associates the main attribute "A1" with "ZZ symbol 2", whose inference result is "○". In this case, the distance can be, for example, the Euclidean distance, the Manhattan distance, or the Chebyshev distance.

[0096] Once the association recognition unit 1024 has associated the main attribute "A1" with "ZZ symbol 2" in this manner, it then repeats the same procedure while changing the main attribute from "A1" to another attribute. In this way, the association recognition unit 1024 associates all attributes included in the attribute group with the symbol. The association recognition unit 1024 then repeatedly performs the same process for all attribute groups on the image diagram 1, thereby associating all attributes on the image diagram 1 with the symbol.

[0097] Furthermore, while the above description illustrates an example of the association recognition unit 1024 associating attributes with symbols, the association recognition unit 1024 also associates attributes with connecting lines in the same manner. This allows the association recognition unit 1024 to recognize information regarding the correspondence between attributes, symbols, and connecting lines in the drawing data.

[0098] Next, we will explain another specific example of inference performed by the association recognition unit 1024 with reference to Figure 8.

[0099] For example, as shown by the symbol (a) in Figure 8, the association recognition unit 1024 sets the main attribute to "400V", and the peripheral attributes that exist within a specific range centered on this main attribute "400V" are "60Hz", "T1", "T2", "T3", "R", "S", and "T", and acquires these attributes as a set of attributes.

[0100] Next, the association recognition unit 1024, as shown by the symbol (b) in Figure 8, sorts the peripheral attributes included in the acquired attribute group in order of their relative relationship to the main attribute (for example, in order of proximity to the main attribute), and then links them to the main attribute to generate a column of attributes.

[0101] Here, the relative relationships of the peripheral attributes to the main attribute "400V" are assumed to be in the order of "R", "S", "60Hz", "T", "T1", "T2", and "T3". Therefore, the association recognition unit 1024 links the peripheral attributes to "400V" in the order described above, generating a sequence of attributes: [[4][0][0][V]][R][S][[6][0][H][z]][T][[T][1]][[T][2]][[T][3]].

[0102] Next, the association recognition unit 1024, as shown by the symbol (c) in Figure 8, refers to the part information recorded in the part recording unit 105, the category to which the part (symbol) belongs, and the corresponding number defined for each category, and deletes from the attribute column generated above any peripheral attributes that are determined to have a discrepancy in the corresponding number, or peripheral attributes that are determined to have a low correlation with the main attribute. For example, in the example in Figure 8, the peripheral attributes "T1", "T2", and "T3" are deleted, and the attribute column becomes [[4][0][0][V]][R][S][[6][0][H][z]][T].

[0103] Next, the association recognition unit 1024 decomposes the attribute column into character units and converts each decomposed character into a numerical value based on the conversion table 135. In this way, the association recognition unit 1024 converts the generated attribute column into numerical features. These numerical features include information about the strength of the relationships between adjacent attributes and between adjacent characters that constitute each attribute.

[0104] Next, the association recognition unit 1024 inputs the numerical features obtained by the above transformation into the fourth machine learning model 134 and obtains the result of inferring the correspondence between the main attribute and the symbol, which is output from the fourth machine learning model 134. Here, the association recognition unit 1024 obtains the inference result that the main attribute "400V" is classified as a "terminal class" with 80% certainty, that is, it is associated with a "terminal", as shown by the symbol (d) in Figure 8.

[0105] The association recognition unit 1024 then repeats the processing of codes (b) to (d) shown in the dotted box in Figure 8, while changing the main attribute to, for example, "60Hz". As a result, the association recognition unit 1024 obtains the inference result of which class each attribute belongs to, that is, which symbol it is associated with, for each attribute "400V", "60Hz", "T1", "T2", "T3", "R", "S", and "T", as shown in code (e) in Figure 8.

[0106] Furthermore, the association recognition unit 1024 recognizes information regarding the correspondence between attributes, symbols, and connecting lines in the drawing data by associating attributes and connecting lines in the same manner as described above.

[0107] In this way, the association recognition unit 1024 recognizes information regarding the correspondence between attributes, symbols, and connecting lines in the drawing data based on the symbol recognition result, attribute information recognition result, and connection line recognition result, as well as the fourth machine learning model. As a result, the drawing recognition device according to Embodiment 1 can recognize the correspondence between symbols, attributes, and connecting lines without requiring layout knowledge.

[0108] Furthermore, the association recognition unit 1024 identifies a primary attribute as described above, and generates a column of attributes based on the identified primary attribute and peripheral attributes, which are attributes that exist within a specific range centered on the primary attribute in the drawing shown in the drawing data. The association recognition unit 1024 then inputs the generated column of attributes into the fourth machine learning model 134 to obtain information about the symbol and connecting lines associated with the primary attribute. As a result, the drawing recognition device 100 according to Embodiment 1 can improve the recognition accuracy of information regarding the correspondence between attributes, symbols, and connecting lines in the drawing data.

[0109] To elaborate on this point, for example, the following methods can be considered for recognizing the correspondence between attributes, symbols, and connecting lines without requiring knowledge of placement.

[0110] Specifically, first, the association between symbols and attributes, and between connecting lines and attributes, is tentatively determined based on the distance between symbols and attributes on the drawing data, and the distance between connecting lines and attributes. Meanwhile, part information is pre-recorded in the part recording unit, and the degree of association between symbols and attributes, and between connecting lines and attributes, according to the characteristics of the drawing, is pre-recorded in the association degree recording unit. Then, based on the pre-recorded degree of association between symbols and attributes, and between connecting lines and attributes, according to the characteristics of the drawing, the appropriateness of the tentatively determined associations is judged, and the final correspondence between symbols, attributes, and connecting lines is recognized.

[0111] However, when associations are made based on relevance in this way, incorrect associations may occur if, for example, the same attribute is associated with different symbols or different connecting lines, potentially reducing the accuracy of the associations.

[0112] In this regard, in the drawing recognition device 100 according to Embodiment 1, the association recognition unit 1024, as described above, inputs a column of attributes generated based not only on the main attribute to be associated, but also on peripheral attributes, which are attributes that exist in a specific range centered on the main attribute in the drawing shown in the drawing data, into the fourth machine learning model 134 when performing association, and recognizes the correspondence relationship. As a result, in the drawing recognition device 100 according to Embodiment 1, accurate association is possible even when the same attribute is associated with different symbols or connecting lines, and a decrease in the recognition accuracy of information regarding the correspondence relationship between attributes, symbols and connecting lines in the drawing data can be suppressed.

[0113] Furthermore, when the association recognition unit 1024 infers the association between the main attribute and the symbol, as described above, it converts the generated column of attributes into numerical features based on the conversion table 135 and inputs these numerical features into the fourth machine learning model 134. This allows the association recognition unit 1024 to absorb variations in the notation of the attributes. This absorption of variations in the notation of attributes will be explained in detail using Figure 9.

[0114] In Figure 9, reference numeral 901 indicates an example of a column of attributes generated by the association recognition unit 1024 in the procedure described above. For the sake of simplicity, the column of attributes is assumed to be [U][0][1][2][Z], with "U" being the primary attribute. In Figure 9, the above column of attributes is simply denoted as "U012Z".

[0115] The association recognition unit 1024 converts the generated attribute column [U][0][1][2][Z] into numerical features based on the conversion table 135.

[0116] The conversion table 135 is a table that can convert all attributes (characters) included in the attribute recognition result into numerical values, and is a table in which numerical values ​​are associated with each attribute, as shown in Figure 9 for example.

[0117] The association recognition unit 1024 decomposes the generated attribute column into character units and converts each decomposed character into a numerical value based on the conversion table 135.

[0118] For example, the association recognition unit 1024 decomposes the generated attribute column "U012Z" into "U", "0", "1", "2", and "Z", and converts "U" to "30", "0" to "0", "1" to "1", "2" to "2", and "Z" to "35" based on the conversion table 135. Then, the association recognition unit 1024 concatenates each of the converted values ​​to obtain the numerical features "30", "0", "1", "2", and "35".

[0119] The association recognition unit 1024 then inputs the numerical features obtained in this way into the fourth machine learning model 134.

[0120] On the other hand, the fourth machine learning model 134, as shown by the symbol 902 in Figure 9, learns the correspondence between a column of attributes, the symbols associated with the primary attributes contained in that column of attributes, and numerical features obtained by transforming that column of attributes based on the transformation table 135. These numerical features include information about the strength of the relationships between adjacent attributes and between adjacent characters that constitute each attribute.

[0121] Furthermore, when a numerical feature is input to the fourth machine learning model 134, it outputs not only symbols associated with the same numerical feature as the input, but also symbols associated with numerical features similar to the input as part of the inference results.

[0122] For example, in the example in Figure 9, the fourth machine learning model 134 has not learned the attribute column [U][0][1][2][Z] and the numerical features "30", "0", "1", "2", and "35", but it has learned numerical features similar to these features, namely "30", "0", "1", "1", and "35", as well as the attribute column [U][0][1][1][Z] from which these numerical features originate.

[0123] In this case, even if the numerical features "30", "0", "1", "2", and "35" are input to the fourth machine learning model 134, it will treat them as if they were input and be able to output an "XX symbol" associated with the attribute column [U][0][1][1][Z] as the inference result (code 903).

[0124] In other words, the association recognition unit 1024 can absorb variations in attribute notation by converting the generated attribute column into numerical features based on the conversion table 135 and inputting these numerical features into the fourth machine learning model 134. As a result, the drawing recognition device according to Embodiment 1 can suppress the decrease in recognition accuracy of information regarding the correspondence between attributes, symbols, and connecting lines in drawing data due to variations in attribute notation.

[0125] In the example above, the extent to which the fourth machine learning model 134 outputs symbols containing numerical features close to those input to the fourth machine learning model 134 can be arbitrarily set through the design.

[0126] Next, the determination process performed by the determination unit 104 will be explained in detail using Figure 10. Figure 10 is an explanatory diagram showing a specific example of the process in Embodiment 1 in which the determination unit 104 comprehensively determines whether the information regarding the recognition result of the correspondence relationship between attributes, symbols, and connecting lines in the inference data is appropriate or not.

[0127] Here, for example, the symbol recognition results, attribute recognition results, and connection line recognition results output from the recognition unit 102 are assumed to be as shown in Figure 6, and the content of the recognition results shown in Figure 10 is the same as the content of the recognition results shown in Figure 6. Also, here, for example, the component information stored in the component recording unit 105 is assumed to be as shown in Figure 4, and the content of the component information shown in Figure 10 is the same as the content of the component information shown in Figure 4.

[0128] The determination unit 104 first refers to the parts recording unit 105 and compares the parts information with the attribute recognition results to determine whether the attributes included in the attribute recognition results match the equipment numbers defined in the parts information. If the attributes included in the attribute recognition results do not match the equipment numbers defined in the parts information, the determination unit 104 deletes those attributes from the attribute recognition results.

[0129] For convenience, here we will assume that "30X004", "30X003", "FL3", and "FL4" included in the attribute recognition results all correspond to the device numbers defined in the part information.

[0130] Furthermore, the determination unit 104 determines whether the information regarding the correspondence between attributes, symbols, and connecting lines recognized by the association recognition unit 1024 is appropriate, based on the determination information set in advance for determining the appropriateness of the information regarding the correspondence between attributes, symbols, and connecting lines in the drawing data. Here, the determination information is, for example, the number of correspondences defined for each category to which a part (symbol) belongs in the part information recorded in the part recording unit 105.

[0131] For example, suppose that in the component information, the number of pins is defined as "2" as the corresponding number for each category to which a certain symbol belongs, while the recognition result of the correspondence relationship indicates that two other symbols are associated with this symbol. In this case, there is no discrepancy between the recognition result of the correspondence relationship and the number of pins defined for each category to which the symbol belongs. Therefore, the determination unit 104 determines that the information regarding the recognition result of the correspondence relationship is "appropriate". Furthermore, the determination unit 104 generates a final determination result (see Figure 3) based on the recognition result of the correspondence relationship, the symbol recognition result, the attribute recognition result, and the connection line recognition result, which have been determined to be "appropriate", and records the generated determination result as confirmed data in the confirmed data recording unit 106.

[0132] On the other hand, for example, in the component information, the number of pins is defined as "2" as the corresponding number for each category to which a certain symbol belongs, but in the correspondence recognition result, three other symbols are associated with this symbol. In this case, there is a discrepancy between the correspondence recognition result and the corresponding number defined for each category to which the symbol belongs. Therefore, the determination unit 104 determines that the information regarding the correspondence recognition result is "no". The determination unit 104 also records the correspondence recognition result, symbol recognition result, attribute recognition result, and connection line recognition result that were determined to be "no" as undetermined data in the undetermined data recording unit 107.

[0133] The unconfirmed data recorded in the unconfirmed data recording unit 107 is acquired by the association recognition unit 1024. The association recognition unit 1024 then re-recognizes the correspondence between attributes, symbols, and connecting lines based on this unconfirmed data.

[0134] In this way, the determination unit 104 determines whether the information regarding the correspondence between attributes, symbols, and connecting lines recognized by the association recognition unit 1024 is appropriate or not, based on the determination information set in advance for determining whether the information regarding the correspondence between attributes, symbols, and connecting lines in the drawing data is appropriate or not. As a result, the drawing recognition device according to Embodiment 1 can improve the accuracy of recognizing the correspondence between attributes, symbols, and connecting lines.

[0135] Furthermore, the unconfirmed data recorded in the unconfirmed data recording unit 107 is acquired by the association recognition unit 1024 and used for the association recognition unit 1024 to re-recognize the correspondence between attributes, symbols, and connecting lines. As a result, the drawing recognition device according to Embodiment 1 can improve the accuracy of recognizing the correspondence between attributes, symbols, and connecting lines.

[0136] In the above example, the case described was one in which the determination information used by the determination unit 104 for determination is a correspondence number defined for each category to which the component (symbol) belongs. However, this is just one example, and the determination information may be other information than the above correspondence number.

[0137] Next, the operation of the drawing recognition device 100 according to Embodiment 1 will be described. Figure 11 is a flowchart illustrating the operation of the drawing recognition device 100 according to Embodiment 1.

[0138] The drawing acquisition unit 101 acquires the image drawing 1 input from the user via the operation input device 200 (step ST1). The drawing acquisition unit 101 performs predetermined processing on the acquired image drawing 1 and generates inference data. The drawing acquisition unit 101 also outputs the inference data generated based on the image drawing 1 to the recognition unit 102.

[0139] Next, the recognition unit 102 performs inference using the first machine learning model 131, the second machine learning model 132, and the third machine learning model 133 stored in the inference unit 103, based on the inference data output from the drawing acquisition unit 101 in step ST1, thereby acquiring symbol recognition results, attribute recognition results, and connection line recognition results. As a result, the recognition unit 102 recognizes symbol information, attribute information, and connection line information.

[0140] Specifically, the symbol recognition unit 1021 acquires the symbol recognition result based on the inference data output from the drawing acquisition unit 101 in step ST1 and the first machine learning model 131.

[0141] More specifically, the symbol recognition unit 1021 inputs image features extracted from the inference data into the first machine learning model 131 to obtain a symbol recognition result (step ST2). The symbol recognition unit 1021 outputs the obtained symbol recognition result to the association recognition unit 1024 and the determination unit 104.

[0142] Furthermore, the attribute information recognition unit 1022 acquires attribute recognition results based on the inference data output from the drawing acquisition unit 101 in step ST1 and the second machine learning model 132.

[0143] More specifically, the attribute information recognition unit 1022 inputs image features extracted from the inference data into the second machine learning model 132 to obtain attribute recognition results (step ST3). The attribute information recognition unit 1022 outputs the obtained attribute recognition results to the association recognition unit 1024 and the determination unit 104.

[0144] Furthermore, the connection line recognition unit 1023 acquires connection line recognition results based on the inference data output from the drawing acquisition unit 101 in step ST1 and the third machine learning model 133.

[0145] More specifically, the connection line recognition unit 1023 inputs image features extracted from the inference data into the third machine learning model 133 to obtain connection line recognition results (step ST4). The connection line recognition unit 1023 outputs the obtained connection line recognition results to the association recognition unit 1024 and the determination unit 104.

[0146] Steps ST2 to ST4 may be performed in any order, and may also be performed in parallel.

[0147] Next, the association recognition unit 1024 uses the fourth machine learning model 134 and the conversion table 135 stored in the inference unit 103 to infer the correspondence between attributes and symbols, and the correspondence between attributes and connection lines, based on the symbol recognition results, attribute recognition results, and connection line recognition results output in steps ST2 to ST4, respectively, and obtains the inference results. Furthermore, the association recognition unit 1024 recognizes the correspondence between attributes, symbols, and connection lines based on the obtained inference results (step ST5), and outputs the recognition result of the correspondence to the determination unit 104.

[0148] Next, the determination unit 104 acquires the symbol recognition results, attribute recognition results, and connection line recognition results output in steps ST2 to ST4, respectively, and the correspondence relationship recognition results output in step ST5 (step ST6).

[0149] Next, the determination unit 104 refers to the parts recording unit 105 and compares the parts information with the recognition result obtained in step ST6 (step ST7).

[0150] Specifically, the determination unit 104 refers to the part recording unit 105 and compares the part information with the attribute recognition result obtained in step ST6 to determine whether the attribute included in the attribute recognition result matches the fixture number defined in the part information. If the attribute included in the attribute recognition result does not match the fixture number defined in the part information, the determination unit 104 deletes the attribute from the attribute recognition result.

[0151] Furthermore, the determination unit 104 determines whether the information regarding the recognition result of the correspondence relationship between attributes, symbols, and connecting lines obtained in step ST6 is appropriate (step ST8), based on pre-set determination information, for example, the number of correspondences defined for each category to which the part (symbol) belongs in the part information recorded in the part recording unit 105.

[0152] If the result is "suitable" (if the answer is "YES" in step ST8), the determination unit 104 generates a final determination result based on the correspondence recognition result, symbol recognition result, attribute recognition result, and connection line recognition result obtained in step ST6, and records the generated determination result as confirmed data in the confirmed data recording unit 106 (step ST9).

[0153] On the other hand, if the answer is "No" (if the answer is "NO" in step ST8), the determination unit 104 records the correspondence recognition result, symbol recognition result, attribute recognition result, and connection line recognition result obtained in step ST6 as undetermined data in the undetermined data recording unit 107 (step ST10).

[0154] Next, the determination unit 104 determines whether the number of times the series of processes from steps ST5 to ST8 have been performed exceeds the threshold number (step ST11). If the determination result is that the number of times the series of processes from steps ST5 to ST8 have been performed does not exceed the threshold number (if the result of step ST11 is "NO"), the process proceeds to step ST12.

[0155] On the other hand, if it is determined that the number of times the series of processes from steps ST5 to ST8 has been executed exceeds the threshold number (i.e., if the answer to step ST11 is "YES"), the process proceeds to step ST13.

[0156] In step ST12, the association recognition unit 1024 acquires the unconfirmed data recorded in the unconfirmed data recording unit 107 (step ST12). Then, the process returns to step ST5, where the association recognition unit 1024 uses the fourth machine learning model 134 and the conversion table 135 stored in the inference unit storage unit 103, based on the unconfirmed data acquired in step ST12, to re-infer the correspondence between attributes and symbols, and the correspondence between attributes and connecting lines, and re-acquires the inference results. At this time, since attributes that do not match the equipment number defined in the part information have been removed from the attribute recognition results included in the unconfirmed data in step ST7, the accuracy of the re-inference is improved.

[0157] Note that step ST12 described above may be omitted if no unconfirmed data is recorded in the unconfirmed data recording unit 107. In that case, the process proceeds to step ST13.

[0158] In step ST13, the drawing display unit 108 generates drawing data based on the finalized data recorded in the finalized data recording unit 106, and outputs the generated drawing data to the display device 300 (step ST13).

[0159] Next, we will explain the details of the correspondence recognition process performed by the association recognition unit 1024 in step ST5 of Figure 11, referring to the flowchart in Figure 12.

[0160] First, the association recognition unit 1024 identifies one main attribute to be used for inferring a correspondence relationship with a symbol, and acquires this main attribute and peripheral attributes that exist within a specific range centered on this main attribute as a set of attributes (step ST51). Then, the association recognition unit 1024 rearranges the peripheral attributes included in the acquired attribute set in order of their relative relevance to the main attribute, concatenates them to the main attribute, and generates a column of attributes (string).

[0161] Next, the association recognition unit 1024 refers to the part recording unit 105 and, referring to the category to which the part (symbol) belongs as defined in the part information, and the corresponding number defined for each category, deletes from the generated attribute column any peripheral attributes that are determined to have a discrepancy in the corresponding number, or peripheral attributes that are determined to have a low correlation with the main attribute (step ST52).

[0162] Next, the association recognition unit 1024 decomposes the attribute column into character units and converts each decomposed character into a numerical feature based on the conversion table 135, thereby converting the generated attribute column into a numerical feature (step ST53).

[0163] Next, the association recognition unit 1024 inputs the numerical features obtained by the above conversion into the fourth machine learning model 134 and obtains the result of inferring the correspondence between the main attribute and the symbol or connecting line (e.g., the classification result) output from the fourth machine learning model 134 (step ST54).

[0164] Next, the association recognition unit 1024 associates the main attribute with the symbol or connecting line based on the inference result obtained above (step ST55).

[0165] The association recognition unit 1024 recognizes the correspondence between attributes, symbols, and connecting lines by repeating steps ST52 to ST55 while changing the main attribute.

[0166] In the above embodiment 1, the drawing recognition device 100 may be provided with an interface that allows the user to select the necessary recognition result from the symbol recognition result, attribute recognition result, connection line recognition result, or correspondence relationship recognition result output by the recognition unit 102.

[0167] Specifically, for example, the display control unit (not shown) of the drawing recognition device 100 displays each recognition result output by the recognition unit 102 on the display 301 of the display device 300. The user understands the content of each recognition result by checking the display 301. If the user finds, for example, a clearly erroneous symbol, attribute, or connecting line, they operate the keyboard 201 or mouse 202 of the operation input device 200 to input a command to delete the clearly erroneous symbol, attribute, or connecting line. The control unit of the drawing recognition device 100 receives the command input by the user and, based on the received command, deletes the information related to the symbol, attribute, or connecting line from the recognition result. With this configuration, the drawing recognition device 100 can select the information related to the symbol, attribute, or connecting line that the user needs.

[0168] Furthermore, in the above embodiment 1, the inference unit storage unit 103 is provided in the drawing recognition device 100, but this is merely an example. The inference unit storage unit 103 may be provided outside the drawing recognition device 100, in a location accessible to the drawing recognition device 100. Specifically, for example, one or more network storage devices (not shown) located on a communication network may have the function of the inference unit storage unit 103 and store the first machine learning model 131, the second machine learning model 132, the third machine learning model 133, the fourth machine learning model 134, and the conversion table 135. The drawing recognition device 100 accesses the network storage devices. As a result, the external network storage devices can store the first machine learning model 131, the second machine learning model 132, the third machine learning model 133, the fourth machine learning model 134, and the conversion table 135, and build a database outside the drawing recognition device 100.

[0169] Furthermore, in the above embodiment 1, the parts recording unit 105, the confirmed data recording unit 106, and the unconfirmed data recording unit 107 are provided in the drawing recognition device 100, but this is merely one example. The parts recording unit 105, the confirmed data recording unit 106, and the unconfirmed data recording unit 107 may be provided outside the drawing recognition device 100, in a location accessible to the drawing recognition device 100.

[0170] Furthermore, in the above embodiment 1, the drawing recognition device 100 is provided with a drawing acquisition unit 101, a recognition unit 102, an inference unit storage unit 103, a determination unit 104, a parts recording unit 105, a confirmed data recording unit 106, a non-confirmed data recording unit 107, and a drawing display unit 108, but this is merely an example. The drawing recognition device 100 only needs to include at least a drawing acquisition unit 101 and a recognition unit 102. In this case, for example, the determination unit 104 and the drawing display unit 108 may be provided in an external device (a drawing generation device; not shown) of the drawing recognition device 100, and the drawing recognition device 100 and the drawing generation device may constitute a drawing generation system. The inference unit storage unit 103, parts recording unit 105, confirmed data recording unit 106, and non-confirmed data recording unit 107 may be provided, for example, in a server that can be accessed by the drawing recognition device 100 and the drawing generation device.

[0171] Furthermore, the above description illustrates an example in which the association recognition unit 1024 infers associations using only the fourth machine learning model 134. However, the association recognition unit 1024 may also infer associations by using statistical information or other information in addition to the fourth machine learning model 134, for example, by a method such as majority voting.

[0172] Furthermore, while the above description illustrates an example where the association recognition unit 1024 associates a primary attribute with a symbol or connecting line, the objects to which primary attributes can be associated are not limited to these. For example, the association recognition unit 1024 may associate primary attributes with other elements other than symbols or connectivity, such as geometric shapes present on the drawing.

[0173] Furthermore, it is not necessary for the association recognition unit 1024 to perform all associations. For example, in cases where attributes and symbols or connecting lines overlap on a drawing, the user of the drawing recognition device 100 may pre-associate the attributes with the symbols or connecting lines. In addition, if an attribute that is clearly going to be associated in advance is included in an attribute group consisting of a main attribute and peripheral attributes, a mechanism may be incorporated to increase the probability that the attribute will be associated with the symbol or connecting line.

[0174] Furthermore, the above description illustrates an example in which the association recognition unit 1024 converts the generated attribute column into numerical features based on the conversion table 135 before inputting them into the fourth machine learning model 134. However, the association recognition unit 1024 is not limited to this; it may also perform arithmetic operations on the numerical features according to predefined rules before inputting them into the fourth machine learning model 134.

[0175] Furthermore, the drawing recognition device 100 according to Embodiment 1 is suitable for use, for example, in restoring CAD data from paper drawings or converting CAD data for different CAD systems.

[0176] Specifically, for example, a user inputs a printed document created in a specific CAD system as image drawing 1 into the drawing recognition device 100. The drawing recognition device 100 recognizes symbols, attributes, or connecting lines present on the input image drawing 1 and records matching data. By inputting the recorded matching data into the specific CAD system, the user can restore the printed document as CAD data and reuse that CAD data for design modifications or improvements, etc.

[0177] Furthermore, users can convert the aforementioned matching data into CAD data usable in other CAD systems by, for example, providing a separate program for converting to data formats of different CAD systems.

[0178] Thus, the drawing recognition device 100 according to Embodiment 1 can be used, for example, in a CAD system to restore CAD data, thereby improving the accuracy of CAD data restoration in the CAD system.

[0179] Figures 13A and 13B show an example of the hardware configuration of the drawing recognition device 100 according to Embodiment 1. In Embodiment 1, the functions of the drawing acquisition unit 101, the recognition unit 102, the determination unit 104, and the drawing display unit 108 are realized by the processing circuit 1301. That is, the drawing recognition device 100 includes a processing circuit 1301 for recognizing symbols and the like on the input image drawing 1. The processing circuit 1301 may be dedicated hardware as shown in Figure 12A, or it may be a CPU (Central Processing Unit) 1304 that executes a program stored in memory 1305 as shown in Figure 12B.

[0180] If the processing circuit 1301 is dedicated hardware, it may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0181] When the processing circuit 1301 is the CPU 1304, the functions of the drawing acquisition unit 101, the recognition unit 102, the determination unit 104, and the drawing display unit 108 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 1305. The processing circuit 1301 executes the functions of the drawing acquisition unit 101, the recognition unit 102, the determination unit 104, and the drawing display unit 108 by reading and executing the program stored in the memory 1305. In other words, the drawing recognition device 100 includes a memory 1305 for storing a program that, when executed by the processing circuit 1301, will result in the execution of steps ST1 to ST13 in Figure 11 and steps ST51 to ST55 in Figure 12. Furthermore, the program stored in memory 1305 can be said to cause the computer to execute the procedures or methods of the drawing acquisition unit 101, the recognition unit 102, the determination unit 104, and the drawing display unit 108. Here, memory 1305 refers to non-volatile or volatile semiconductor memory such as RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), or magnetic disks, flexible disks, optical disks, compact disks, minidiscs, DVDs (Digital Versatile Discs), etc.

[0182] Furthermore, the functions of the drawing acquisition unit 101, the recognition unit 102, the determination unit 104, and the drawing display unit 108 may be partially implemented by dedicated hardware and partially by software or firmware. For example, the drawing acquisition unit 101 can be implemented by a processing circuit 1301 as dedicated hardware, while the recognition unit 102, the determination unit 104, and the drawing display unit 108 can be implemented by the processing circuit 1301 reading and executing a program stored in memory 1305. Furthermore, the inference unit storage unit 103, component recording unit 105, confirmed data recording unit 106, and unconfirmed data recording unit 107 may use, for example, an HDD. This is just one example, and the inference unit storage unit 103, component recording unit 105, confirmed data recording unit 106, and unconfirmed data recording unit 107 may be composed of memory 1305, SSD (Solid State Drive), or DVD, etc. Furthermore, the drawing recognition device 100 includes an operation input device 200 or a display device 300, as well as an input interface device 1302 and an output interface device 1303 that perform wired or wireless communication.

[0183] In Embodiment 1, the first machine learning model 131, the second machine learning model 132, the third machine learning model 133, and the fourth machine learning model 134 are generated by the learning device 400 through machine learning training. The learning device 400 will be described below.

[0184] Figure 14 shows an example of the configuration of the learning device 400 according to Embodiment 1. The learning device 400 is connected to the drawing recognition device 100 via a network. The learning device 400 generates a first machine learning model 131, a second machine learning model 132, a third machine learning model 133, a fourth machine learning model 134, and a conversion table 135 based on the input image drawing 1.

[0185] Image diagram 1 is input along with the teacher label. The learning device 400 trains a machine learning model composed of a neural network using so-called supervised learning. The teacher label is information indicating the symbol, attributes, and connection lines on image diagram 1.

[0186] Regarding teacher labels, for example, a teacher label for a symbol is text data that associates the coordinates indicating the symbol's position on Image Diagram 1 with information that identifies that symbol. The symbol's position is represented, for example, by the coordinates on Image Diagram 1 of the four corners of a rectangle that encloses the symbol. The information that identifies the symbol may be, for example, the symbol's name.

[0187] Teacher labels for attributes and teacher labels for connecting lines, like teacher labels for symbols, are text data that associates coordinates indicating the location of an attribute or connecting line on Image Drawing 1 with information that identifies that attribute or connecting line. Note that teacher labels are pre-generated for each Image Drawing 1, for example, by the user.

[0188] The learning device 400 stores the generated first machine learning model 131, second machine learning model 132, third machine learning model 133, fourth machine learning model 134, and conversion table 135 in the inference unit storage unit 103 of the drawing recognition device 100.

[0189] Note that in Figure 14, for the sake of simplicity, only the inference unit storage unit 103 and the parts recording unit 105 are shown as components of the drawing recognition device 100. However, the configuration example of the drawing recognition device 100 is the same as the configuration example of the drawing recognition device 100 explained using Figure 1.

[0190] Furthermore, as shown in Figure 14, the learning device 400 is a separate device from the drawing recognition device 100 and is connected to the drawing recognition device 100, but this is merely one example. The learning device 400 may also be mounted on the drawing recognition device 100.

[0191] The learning device 400 includes a learning data acquisition unit 401, a symbol model generation unit 402, an attribute information model generation unit 403, a connection line model generation unit 404, and an association model generation unit 405.

[0192] The training data acquisition unit 401 performs data acquisition processing for training.

[0193] Specifically, the training data acquisition unit 401 acquires multiple image drawings 1 to which teacher labels have been assigned. The teacher labels are, for example, text data as described above. Specifically, the training data acquisition unit 401 acquires multiple image drawings 1 and the text data for each image drawing 1.

[0194] Furthermore, multiple image diagrams 1, each assigned a teacher label, are input by the user. For example, the learning device 400 is connected to an operation input device 200 (see Figure 1), and the user inputs multiple image diagrams 1 from the operation input device 200.

[0195] Then, the training data acquisition unit 401 generates training data based on the acquired image diagram 1.

[0196] Specifically, the training data acquisition unit 401 first divides the acquired image drawing 1 into data of, for example, 224 x 224 pixels in size. The division interval of the image drawing 1 can be equal to 224 x 224 pixels in all directions (top, bottom, left, and right), and there may be overlapping parts when dividing the image drawing 1. Note that the training data acquisition unit 401 does not need to perform the above processing if, for example, the acquired image drawing 1 is within the size limit that can be used as input for a machine learning model.

[0197] The training data acquisition unit 401 generates the segmented image drawing 1, to which teacher labels have been assigned, as training data. The training data acquisition unit 401 outputs the generated training data to the symbol model generation unit 402, the attribute information model generation unit 403, the connection line model generation unit 404, and the association model generation unit 405.

[0198] In this example, the training data acquisition unit 401 divides the image diagram 1, but this is just one example. For example, the division of the image diagram 1 as described above may be performed by the symbol model generation unit 402, attribute information model generation unit 403, connection line model generation unit 404, and association model generation unit 405, which will be described later. In this case, the training data acquisition unit 401 includes the acquired image diagram 1 as training data, and the symbol model generation unit 402, attribute information model generation unit 403, connection line model generation unit 404, and association model generation unit 405 each divide the image diagram 1 when generating the machine learning model.

[0199] The symbol model generation unit 402 learns symbol recognition based on the training data output by the training data acquisition unit 401, and takes the image drawing 1 as input to generate a first machine learning model 131 that outputs information such as the number of symbols on the image drawing 1, the position information of each symbol, the type of each symbol, and the accuracy of each symbol as a symbol recognition result.

[0200] In Embodiment 1, the symbol model generation unit 402 uses the training data output by the training data acquisition unit 401 to perform training related to object detection of the neural network. That is, it performs training by extracting image features, classifying them, and correcting coordinates. As for the training method, well-known training methods such as SSD (Single Shot Detector) and YOLO (You Only Look Once) may be used.

[0201] The symbol model generation unit 402 stores the generated first machine learning model 131 in the inference unit 103.

[0202] The attribute information model generation unit 403 learns attribute determination based on the training data output by the training data acquisition unit 401, and takes the image drawing 1 as input to generate a second machine learning model 132 that outputs information such as the number of attributes on the image drawing 1, the position information of each attribute, the type of each attribute, and the accuracy of each attribute as attribute recognition results.

[0203] In Embodiment 1, the attribute information model generation unit 403 uses the training data output by the training data acquisition unit 401 to perform training related to object detection of the neural network. That is, it performs training by extracting image features, classifying them, and correcting coordinates. As for the training method, a well-known training method such as VGSL (Variable-size Graph Specification Language) may be used.

[0204] The attribute information model generation unit 403 stores the generated second machine learning model 132 in the inference unit 103.

[0205] The connection line model generation unit 404 learns how to determine connection lines based on the training data output by the training data acquisition unit 401, and takes the image drawing 1 as input to generate a third machine learning model 133 that outputs information such as the number of connection lines on the image drawing 1, the position information of each connection line, the type of each connection line, and the accuracy of each connection line as connection line recognition results.

[0206] In Embodiment 1, the connection line model generation unit 404 uses the training data output by the training data acquisition unit 401 to perform training related to object detection of the neural network. That is, it performs training by extracting image features, classifying them, and correcting the coordinates. As for the training method, well-known training methods such as SSD (Single Shot Detector) and YOLO (You Only Look Once) can be used.

[0207] The connection line model generation unit 404 stores the generated third machine learning model 133 in the inference unit 103.

[0208] The association model generation unit 405 generates a conversion table 135 that can convert all attributes (characters) included in the attribute information into numerical values, based on the training data output by the training data acquisition unit 401. This conversion table 135 is, for example, a table in which numerical values ​​are associated with each attribute included in the attribute recognition result output by the second machine learning model 132.

[0209] Furthermore, the association model generation unit 405 learns the correspondence between attributes and symbols, and the correspondence between attributes and connecting lines, using the conversion table 135 described above, based on the training data output by the training data acquisition unit 401. As a result, the association model generation unit 405 generates a fourth machine learning model 134 that takes attribute information as input and outputs the results of inferring the correspondence between attributes and symbols on the image drawing 1, and the correspondence between attributes and connecting lines.

[0210] In Embodiment 1, the association model generation unit 405 uses the training data output by the training data acquisition unit 401 to perform training related to the classification of the neural network. That is, the association model generation unit 405 performs class classification and modification on the symbols and connecting lines associated with a certain attribute (main attribute) on the image diagram 1, based on a sequence of attributes including that attribute and the surrounding attributes (peripheral attributes) on the image diagram 1. As for the training method, multi-class classification can be performed using well-known training methods such as LSTM (Long Short-Term Memory).

[0211] The association model generation unit 405 stores the generated fourth machine learning model 134 and the transformation table 135 in the inference unit 103.

[0212] Here, Figure 15 shows a specific example of the learning process when the association model generation unit 405 generates the fourth machine learning model 134.

[0213] In Figure 15, reference numeral 1501 indicates the training data output by the training data acquisition unit 401. In the training data 1501, one or more attributes are pre-associated with each symbol by the user, as shown by the dotted line frame in Figure 15.

[0214] When multiple attributes are associated with a single symbol, these multiple attributes include the attribute that is associated with the symbol (the primary attribute) and the attributes that exist within a specific range centered on the primary attribute (the peripheral attributes). For example, in the example in the upper left frame of Figure 15, the primary attribute associated with symbol S is "A1", and its peripheral attributes are "OP12" and "10Ω".

[0215] The association model generation unit 405 acquires the primary and peripheral attributes contained within the dotted frame in the training data 1501 as a set of attributes. In the example above, the association model generation unit 405 acquires the three attributes "A1", "OP12", and "10Ω" as a set of attributes.

[0216] Next, the association model generation unit 405 refers to the part recording unit 105 and compares the part information with the attributes included in the acquired attribute group to determine whether the attributes included in the acquired attribute group match the fixture number defined in the part information. If the attributes included in the acquired attribute information do not match the fixture number defined in the part information, the association model generation unit 405 discards the attribute from the acquired attribute group. In the above example, it is assumed that the attributes included in the acquired attribute group match the fixture number defined in the part information.

[0217] Next, the association model generation unit 405 sorts the peripheral attributes in order of their relative relevance to the main attribute, concatenates them to the main attribute, and generates a column of attributes (strings). Here, the relevance of a peripheral attribute to the main attribute is considered to be stronger the closer it is to the coordinate position of the main attribute. In the example above, "10Ω" is closer to "A1" than "OP12". Therefore, in the example above, "10Ω" has a stronger relevance to "A1" than "OP12". Thus, the association model generation unit 405 concatenates the peripheral attributes "10Ω" and "OP12" in that order to the main attribute "A1", generating a column of attributes named [[A][1]][[1][0][Ω]][[O][P][1][2]].

[0218] Next, the association model generation unit 405 decomposes the generated attribute column into character units and converts each decomposed character into a numerical value based on the conversion table 135. In this way, the association model generation unit 405 converts the generated attribute column into numerical features.

[0219] The association model generation unit 405 then uses machine learning to learn so that the converted numerical features are classified into the classes of symbol S. These numerical features include information about the strength of the relationships between the matching attributes and between adjacent characters that make up each attribute. Therefore, when the association model generation unit 405 performs the learning, it can take into account the strength of the relationships between adjacent attributes and between adjacent characters that make up each attribute.

[0220] The association model generation unit 405 then repeats the same procedure as above, changing the main attribute from "A1" to another attribute. The association model generation unit 405 also repeats this procedure for all dotted-line frames included in the training data 1501. As a result, the association recognition unit 1024 learns the association between all attributes on the image diagram 1 and the symbols.

[0221] In Figure 15, an example was shown in which the association model generation unit 405 learns the association between attributes and symbols. However, the association model generation unit 405 can also learn the association between attributes and connecting lines in the same manner as described above.

[0222] The operation of the learning device 400 according to Embodiment 1 will be described. Figure 16 is a flowchart illustrating the operation of the learning device 400 according to Embodiment 1.

[0223] The user of the drawing recognition device 100 creates multiple image drawings 1 to which teacher labels have been assigned in advance. These multiple image drawings 1 to which teacher labels have been assigned are input by the user. For example, the learning device 400 is connected to the operation input device 200 (see Figure 1), and the user uses the operation input device 200 to input multiple image drawings 1 to which teacher labels have been assigned in advance.

[0224] Furthermore, the user of the drawing recognition device 100 associates one or more attributes with a single symbol and adds text data indicating the result as a training label to the image drawing 1. Similarly, the user of the drawing recognition device 100 associates one or more attributes with a single connecting line and adds text data indicating the result as a training label to the image drawing 1.

[0225] The training data acquisition unit 401 performs data acquisition processing for training (step ST21).

[0226] Specifically, the training data acquisition unit 401 acquires multiple image drawings 1 to which teacher labels are assigned. Then, the training data acquisition unit 401 generates training data based on the acquired image drawings 1. The training data acquisition unit 401 outputs the generated training data to the symbol model generation unit 402, the attribute information model generation unit 403, the connection line model generation unit 404, and the association model generation unit 405.

[0227] The symbol model generation unit 402 learns symbol recognition based on the training data output by the training data acquisition unit 401 in step ST21 and generates a first machine learning model 131 (step ST22).

[0228] The symbol model generation unit 402 stores the generated first machine learning model 131 in the inference unit 103.

[0229] The attribute information model generation unit 403 learns attribute determination based on the training data output by the training data acquisition unit 401 in step ST21 and generates a second machine learning model 132 (step ST23).

[0230] The attribute information model generation unit 403 stores the generated second machine learning model 132 in the inference unit 103.

[0231] The connection line model generation unit 404 learns how to determine connection lines based on the training data output by the training data acquisition unit 401 in step ST21, and generates a third machine learning model 133 (step ST24).

[0232] The connection line model generation unit 404 stores the generated third machine learning model 133 in the inference unit 103.

[0233] The association model generation unit 405 generates a conversion table 135 based on the training data output by the training data acquisition unit 401 in step ST21. The association model generation unit 405 also uses the conversion table 135 to learn the correspondence between attributes and symbols, and the correspondence between attributes and connecting lines, based on the above training data, and generates a fourth machine learning model 134 (step ST25).

[0234] The association model generation unit 405 stores the generated fourth machine learning model 134 in the inference unit 103.

[0235] Here, Figure 17 is a flowchart illustrating the details of the generation process of the fourth machine learning model 134 performed by the association model generation unit 405 in step ST25 of Figure 16.

[0236] The association model generation unit 405 generates a conversion table 135 that can convert all attributes (characters) included in the attribute information into numerical values, based on the training data output by the training data acquisition unit 401 in step ST21 (step ST2501).

[0237] Next, the association model generation unit 405 obtains the primary attribute and peripheral attributes as a set of attributes from the training data output by the training data acquisition unit 401 in step ST21. The association model generation unit 405 also rearranges the peripheral attributes in order of their relative relationship to the primary attribute, concatenates them to the primary attribute, and generates a column (string) of attributes (step ST2502).

[0238] Next, the association model generation unit 405 decomposes the attribute column generated in step ST2502 into character units and converts each decomposed character into a numerical value based on the conversion table 135. In this way, the association model generation unit 405 converts the generated attribute column into numerical features (step ST2503).

[0239] Next, the association model generation unit 405 uses machine learning to learn the associations of the numerical features transformed in step ST2503, i.e., the primary attributes, so that they are classified into either the symbol or connecting line class (step ST2504).

[0240] In this way, the learning device 400 can generate a first machine learning model 131, a second machine learning model 132, a third machine learning model 133, and a fourth machine learning model 134 based on multiple image drawings 1 through so-called supervised learning.

[0241] Furthermore, attributes associated with symbols or connecting lines tend to be located near the symbols or connecting lines on the drawing data. The learning device 400 utilizes this tendency to acquire a group of attributes consisting of the primary attribute to be associated with the symbol or connecting line, and peripheral attributes located within a specific range centered on this primary attribute. It then performs learning based on the sequence of attributes generated from the acquired group of attributes. As a result, the learning device 400 can learn associations including not only primary attributes but also peripheral attributes, and can generate a fourth machine learning model 134 that can accurately associate attributes even when the same attribute is associated with different symbols or different connecting lines.

[0242] Figures 18A and 18B show an example of the hardware configuration of the learning device 400 according to Embodiment 1.

[0243] In Embodiment 1, the functions of the learning data acquisition unit 401, the symbol model generation unit 402, the attribute information model generation unit 403, the connection line model generation unit 404, and the association model generation unit 405 are realized by the processing circuit 1801. That is, the learning device 400 includes a processing circuit 1801 for generating a first machine learning model 131, a second machine learning model 132, a third machine learning model 133, a fourth machine learning model 134, and a conversion table 135, based on the image diagram 1. The processing circuit 1801 may be dedicated hardware as shown in Figure 18A, or it may be a CPU (Central Processing Unit) 1804 that executes a program stored in memory 1805 as shown in Figure 18B.

[0244] If the processing circuit 1801 is dedicated hardware, it may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0245] When the processing circuit 1801 is the CPU 1804, the functions of the learning data acquisition unit 401, the symbol model generation unit 402, the attribute information model generation unit 403, the connection line model generation unit 404, and the association model generation unit 405 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in memory 1805. The processing circuit 1801 executes the functions of the learning data acquisition unit 401, the symbol model generation unit 402, the attribute information model generation unit 403, the connection line model generation unit 404, and the association model generation unit 405 by reading and executing the program stored in memory 1805. In other words, the learning device 400 includes memory 1805 for storing a program that, when executed by the processing circuit 1801, will result in the execution of steps ST21 to ST25 in Figure 16 and steps ST2501 to ST2504 in Figure 17. Furthermore, the program stored in memory 1805 can be said to cause the computer to execute the procedures or methods of the learning data acquisition unit 401, the symbol model generation unit 402, the attribute information model generation unit 403, the connection line model generation unit 404, and the association model generation unit 405. Here, memory 1805 refers to non-volatile or volatile semiconductor memory such as RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), or magnetic disks, flexible disks, optical disks, compact disks, minidiscs, DVDs (Digital Versatile Discs), etc.

[0246] Furthermore, the functions of the learning data acquisition unit 401, the symbol model generation unit 402, the attribute information model generation unit 403, the connection line model generation unit 404, and the association model generation unit 405 may be partially implemented by dedicated hardware and partially by software or firmware. For example, the learning data acquisition unit 401 can be implemented by a processing circuit 1801 as dedicated hardware, while the symbol model generation unit 402, the attribute information model generation unit 403, the connection line model generation unit 404, and the association model generation unit 405 can be implemented by the processing circuit 1801 reading and executing a program stored in memory 1805. Furthermore, the learning device 400 includes devices such as the drawing recognition device 100, and input interface devices 1802 and output interface devices 1803 that perform wired or wireless communication.

[0247] As described above, the drawing recognition device 100 of Embodiment 1 includes a drawing acquisition unit 101 that acquires drawing data (image drawing 1), a symbol recognition unit (first recognition unit) 1021 that recognizes information about symbols in the drawing data based on the drawing data acquired by the drawing acquisition unit 101 and a first machine learning model 131 that takes the drawing data as input and outputs recognition results about symbols on the drawing data, an attribute information recognition unit (second recognition unit) 1022 that recognizes information about attributes in the drawing data based on the drawing data acquired by the drawing acquisition unit 101 and a second machine learning model 132 that takes the drawing data as input and outputs recognition results about attributes on the drawing data, and the drawing recognition device 100 of Embodiment 1 that takes the drawing data as input and The drawing recognition device 100 includes a connection line recognition unit (third recognition unit) 1023 that recognizes information about connection lines in the drawing data based on a third machine learning model 133 that outputs recognition results regarding connection lines connecting symbols on the drawing data, and an association recognition unit (fourth recognition unit) 1024 that recognizes information about the correspondence between attributes, symbols, and connection lines in the drawing data based on a fourth machine learning model 134 that takes as input the information recognized by the symbol recognition unit 1021, the attribute information recognition unit 1022, and the attribute information recognized by the connection line recognition unit 1023, and outputs information about the correspondence between attributes and symbols in the drawing data, and information about the correspondence between attributes and connection lines in the drawing data. As a result, the drawing recognition device 100 can recognize the correspondence between symbols, attributes, and connection lines without requiring layout knowledge.

[0248] Furthermore, the fourth machine learning model 134 is a model trained to output symbols and connecting lines associated with a primary attribute in response to a column of attributes generated based on primary attributes, which are attributes that are the subject of inference regarding the correspondence between symbols and connecting lines in the drawing shown by the drawing data, and peripheral attributes, which are attributes that exist in a specific range centered on the primary attribute in the drawing shown by the drawing data. The association recognition unit 1024 identifies a primary attribute based on the attribute information recognized by the attribute information recognition unit 1022, and inputs a column of attributes generated based on the identified primary attribute and peripheral attributes, which are attributes that exist in a specific range centered on the primary attribute in the drawing shown by the drawing data, into the fourth machine learning model 134 to obtain information regarding symbols and connecting lines associated with the primary attribute. By repeatedly generating a column of attributes and inputting the generated column of attributes into the fourth machine learning model 134 while changing the primary attribute based on the attribute information recognized by the attribute information recognition unit 1022, the association recognition unit 1024 recognizes information regarding the correspondence between attributes, symbols and connecting lines in the drawing data. This allows the drawing recognition device 100 to improve the accuracy of its recognition of information regarding the correspondence between attributes, symbols, and connecting lines in the drawing data.

[0249] To elaborate on this point, for example, the following methods can be considered for determining the correspondence between characters and symbols without requiring knowledge of their arrangement.

[0250] In other words, the association between symbols and attributes, and the association between connecting lines and attributes are tentatively determined based on the distance between symbols and attributes on the drawing data, and the distance between connecting lines and attributes. Then, by referring to the part information recorded in advance in the part recording unit and the drawing characteristics and relevance information recorded in advance in the relevance recording unit, the final correspondence relationship is determined based on the degree of relevance between symbols and attributes, and the degree of relevance between connecting lines and attributes.

[0251] However, this method may result in incorrect associations if the same attribute is associated with different symbols or connecting lines, potentially reducing the accuracy of the associations.

[0252] In this regard, in the drawing recognition device 100 according to Embodiment 1, the association recognition unit 1024, as described above, inputs a column of attributes generated based not only on the main attribute to be associated, but also on peripheral attributes, which are attributes that exist in a specific range centered on the main attribute in the drawing shown in the drawing data, into the fourth machine learning model 134 when performing association, and recognizes the correspondence relationship. As a result, the drawing recognition device 100 according to Embodiment 1 can perform accurate association even when the same attribute is associated with different symbols or connecting lines, and can improve the recognition accuracy of information regarding the correspondence relationship between attributes, symbols and connecting lines in the drawing data.

[0253] Furthermore, the fourth machine learning model 134 is a model trained to output, in response to input of numerical features obtained by transforming a column of attributes, the column of attributes to be transformed into the numerical features, and the symbols and connecting lines associated with the main attributes included in the column of attributes to be transformed into numerical features that approximate the numerical features. The association recognition unit 1024 generates a column of attributes by rearranging the surrounding attributes in order of proximity to the main attribute in the drawing shown in the drawing data and then linking them to the main attribute, transforms the generated column of attributes into numerical features, and inputs the transformed numerical features into the fourth machine learning model 134 to obtain information regarding the symbols and connecting lines associated with the main attributes included in the column of attributes to be transformed into the numerical features, and the column of attributes to be transformed into numerical features that approximate the numerical features. As a result, the drawing recognition device 100 according to Embodiment 1 can absorb inconsistencies in attribute notation, and can suppress a decrease in the recognition accuracy of information regarding the correspondence between attributes, symbols, and connecting lines in the drawing data due to such inconsistencies in attribute notation.

[0254] Furthermore, when the association recognition unit 1024 inputs the generated attribute column to the fourth machine learning model 134, it refers to the correspondence number, which is set in advance for each category to which the symbol belongs and indicates the number of main attributes that can be associated with the symbol. If the number of main attributes included in the generated attribute column does not match the correspondence number, it matches the number of main attributes included in the generated attribute column with the correspondence number before inputting it to the fourth machine learning model 134. As a result, the drawing recognition device 100 according to Embodiment 1 can suppress a decrease in inference accuracy due to the use of an inappropriate attribute column in inference.

[0255] Furthermore, the drawing recognition device 100 includes a determination unit 104 that determines whether the information regarding the correspondence between attributes, symbols, and connecting lines in the drawing data, recognized by the association recognition unit 1024, is appropriate, based on predetermined determination information for determining the appropriateness of the information regarding the correspondence between attributes, symbols, and connecting lines in the drawing data. As a result, the drawing recognition device 100 according to Embodiment 1 can improve the recognition accuracy of the correspondence between attributes, symbols, and connecting lines.

[0256] Furthermore, the determination unit 104 records the information regarding the correspondence relationship that it determined to be negative as unconfirmed data, along with the information regarding the symbol, attribute, and connecting line that was the source of the information regarding the correspondence relationship. The association recognition unit 1024 then recognizes the information regarding the correspondence relationship between attributes, symbols, and connecting lines in the drawing data again, based on the information recorded as unconfirmed data by the determination unit 104 and the fourth machine learning model 134. As a result, the drawing recognition device 100 according to Embodiment 1 can improve the recognition accuracy of the correspondence relationship between attributes, symbols, and connecting lines.

[0257] It should be noted that this disclosure allows for modifications of any component of the embodiment, or the omission of any component of the embodiment. [Explanation of Symbols]

[0258] 1 Image drawing, 100 Drawing recognition device, 101 Drawing acquisition unit, 102 Recognition unit, 103 Inference unit storage unit, 104 Judgment unit, 105 Parts recording unit, 106 Confirmed data recording unit, 107 Unconfirmed data recording unit, 108 Drawing display unit, 131 First machine learning model, 132 Second machine learning model, 133 Third machine learning model, 134 Fourth machine learning model, 135 Conversion table, 200 Operation input device, 201 Keyboard, 202 Mouse, 300 Display device, 301 Display, 400 Learning device, 401 Learning data acquisition unit, 402 Symbol model generation unit, 403 Attribute information model generation unit, 404 Connection line model generation unit, 405 Association model generation unit, 611, 612, 613 Symbol, 621, 622, 623, 624 Attribute, 631 Connection line, 701 Symbol and attributes, 702 Attribute group, 703 Inference result, 901 Attribute column, 902 Correspondence, 903 Inference result, 1021 Symbol recognition unit, 1022 Attribute information recognition unit, 1023 Connection line recognition unit, 1024 Association recognition unit, 1301 Processing circuit, 1302 Input interface device, 1303 Output interface device, 1304 CPU, 1305 Memory, 1501 Training data, 1801 Processing circuit, 1802 Input interface device, 1803 Output interface device, 1804 CPU, 1805 Memory, S Symbol.

Claims

1. A drawing acquisition unit that acquires drawing data, A first recognition unit recognizes information about symbols in the drawing data based on the drawing data acquired by the drawing acquisition unit and a first machine learning model that takes the drawing data as input and outputs recognition results regarding symbols on the drawing data. A second recognition unit recognizes information regarding attributes in the drawing data based on the drawing data acquired by the drawing acquisition unit and a second machine learning model that takes the drawing data as input and outputs recognition results regarding attributes on the drawing data. A third recognition unit recognizes information about connecting lines in the drawing data based on the drawing data acquired by the drawing acquisition unit and a third machine learning model that takes the drawing data as input and outputs recognition results regarding connecting lines that connect symbols on the drawing data. A fourth recognition unit recognizes information regarding the correspondence between attributes, symbols, and connecting lines in the drawing data, based on information regarding symbols recognized by the first recognition unit using the first machine learning model, information regarding attributes recognized by the second recognition unit using the second machine learning model, and information regarding connecting lines recognized by the third recognition unit using the third machine learning model, and information regarding attributes recognized by the second recognition unit as input, and a fourth machine learning model that outputs information regarding the correspondence between attributes and symbols in the drawing data, and information regarding the correspondence between attributes and connecting lines in the drawing data, A drawing recognition device equipped with the following features.

2. The fourth machine learning model described above is: The model is trained to output symbols and connecting lines associated with a given primary attribute, which is the attribute whose correspondence to symbols and connecting lines in the drawing shown in the drawing data is inferred, and peripheral attributes, which are attributes that exist in a specific range centered on the primary attribute in the drawing shown in the drawing data, as input to a column of attributes. The fourth recognition unit is, Based on the attribute information recognized by the second recognition unit, the primary attribute is identified, and a column of attributes generated based on the identified primary attribute and peripheral attributes which are attributes that exist in a specific range centered on the primary attribute in the drawing shown in the drawing data is input to the fourth machine learning model, thereby obtaining information on the symbol and connecting lines associated with the primary attribute. Based on the attribute information recognized by the second recognition unit, the system repeatedly generates a column of attributes and inputs the generated column of attributes into the fourth machine learning model, while changing the primary attribute, thereby recognizing information regarding the correspondence between attributes, symbols, and connecting lines in the drawing data. The drawing recognition device according to feature 1.

3. The fourth machine learning model described above is: This model is trained to output, in response to input of a numerical feature obtained by transforming the aforementioned attribute column, the column of attributes to be transformed into the numerical feature, and the symbols and connecting lines associated with the principal attributes included in the column of attributes to be transformed into a numerical feature that approximates the said numerical feature. The fourth recognition unit is, By rearranging the surrounding attributes in order of proximity to the main attribute in the drawing shown in the drawing data and then linking them to the main attribute, a column of attributes is generated. The generated attribute column is converted into numerical features, and the resulting numerical features are input to the fourth machine learning model. This process obtains information about the symbols and connecting lines associated with the principal attributes included in the attribute column converted into numerical features, and the attribute column converted into numerical features that approximate the numerical features. The drawing recognition device according to feature 2.

4. The fourth recognition unit is, When inputting the generated column of attributes into the fourth machine learning model, the corresponding number, which is set in advance for each category to which the symbol belongs and indicates the number of primary attributes that can be associated with the symbol, is referenced. If the number of primary attributes in the generated attribute column does not match the corresponding number, the number of primary attributes in the generated attribute column and the corresponding number are made to match before being input into the fourth machine learning model. The drawing recognition device according to feature 2.

5. The system includes a determination unit that determines whether the information regarding the correspondence between attributes, symbols, and connecting lines in the drawing data, recognized by the fourth recognition unit, is appropriate, based on predetermined determination information for determining the appropriateness of the information regarding the correspondence between attributes, symbols, and connecting lines in the drawing data. A drawing recognition device according to any one of claims 1 to 4.

6. The determination unit records the information regarding the correspondence relationship that was determined to be negative by the determination, along with the information regarding the symbol, attribute, and connection line that formed the basis of the information regarding the correspondence relationship, as undetermined data. The fourth recognition unit, based on the information recorded as undetermined data by the determination unit and the fourth machine learning model, re-recognizes information regarding the correspondence between attributes, symbols, and connecting lines in the drawing data. The drawing recognition device according to claim 5, characterized in that it is a drawing recognition device.

7. Computers, A drawing acquisition unit that acquires drawing data, A first recognition unit recognizes information about symbols in the drawing data based on the drawing data acquired by the drawing acquisition unit and a first machine learning model that takes the drawing data as input and outputs recognition results regarding symbols on the drawing data. A second recognition unit recognizes information regarding attributes in the drawing data based on the drawing data acquired by the drawing acquisition unit and a second machine learning model that takes the drawing data as input and outputs recognition results regarding attributes on the drawing data. A third recognition unit recognizes information about connecting lines in the drawing data based on the drawing data acquired by the drawing acquisition unit and a third machine learning model that takes the drawing data as input and outputs recognition results regarding connecting lines that connect symbols on the drawing data. A fourth recognition unit recognizes information regarding the correspondence between attributes, symbols, and connecting lines in the drawing data, based on information regarding symbols recognized by the first recognition unit using the first machine learning model, information regarding attributes recognized by the second recognition unit using the second machine learning model, and information regarding connecting lines recognized by the third recognition unit using the third machine learning model, and information regarding attributes recognized by the second recognition unit as input, and a fourth machine learning model that outputs information regarding the correspondence between attributes and symbols in the drawing data, and information regarding the correspondence between attributes and connecting lines in the drawing data. A drawing recognition program designed to function as such.

Images