Drawing recognition device and drawing recognition program

Abstract

To provide a drawing recognition device capable of determining a correspondence relationship between a character and a symbol without requiring layout knowledge.SOLUTION: The drawing recognition device comprises: a drawing acquisition section 101 for acquiring drawing data; a recognition section 102 for recognizing information on a symbol in the drawing data acquired by the drawing acquisition section 101, information on an attribute in the drawing data acquired by the drawing acquisition section 101, and information on a connection line in the drawing data acquired by the drawing acquisition section 101 on the basis of the drawing data and a machine learning model; and a determination section 104 for determining a correspondence relationship among the symbol in the drawing data acquired by the drawing acquisition section 101, the attribute, the other symbol on the basis of the information on the symbol, the information on the attribute, and the information on the connection line which are recognized by the recognition section 102.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a drawing recognition device and a drawing recognition program. [Background technology]

[0002] 2. Description of the Related Art Drawing recognition techniques are known that recognize characters or symbols present on a drawing containing a mixture of characters and symbols and determine their correspondence. For example, Patent Document 1 discloses a drawing recognition technology that, in drawing data containing a mixture of characters and symbols, determines the correspondence between individually recognized characters and which symbols or piping they correspond to, based on previously prepared placement knowledge of the relative positions between characters and symbols. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2004-234424 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the conventional technology disclosed in Patent Document 1, there was a problem that in order to perform drawing recognition, in order to determine the correspondence between characters, symbols, etc., placement knowledge must be prepared in advance.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a drawing recognition device that is capable of determining the correspondence between characters and symbols without requiring placement knowledge. [Means for solving the problem]

[0006] The drawing recognition device according to the present disclosure includes a drawing acquisition unit that acquires drawing data, a recognition unit that recognizes, based on the drawing data acquired by the drawing acquisition unit and a machine learning model that receives the drawing data as an input and outputs recognition results related to symbols, attributes, and connection lines connecting the symbols in the drawing data, information about symbols including position information of the symbols in the drawing data acquired by the drawing acquisition unit, information about attributes including position information of the attributes in the drawing data acquired by the drawing acquisition unit, and information about connection lines including position information of the connection lines in the drawing data acquired by the drawing acquisition unit, and a determination unit that determines a correspondence relationship between the symbols, attributes, and other symbols in the drawing data acquired by the drawing acquisition unit, based on the information about the symbols, information about the attributes, and information about the connection lines recognized by the recognition unit. The determination unit determines the correspondence between the symbol, attribute, and other symbols in the drawing data acquired by the drawing acquisition unit based on the information on the symbol, attribute, and connection line recognized by the recognition unit, and on the degree of association between the symbol and the attribute shown on the drawing data, which is represented by the number of times that the symbol and the attribute related to the symbol are shown together on the drawing data, or on the degree of association between the symbols, which is represented by the number of times that the symbol and the connection destination symbol to which the symbol is connected are shown connected on the drawing data. It is something. Effect of the Invention

[0007] According to the present disclosure, it is possible to determine the correspondence between characters and symbols without requiring placement knowledge. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of a configuration of a drawing recognition device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram for explaining an example of an image drawing in the first embodiment. [Diagram 3] FIG. 2 is a diagram for explaining an image of an example of drawing characteristic information in the first embodiment. [Figure 4] 2 is a diagram illustrating a detailed configuration example of a recognition unit included in the drawing recognition device according to the first embodiment. FIG. [Diagram 5]FIG. 5A shows an example image of the symbol recognition result acquired by the symbol recognition unit, the attribute recognition result acquired by the attribute information recognition unit, and the connection line recognition result acquired by the connection line recognition unit in embodiment 1, where FIG. 5A shows an image in which the symbol recognition result, attribute recognition result, and connection line recognition result are mapped onto an image drawing acquired by the drawing acquisition unit, and FIG. 5B shows an image in which the contents of the symbol recognition result, attribute recognition result, and connection line recognition result are displayed in text. [Figure 6] FIG. 2 is a diagram for explaining an example of an image of the contents of part information stored in a part recording unit in the first embodiment. [Figure 7] FIG. 2 is a diagram for explaining an example of an image of the contents of relevance information stored in a relevance recording section in the first embodiment. [Figure 8] 10 is an explanatory diagram showing an image of a specific example of processing in which a determination unit in the first embodiment comprehensively determines whether or not symbols, attributes, and connections between symbols in inference data have been correctly recognized. FIG. [Figure 9] 4 is a diagram for explaining an image of an example of a determination result output by a determination unit in the first embodiment. FIG. [Figure 10] FIG. 11 is an explanatory diagram showing an image of a specific example of a process in which a comparison unit compares a post-determination image drawing output by a generation unit with an image drawing acquired by a drawing acquisition unit in the first embodiment. [Figure 11] 4 is a flowchart for explaining the operation of the drawing recognition device according to the first embodiment. [Figure 12] 12 is a flowchart for explaining details of the inference operation performed by the recognition unit in step ST1102 of FIG. [Figure 13] 13A and 13B are diagrams illustrating an example of a hardware configuration of the drawing recognizing device according to the first embodiment. [Figure 14] FIG. 2 is a diagram illustrating an example of the configuration of a learning device according to the first embodiment. [Figure 15] 4 is a flowchart for explaining the operation of the learning device according to the first embodiment. [Figure 16] 16A and 16B are diagrams illustrating an example of a hardware configuration of learning device 400 according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1

[0010] FIG. 1 is a diagram showing an example of a configuration of a drawing recognition device 100 according to the first embodiment. The drawing recognition device 100 is connected to an operation input device 200 and a display device 300 . The operation input device 200 includes, for example, a keyboard 201 and a mouse 202 . The display device 300 includes a display 301. The display 301 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display. The display device 300 may be mounted on the operation input device 200.

[0011] The operation input device 200 accepts operations for the drawing recognition device 100 by a user of the drawing recognition device 100 . For example, the user operates the keyboard 201 or the mouse 202 provided in the operation input device 200 to input information to be input to the drawing recognition device 100. Specifically, the user operates the keyboard 201 or the mouse 202 to display, for example, a list of image drawings 1 or a list of drawing characteristic information 2 on the display 301 of the display device 300, and selects the image drawing 1 and the drawing characteristic information 2.

[0012] FIG. 2 is a diagram for explaining an example of image drawing 1 in the first embodiment. The image drawing 1 is, for example, drawing data obtained by digitizing a paper drawing on which a circuit diagram as shown in Fig. 2 is printed, using a scanner or the like. The image drawing 1 may also be, for example, electronic data created on a computer from a paper drawing on which a circuit diagram as shown in Fig. 2 is printed. The format of the image drawing 1 is, for example, a general image format such as jpg or tiff format. In the first embodiment, as an example, the image drawing 1 is assumed to be drawing data obtained by digitizing a circuit diagram.

[0013] FIG. 3 is a diagram for explaining an example of drawing characteristic information 2 according to the first embodiment. The drawing characteristic information 2 is information about the characteristics of the image drawing 1, which is set for each image drawing 1. Specifically, the drawing characteristic information 2 is information that associates information capable of identifying the image drawing 1, such as a name or ID, with the purpose of the image drawing 1, and a model in which the image drawing 1 is used. In the first embodiment, the drawing characteristic information 2 is information that associates information capable of identifying a circuit diagram, such as a name or ID, with the purpose of the circuit diagram, and a model in which the circuit diagram is used. In this embodiment, the drawing characteristic information 2 is text data, but this is merely an example. The drawing characteristic information 2 may be any information that associates the drawing with the characteristics of the drawing.

[0014] For example, a plurality of image drawings 1 are stored in advance in a drawing recording unit (not shown) provided at a location that can be referenced by the drawing recognition device 100 and the operation input device 200. Also, drawing characteristic information 2 is stored in advance in a relevance recording unit 106 (details of which will be described later). The user, for example, operates the keyboard 201 or the mouse 202 to cause the drawing recognition device 100 to display on the display 301 a list of a plurality of image drawings 1 stored in the drawing recording unit and the drawing characteristic information 2 stored in the relevance recording unit 106. Then, the user, for example, operates the keyboard 201 or the mouse 202 to select a desired image drawing 1 from the displayed list of image drawings 1, and selects the drawing characteristic information 2 corresponding to the selected image drawing 1. The operation input device 200 receives the image drawing 1 selected by the user and the drawing characteristic information 2. The image drawing 1 and the drawing characteristic information 2 received by the operation input device 200 are input to the drawing recognition device 100.

[0015] The drawing recognition device 100 performs drawing recognition of the image drawing 1 based on the image drawing 1 and drawing characteristic information 2 inputted from the operation input device 200 . In addition, in embodiment 1, drawing recognition performed by the drawing recognition device 100 refers to recognizing information on one or more symbols (hereinafter referred to as "symbol information") on the image drawing 1, information on attributes associated with the symbols (hereinafter referred to as "attribute information"), information on connecting lines between the symbols (hereinafter referred to as "connecting line information"), and the correspondence between symbols, attributes, and connecting lines. The symbol information includes, for example, the number of symbols, the position information of each symbol, the type of each symbol, and information on the accuracy of each symbol. The attribute information includes, for example, the number of attributes, the position information of each attribute, the type of each attribute, and information on the accuracy of each attribute. The connection line information includes, for example, the number of connection lines, the position information of each connection line, the type of each connection line, and information on the accuracy of each connection line. Note that in the first embodiment, "accuracy" refers to the probability of accuracy. For example, the accuracy of a symbol refers to the probability that it is a symbol, the accuracy of an attribute refers to the probability that it is an attribute, and the accuracy of a connection line refers to the probability that it is a connection line.

[0016] In the first embodiment, a symbol in the image drawing 1 is, for example, a part shown on the image drawing 1. Also, an attribute in the image drawing 1 is, for example, a character associated with a symbol and for identifying the symbol. Specifically, in the first embodiment, an attribute is a character indicating the tool number of a part or the name of a part. Also, a connection line in the image drawing 1 is, for example, a line connecting symbols. Note that in the first embodiment, the connection line also includes an intersection.

[0017] The drawing recognition device 100 includes a drawing acquisition unit 101, a recognition unit 102, an inference unit storage unit 103, a judgment unit 104, a part recording unit 105, an association degree recording unit 106, a generation unit 107, a comparison unit 108, a matching data recording unit 109, an unmatched data recording unit 110, and a drawing display unit 111. The inference unit storage unit 103 stores a first machine learning model 131, a second machine learning model 132, and a third machine learning model 133. The first machine learning model 131, the second machine learning model 132, and the third machine learning model 133 are models that have been trained to input the image drawing 1 and the drawing characteristic information 2, and output information relating to the result of drawing recognition of the image drawing 1. In the following first embodiment, the first machine learning model 131, the second machine learning model 132, and the third machine learning model 133 are also collectively referred to simply as the "machine learning models."

[0018] More specifically, in the first embodiment, the first machine learning model 131 is a machine learning model that receives an image drawing 1 and drawing characteristic information 2 as input, and outputs a result of recognizing symbol information on the image drawing 1 (hereinafter referred to as a "symbol recognition result"). That is, in the first embodiment, the first machine learning model 131 is a machine learning model for recognizing symbol information. The second machine learning model 132 is a machine learning model that receives the image drawing 1 and drawing characteristic information 2 as input, and outputs a result of recognizing attribute information associated with a symbol on the image drawing 1 (hereinafter referred to as an "attribute recognition result"). That is, in the first embodiment, the second machine learning model 132 is a machine learning model for attribute information recognition. The third machine learning model 133 is a machine learning model that outputs a result of recognizing connection line information between symbols on the image drawing 1 (hereinafter referred to as a "connection line recognition result"). That is, in the first embodiment, the third machine learning model 133 is a machine learning model for recognizing connection line information.

[0019] The machine learning model is generated by a learning device 400 (see FIG. 14 described later) performing learning in machine learning. Details of the learning device 400 will be described later.

[0020] The drawing acquisition unit 101 acquires the image drawing 1 and drawing characteristic information 2 inputted from the user via the operation input device 200 . The drawing acquisition unit 101 performs a predetermined process on the image drawing 1 to generate inference data. As described above, the drawing recognition device 100 inputs the image drawing 1 into a machine learning model to obtain a drawing recognition result for the image drawing 1. Here, there is a limit to the image size that is input to the machine learning model. Specifically, for example, if the image drawing 1 is FHD (Full High Definition) with an image size of 1920×1080 pixels, this exceeds the upper limit of the image size that can be input to the machine learning model. Therefore, the drawing acquisition unit 101 divides the acquired image drawing 1 into data of, for example, 224 x 224 pixels each, and the divided data is used as inference data. The intervals at which the image drawing 1 is divided may be equal intervals of 224 x 224 pixels in the top, bottom, left, and right directions, and overlapping parts may be included when dividing the image drawing 1. In addition, for example, if the acquired image drawing 1 is within the size limit that allows it to be used as input to a machine learning model, the drawing acquisition unit 101 does not need to perform the processing described above, and can simply use the acquired image drawing 1 as inference data. The drawing acquisition unit 101 outputs the inference data generated based on the image drawing 1 and the acquired drawing characteristic information 2 to the recognition unit 102.

[0021] In this embodiment, the division of the image drawing 1 as described above is performed by the drawing acquisition unit 101, but this is merely an example. For example, the division of the image drawing 1 may be performed by the recognition unit 102 described later. In this case, the drawing acquisition unit 101 outputs the acquired image drawing 1 as is to the recognition unit 102 described later, and when the recognition unit 102 described later performs inference using a machine learning model, it divides the image drawing 1 as necessary.

[0022] The recognition unit 102 acquires a symbol recognition result, an attribute recognition result, or a connection line recognition result by performing inference using the first machine learning model 131, the second machine learning model 132, or the third machine learning model 133 stored in the inference device storage unit 103 based on the inference data and the drawing characteristic information 2 output from the drawing acquisition unit 101. In this way, the recognition unit 102 recognizes symbol information, attribute information, or connection line information. The recognition unit 102 outputs the inference result, in other words, the symbol recognition result, the attribute recognition result, or the connection line recognition result, to the determination unit 104 .

[0023] FIG. 4 is a diagram showing a detailed configuration example of the recognizing unit 102 included in the drawing recognizing apparatus 100 according to the first embodiment. For ease of explanation, FIG. 4 shows only the drawing acquisition unit 101, the recognition unit 102, and the inference unit storage unit 103 of the configuration example of the drawing recognition device 100 shown in FIG. As shown in FIG. 4, the recognition unit 102 includes a symbol recognition unit 1021 , an attribute information recognition unit 1022 , and a connection line recognition unit 1023 .

[0024] The symbol recognition unit 1021 acquires a symbol recognition result based on the inference data and drawing characteristic information 2 output from the drawing acquisition unit 101 and the first machine learning model 131. Specifically, the symbol recognition unit 1021 analyzes the inference data and extracts image features from the inference data. Also, the symbol recognition unit 1021 analyzes the drawing characteristic information 2 and extracts features from the drawing characteristic information 2. The symbol recognition unit 1021 extracts the features from the drawing characteristic information 2 as numerical values, for example, "0" for "home appliance", "1" for "distribution board", and "2" for "car". The symbol recognition unit 1021 inputs the image feature amount extracted from the inference data and the feature amount extracted from the drawing characteristic information 2 to the first machine learning model 131 to obtain a symbol recognition result. In embodiment 1, the symbol recognition unit 1021 obtains, based on the inference data and drawing characteristic information 2, and the first machine learning model 131, the number of symbols in the inference data, position information of each symbol, the type of each symbol, and the accuracy of each symbol, as symbol recognition results. The symbol recognition unit 1021 outputs the obtained symbol recognition result to the determination unit 104 .

[0025] The attribute information recognition unit 1022 acquires an attribute recognition result based on the inference data and drawing characteristic information 2 output from the drawing acquisition unit 101 and the second machine learning model 132. Specifically, the attribute information recognition unit 1022 analyzes the inference data and extracts image feature amounts from the inference data. Also, the attribute information recognition unit 1022 analyzes the drawing characteristic information 2 and extracts feature amounts from the drawing characteristic information 2. The attribute information recognition unit 1022 inputs the image feature amount extracted from the inference data and the feature amount extracted from the drawing characteristic information 2 to the second machine learning model 132 to obtain an attribute recognition result. In embodiment 1, the attribute information recognition unit 1022 obtains, based on the inference data and drawing characteristic information 2, and the second machine learning model 132, the number of attributes in the inference data, the position information of each attribute, the type of each attribute, and the accuracy of each attribute, as attribute recognition results. The attribute information recognition unit 1022 outputs the acquired attribute recognition result to the determination unit 104 .

[0026] The connection line recognition unit 1023 acquires a connection line recognition result based on the inference data and drawing characteristic information 2 output from the drawing acquisition unit 101 and the third machine learning model 133. Specifically, the connection line recognition unit 1023 analyzes the inference data and extracts image features from the inference data. Also, the connection line recognition unit 1023 analyzes the drawing characteristic information 2 and extracts features from the drawing characteristic information 2. The connection line recognition unit 1023 inputs the image feature amount extracted from the inference data and the feature amount extracted from the drawing characteristic information 2 to the third machine learning model 133 to obtain a connection line recognition result. In embodiment 1, based on the inference data and drawing characteristic information 2, and the third machine learning model 133, the connection line recognition unit 1023 acquires, as connection line recognition results, the number of connection lines in the inference data, position information of each connection line, the type of each connection line, and the accuracy of each connection line, etc. The connection line recognition unit 1023 outputs the acquired connection line recognition result to the determination unit 104 .

[0027] The inference performed by the recognition unit 102 based on the inference data, the drawing characteristic information 2, and the machine learning model will be specifically described with reference to FIG. Fig. 5 is a diagram showing an example of the symbol recognition result acquired by the symbol recognition unit 1021, the attribute recognition result acquired by the attribute information recognition unit 1022, and the connection line recognition result acquired by the connection line recognition unit 1023 in the first embodiment. Fig. 5A shows an image in which the symbol recognition result, the attribute recognition result, and the connection line recognition result are mapped onto the image drawing 1 acquired by the drawing acquisition unit 101, and Fig. 5B shows an image in which the contents of the symbol recognition result, the attribute recognition result, and the connection line recognition result are displayed in text.

[0028] 5A and 5B are linked. Specifically, for example, symbols indicated by 511, 512, and 513 in FIG. 5A correspond to symbol No. 1 "relay (a)", symbol No. 2 "relay (b)", and symbol No. 3 "fuse" in FIG. 5B, respectively. Also, for example, attributes indicated by 521, 522, 523, and 524 in FIG. 5A correspond to attribute No. 1 "30X004", attribute No. 2 "30X003", attribute No. 3 "FL3", and attribute No. 5 "62LCTX" in FIG. 5B, respectively. Also, for example, connection lines indicated by 531, 532, 533, and 534 in FIG. 5A correspond to connection lines No. 2, 5, 6, and 7 in FIG. 5B.

[0029] The symbols, attributes, and connection lines on the image drawing 1, in other words, the inference data, are recognized in rectangular areas on the image drawing 1. FIG. 5A shows an image in which a rectangular area indicating a recognized symbol, a rectangular area indicating a recognized attribute, and a rectangular area indicating a recognized connecting line are mapped onto image drawing 1 acquired by drawing acquisition unit 101. In Fig. 5B, for example, the symbol recognition result includes the symbol name and the position information of the recognized symbol. The position of the symbol is represented by the minimum and maximum values ​​of the X coordinate and the minimum and maximum values ​​of the Y coordinate of the rectangular area showing the symbol on the image drawing 1. Note that each coordinate shown in Fig. 5B is the coordinate with the upper left corner of the drawing as the origin in Fig. 5A. 5B, for example, the attribute recognition result includes the attribute, in other words, the character, and the position information of the recognized character. The position of the character is represented by the minimum and maximum values ​​of the X coordinate and the minimum and maximum values ​​of the Y coordinate of the rectangular area showing the character on the image drawing 1. 5B, for example, the connection line recognition result includes the type of the connection line and the position information of the recognized connection line. The position of the connection line is represented by the minimum and maximum values ​​of the X coordinate and the minimum and maximum values ​​of the Y coordinate of the rectangular area showing the connection line on the image drawing 1.

[0030] Note that the contents of the symbol recognition result, attribute recognition result, and connection line recognition result shown in Figures 5A and 5B are merely examples. The symbol recognition result, attribute recognition result, and connection line recognition result may include information other than the information shown in Figures 5A and 5B, such as an ID.

[0031] Returning to the explanation of Figure 1. The inference unit storage unit 103 stores a first machine learning model 131, a second machine learning model 132, and a third machine learning model 133.

[0032] The judgment unit 104 judges the correspondence between the symbols, attributes, and other symbols in the inference data based on the symbol recognition results, attribute recognition results, and connection line recognition results output by the recognition unit 102, the part information stored in the part recording unit 105, and the relevance information stored in the relevance recording unit 106, and makes a comprehensive judgment as to whether the symbols, attributes, and connections between symbols have been correctly recognized. Furthermore, the determination unit 104 determines the correspondence between the symbols, attributes, and other symbols in the inference data based on the mismatch data stored in the mismatch data recording unit 110, the part information stored in the part recording unit 105, and the relevance information stored in the relevance recording unit 106, and re-determines whether the symbols, attributes, and connections between the symbols have been correctly recognized. Details of the mismatch data stored in the mismatch data recording unit 110 will be described later.

[0033] The part information stored in the part recording unit 105 and the association degree information stored in the association degree recording unit 106 will be described.

[0034] FIG. 6 is a diagram for explaining an example of an image of the contents of part information stored in part recording unit 105 in the first embodiment. The part information is information relating to the part shown in the image drawing 1. Specifically, the part information is, for example, part list data including the tool number and part code of the part, as shown in FIG. In the first embodiment, the parts are shown as symbols on the image drawing 1.

[0035] FIG. 7 is a diagram for explaining an example of an image of the contents of the relevance information stored in relevance recording section 106 in the first embodiment. The relevance information is generated in advance by, for example, a user and recorded in the relevance recording unit 106. The user generates the relevance information in advance, for example, based on the image drawing 1 and the drawing characteristic information 2. In the first embodiment, as shown in Fig. 7, the relevance information includes symbol-attribute information (indicated by 701 in Fig. 7), symbol-connection information (indicated by 702 in Fig. 7), usage relevance information (indicated by 703 and 704 in Fig. 7), and model relevance information (indicated by 705 and 706 in Fig. 7).

[0036] The symbol-attribute information is information that defines, for each symbol, the degree of association between the symbol and an attribute related to the symbol. In the symbol-attribute information, for each symbol, an associated attribute is defined in association with the number of times that the attribute is shown together with the symbol on the image drawing 1. For example, a user generates the symbol-attribute information in advance based on multiple image drawings 1 to be used in the learning phase. The number of times defined in the symbol-attribute information indicates the number of times that the attribute associated with the symbol appears on the image drawing 1 used to generate the symbol-attribute information, in other words, on the image drawing 1 to be used in the learning phase. In the symbol-attribute information, an ID is assigned to each of the information in which a symbol, an attribute, and the frequency of occurrence of the attribute associated with the symbol are associated with each other.

[0037] The symbol-connection information is information that defines, for each symbol, the degree of association between the symbol and other symbols connected to the symbol. ToIn the symbol-connection information, a symbol to which the connection is to be made (hereinafter referred to as a "connection destination symbol") and the number of times that the connection destination symbol is shown connected to the symbol on the image drawing 1 are defined in association with each other for each symbol. For example, a user generates symbol-connection information in advance based on multiple image drawings 1 to be used in the learning phase. The number of times defined in the symbol-connection information indicates the number of times that the connection destination symbol appears on the image drawing 1 used to generate the symbol-connection information, in other words, on the image drawing 1 to be used in the learning phase. In the symbol-connection information, an ID is assigned to each of the pieces of information in which a symbol, a connected symbol, and the appearance frequency of the connected symbol are associated with each other.

[0038] The usage association information includes symbol-attribute usage association information (indicated by 703 in FIG. 7) and symbol-connection usage association information (indicated by 704 in FIG. 7). The symbol-attribute usage association information is information that defines the association between a symbol and an attribute for each purpose for which the image drawing 1 is used. More specifically, the symbol-attribute usage association information is information that classifies, for each purpose, the frequency of appearance of a combination of a symbol and an attribute defined in the symbol-attribute information in the image drawing 1. In the symbol-attribute usage association information, for each purpose, information indicating an associated symbol and attribute is associated with the number of times the combination of the associated symbol and attribute appears. The information indicating the associated symbol and attribute is, for example, an ID in the symbol-attribute information. That is, the symbol-attribute use association information shown in 703 in Fig. 7 indicates that the combination of "relay (a)" and "30X004" appears 22 times in the image drawing 1 used for the purpose of "home appliances." For example, the user generates symbol-attribute use relevance information in advance based on multiple image drawings 1 to be used in the learning phase and the generated symbol-attribute information. The user can specify the use of the image drawing 1 based on the drawing characteristic information 2 as shown in FIG.

[0039] The symbol-connection usage association information is information that defines the association between a symbol and a destination symbol for each purpose for which the image drawing 1 is used. More specifically, the symbol-connection usage association information is information that classifies, for each purpose, the frequency of occurrence of a combination of a symbol and a destination symbol defined in the symbol-connection information in the image drawing 1. In the symbol-connection usage association information, for each purpose, information indicating an associated symbol and a destination symbol is associated with the number of times the combination of the associated symbol and the destination symbol occurs. The information indicating the associated symbol and the connected symbol is, for example, an ID in the symbol-connection information. That is, the symbol-connection use association information shown in 704 in Fig. 7 indicates that the combination of "relay (a)" and "resistor" appears five times in image drawing 1 used for the purpose of "home appliances." For example, the user generates symbol-connection usage relevance information in advance based on a plurality of image drawings 1 used in the learning phase and the generated symbol-connection information.

[0040] The model relevance information includes symbol-attribute model relevance information (indicated by 705 in FIG. 7) and symbol-connection model relevance information (indicated by 706 in FIG. 7). The symbol-attribute model relevance information is information that defines the relevance between symbols and attributes for each model of device in which the image drawing 1 is used. More specifically, the model relevance information is information that classifies, for each model, the frequency of appearance of combinations of symbols and attributes defined in the symbol-attribute information in the image drawing 1. In the symbol-attribute model relevance information, for each model, information indicating associated symbols and attributes is associated with the number of times the combination of the associated symbol and attribute appears. The information indicating the associated symbol and attribute is, for example, an ID in the symbol-attribute information. That is, the symbol-attribute model association information shown in 705 in Fig. 7 indicates that the combination of "relay (a)" and "30X004" appears 18 times in the image drawing 1 used for the model "N-1001A". For example, the user generates symbol-attribute model relevance information in advance based on multiple image drawings 1 used in the learning phase and the generated symbol-attribute information. The user can identify the model of the image drawing 1 based on the drawing characteristic information 2 as shown in FIG.

[0041] The symbol-connection model relevance information is information that defines the relevance between a symbol and a destination symbol for each model of device in which the image drawing 1 is used. More specifically, the symbol-connection model relevance information is information that classifies, for each model, the frequency of appearance of a combination of a symbol and a destination symbol defined in the symbol-connection information in the image drawing 1. In the symbol-connection model relevance information, for each application, information indicating an associated symbol and a destination symbol is associated with the number of times the combination of the associated symbol and the destination symbol appears. The information indicating the associated symbol and the connected symbol is, for example, an ID in the symbol-connection information. That is, the symbol-connection model association information shown in 706 in Fig. 7 indicates that the combination of "relay (a)" and "resistor" appears 15 times in the image drawing 1 used for the model "N-1001A". For example, the user generates symbol-connection model relevance information in advance based on a plurality of image drawings 1 used in the learning phase and the generated symbol-connection information.

[0042] In the relevance information, the greater the number of times that something is associated, the higher the relevance. In the first embodiment, the relevance information includes symbol-attribute information, symbol-connection information, use relevance information, and model relevance information as shown in Fig. 7, but it is not essential that the relevance information includes all of the symbol-attribute information, symbol-connection information, use relevance information, and model relevance information. The relevance information may include at least one of the symbol-attribute information, symbol-connection information, use relevance information, and model relevance information. Furthermore, the relevance information is not limited to symbol-attribute information, symbol-connection information, use relevance information, or model relevance information, but may be information about the relevance of symbols, attributes, or connection lines shown on the image drawing 1. For example, the relevance information may include country / region relevance information that classifies the frequency of appearance of combinations of symbols and attributes, or combinations of symbols and connection destination symbols in the image drawing 1, by country or region where the image drawing 1 is mainly used.

[0043] Returning to the explanation of the determination unit 104, the determination by the determination unit 104 as to whether or not the symbols, attributes, and connections between symbols in the inference data have been correctly recognized will be specifically explained with reference to FIG. FIG. 8 is an explanatory diagram showing an image of a specific example of processing in which the determining unit 104 in the first embodiment comprehensively determines whether or not the symbols, attributes, and connections between symbols in the inference data have been correctly recognized. Here, for example, it is assumed that the symbol recognition result, the attribute recognition result, and the connection line recognition result output from the recognition unit 102 have contents as shown in Fig. 5, and the contents of the recognition result shown in Fig. 8 are the same as the contents of the recognition result shown in Fig. 5. Also, for example, it is assumed that the parts information stored in the parts recording unit 105 has contents as shown in Fig. 6, and the contents of the parts information shown in Fig. 8 are the same as the contents of the parts information shown in Fig. 6. In the following, as an example, the judgment process by the judgment unit 104 will be explained using an example in which the judgment unit 104 judges whether the connection relationship between the symbols "relay (a)" and "relay (b)" has been correctly recognized based on the symbol recognition result, attribute recognition result, and connection line recognition result output from the recognition unit 102.

[0044] The determination unit 104 first refers to the part recording unit 105, compares the part information with the attribute recognition result, and determines whether or not an attribute included in the attribute recognition result matches the tool number defined in the part information. If an attribute included in the attribute recognition result does not match the tool number defined in the part information, the determination unit 104 discards the attribute from the attribute recognition result. For the sake of convenience, it is assumed here that "30X004," "30X003," "FL3," "FL4," and "6LCTX57" included in the attribute recognition results all match the tool numbers defined in the part information.

[0045] Next, the determination unit 104 refers to the symbol recognition result, the attribute recognition result, or the connection line recognition result, and infers the association between the symbol, the attribute, and the connection line based on the coordinates indicating the position of the symbol, the coordinates indicating the position of the attribute, and the coordinates indicating the position of the connection line from each recognition result, and tentatively determines the correspondence between the symbol, the attribute, and the connection line. Specifically, for example, the determining unit 104 calculates the distance between the symbol, the attribute, and the connection line, and infers the association based on the calculated distance. For example, the determination unit 104 calculates the distance between "relay (a)" and "30X004" based on the coordinates indicating the position of "relay (a)" No. 1 in the symbol recognition result and the coordinates indicating the position of "30X004" No. 1 in the attribute recognition result. Since the calculated distance is close, the determination unit 104 infers that "relay (a)" and "30X004" are associated with each other. Note that in the first embodiment, the determination unit 104 obtains the distance between the symbol and the attribute, or the distance between the symbol and the connection line, by, for example, calculating the center position of a rectangle from the position information in the recognition result and using a difference vector of the center position. For example, the determination unit 104 may determine whether the calculated distance is close or not by comparing the calculated distance with a preset threshold (hereinafter referred to as "threshold for distance determination"). For example, the determination unit 104 determines that the distance is close if the calculated distance is equal to or less than the threshold for distance determination. The determining unit 104 may determine, for example, from the position information, whether the rectangles overlap, and determine that they are likely to overlap.

[0046] Furthermore, for example, the determination unit 104 calculates the distance between "relay (a)" and the No. 2 connection line based on the coordinates indicating the position of "relay (a)" No. 1 in the symbol recognition result and the coordinates indicating the position of the No. 2 connection line in the connection line recognition result. Since the calculated distance is close, the determination unit 104 infers that "relay (a)" and the No. 2 connection line are associated with each other. Furthermore, for example, the determination unit 104 determines the distance between the No. 2 connection line and the No. 5 connection line, and the distance between the No. 5 connection line and the No. 6 connection line, based on the coordinates indicating the position of the No. 2 connection line, the coordinates indicating the position of the No. 5 connection line, and the coordinates indicating the position of the No. 6 connection line in the connection line recognition result. Since the calculated distances are close to each other, the determination unit 104 infers that the No. 2 connection line, the No. 5 connection line, and the No. 6 connection line are associated with each other.

[0047] Furthermore, for example, the determination unit 104 calculates the distance between "relay (b)" and the No. 7 connection line based on the coordinates indicating the position of "relay (b)" No. 2 in the symbol recognition result and the coordinates indicating the position of the No. 7 connection line in the connection line recognition result. Since the calculated distance is close, the determination unit 104 infers that "relay (b)" and the No. 7 connection line are associated with each other. Also, for example, the determination unit 104 calculates the distance between "relay (b)" and "30X003" based on the coordinates indicating the position of "relay (b)" No. 2 in the symbol recognition result and the coordinates indicating the position of "30X003" No. 2 in the attribute recognition result. Since the calculated distance is close, the determination unit 104 infers that "relay (b)" and "30X003" are associated with each other.

[0048] From the results of the above estimation, the judgment unit 104 provisionally judges that "relay (a)" is "30X004", "relay (b)" is "30X003", and "relay (a)" and "relay (b)" are connected by connection lines No. 2, 5, 6, and 7.

[0049] When the correspondence relationships between the symbols, attributes, and connection lines are provisionally determined, the determination unit 104 refers to the association information stored in the association recording unit 106 and determines whether the correspondence relationships between the symbols, attributes, and connection lines have been correctly determined based on the provisionally determined correspondence relationships. Specifically, the determination unit 104 determines whether the provisionally determined correspondence relationships are correct by comparing the association information stored in the association recording unit 106 with a preset threshold value. For example, here, as described above, the judgment unit 104 provisionally judges that the relay (a) and the relay (b) are connected. Then, the judgment unit 104, for example, refers to the symbol-connection information of the relevance information, and compares the number of times that the relay (a) and the relay (b) are associated with each other with a preset threshold (hereinafter referred to as the "threshold for symbol connection judgment"). For example, if the number of times defined in the symbol-connection information exceeds the threshold for symbol connection judgment, the judgment unit 104 judges that the provisional judgment that the relay (a) and the relay (b) are connected is correct. Here, for example, if the threshold for symbol connection judgment is "10", the number of times that the relay (a) and the relay (b) are associated with each other in the symbol-connection information is "25" (see 702 in FIG. 7), so the judgment unit 104 judges that the relay (a) and the relay (b) are connected. That is, the judgment unit 104 judges that the relay (a) and the relay (b) are recognized in a correct correspondence relationship.

[0050] In addition, the determination unit 104 may delete information on the provisional determination that the relay (a) and the relay (b) are connected from each recognition result output from the recognition unit 102 when the number of times defined in the relevance information does not exceed a preset threshold, for example, when the number of times associated with the relay (a) and the relay (b) does not exceed the symbol connection determination threshold. Here, the determination unit 104 may delete the recognition results of "relay (a)", "relay (b)", "30X004", "30X003", and the connection lines No. 2, 5, 6, and 7, for example. Note that this is merely an example, and the determination unit 104 may not delete the provisionally determined information from the recognition result even when the number of times defined in the relevance information does not exceed a preset threshold.

[0051] For example, when the degree of association between a symbol and a destination symbol defined in the symbol-connection usage association information (see 704 in FIG. 7) is high but no association based on the symbol-connection information (see 702 in FIG. 7) is found, the determination unit 104 may associate the symbol with the destination symbol that is determined to have a high degree of association based on the symbol-connection usage association information. To give a specific example, suppose that the symbol recognition results show that symbols A and B are recognized, but not symbol C. Symbols A and C are symbols that are preferentially used in image drawing 1 used for the purpose of "home appliances," and symbol B is a symbol that is preferentially used in image drawing 1 used for the purpose of "switchboard." In addition, it is assumed that the symbol-connection information in the association degree information defines that the association degree between the A symbol and the C symbol is high, and the association degree between the A symbol and the B symbol is low. In this case, the determination unit 104 provisionally determines that the A symbol and the B symbol are connected based on the recognition result output from the recognition unit 102, but as a result of referring to the relevance information, the determination unit 104 re-determines that the A symbol and the B symbol are not highly related. On the other hand, it is assumed that the determination unit 104 is able to determine that the A symbol and the C symbol are highly related, for example, in the application of "home appliances", based on the relevance information. In this case, the determination unit 104 may, for example, replace the C symbol with the B symbol in the recognition result.

[0052] Furthermore, for example, when the determination unit 104 is unable to provisionally determine the correspondence between the symbol, the attribute, and the connection line from the recognition result, the determination unit 104 may perform association based on the relevance information. As a specific example, the determination unit 104 calculates the distance between "Fuse" and "FL3" based on the coordinates indicating the position of "Fuse" No. 3 in the symbol recognition result and the coordinates indicating the position of "FL3" No. 3 in the attribute recognition result. The determination unit 104 also calculates the distance between "Fuse" and "62LCTX" based on the coordinates indicating the position of "Fuse" No. 3 in the symbol recognition result and the coordinates indicating the position of "62LCTX" No. 5 in the attribute recognition result. As a result, the distance between "Fuse" and "FL3" and the distance between "Fuse" and "62LCTX" are the same, and it is not possible to determine which of "FL3" and "62LCTX" is associated with "Fuse". In this case, for example, the judgment unit 104 may perform association based on the relevance information to determine which combination has a higher relevance, that is, the combination of "Fuse" and "62LCTX" or the combination of "FL3" and "62LCTX", in other words, which combination appears more frequently.

[0053] In this way, the judgment unit 104 judges the correspondence between symbols, attributes, and other symbols in the inference data, taking into account, for example, the association between symbols and attributes, the association between symbols and connected symbols, the purpose of image drawing 1, or the model on which image drawing 1 is used, and makes a comprehensive judgment as to whether the symbols, attributes, and connections between symbols have been correctly recognized. Furthermore, for symbols or attributes that could not be associated based on the recognition result, the determination unit 104 can complement the association based on the association degree information.

[0054] The determination unit 104 outputs to the generation unit 107 the determination result as to whether or not the symbols, attributes, and connections between symbols in the inference data have been correctly recognized.

[0055] FIG. 9 is a diagram for explaining an image of an example of a determination result output by determining unit 104 in the first embodiment. The judgment result is information (hereinafter referred to as "associated judgment result information") set for each symbol, in which the symbol, the position of the symbol, the attribute associated with the symbol, the position of the attribute, the symbol's destination symbol, and the position of the destination symbol are associated, as shown in Fig. 9. If there are multiple destination symbols, all of the multiple destination symbols are associated with the associated judgment result information. In the associated judgment result information, the position of the symbol, the position of the attribute, and the position of the destination symbol are represented by, for example, the coordinates on the image drawing 1 of the center of the rectangle in which the symbol, the attribute, and the destination symbol are recognized, respectively.

[0056] When the determination unit 104 receives a notice from the control unit (not shown) that the determination end condition is not satisfied, the determination unit 104 re-determines whether or not the symbols, attributes, and connections between symbols in the inference data have been correctly recognized, based on the mismatch data stored in the mismatch data recording unit 110, the part information stored in the part recording unit 105, and the relevance information stored in the relevance recording unit 106. The re-determination performed by the determination unit 104 will be described later.

[0057] The generating unit 107 generates generated data equivalent to the image drawing 1 from the judgment result output by the judging unit 104 (hereinafter referred to as "post-judgment image drawing data"). More specifically, the generating unit 107 restores the image drawing 1 based on each piece of related judgment result information set in the judgment result output by the judging unit 104, and generates the post-judgment image drawing data. In the post-judgment image drawing data, the symbols, attributes, and connection destination symbols associated in each piece of related judgment result information are shown at the positions associated in each piece of related judgment result information. The generating unit 107 outputs the generated post-determination image drawing data to the comparing unit 108. The generating unit 107 outputs the determination result output by the determining unit 104 to the comparing unit 108 together with the post-determination image drawing data.

[0058] The comparison unit 108 compares the determined image drawing output by the generation unit 107 with the image drawing 1 acquired by the drawing acquisition unit 101. The comparison unit 108 may acquire the image drawing 1 from the drawing acquisition unit 101 via the recognition unit 102, the determination unit 104, and the generation unit 107.

[0059] The comparison between the post-determination image drawing and the image drawing 1 by the comparison unit 108 will be specifically described with reference to FIG. The comparison unit 108 compares the post-determination image drawing with the image drawing 1 for each symbol in the post-determination image drawing and each attribute associated with the symbol. Note that the symbol and the attribute are associated with each other by the related determination result information in the determination result output by the determination unit 104. Specifically, the comparison unit 108 compares, for example, the post-determination image drawing with the image drawing 1, and determines whether or not the symbols on the post-determination image drawing and the attributes associated with the symbols are also present at the same positions on the image drawing 1. Note that, at this time, the comparison unit 108 mainly compares the symbols and attributes, and excludes the connection lines.

[0060] The comparison unit 108 may determine the attribute associated with the symbol from the determination result output by the determination unit 104. As a specific example, if the determination result output by the determination unit 104 is as shown in Fig. 9, the comparison unit 108 determines whether or not "relay (a)" at the position (x,y)=(30,18) associated with the characters "30X004" at the position (x,y)=(31,10) exists on the image drawing 1. In addition, the comparison unit 108 does not necessarily require that the symbol on the judged image drawing and the attribute associated with the symbol are in the same position on the image drawing 1. For example, the comparison unit 108 extracts a range (first range) set in advance around the first reference point, with respect to the judged image drawing, as a reference point (first reference point). The first range is usually a rectangular area in which the symbol is recognized by the recognition unit 102. In addition, the comparison unit 108 extracts a range (second range) set in advance around the second reference point, with respect to the judged image drawing, as a reference point (second reference point). The second range is usually a rectangular area in which the attribute is recognized by the recognition unit 102. In addition, the comparison unit 108 extracts a range (third range) set in advance around the third reference point, with respect to the image drawing 1, as a reference point (third reference point). The third range is the same size as the first range. Also, the comparison unit 108 extracts a preset range (fourth range) centered on the fourth reference point, using a position on the image drawing 1 that corresponds to the position of the attribute associated with the symbol on the judged image drawing as a reference point (fourth reference point) for the image drawing 1. The fourth range is the same size as the second range. Then, the comparison unit 108, for example, compares the extracted post-judgment image drawing of the first range with the extracted image drawing 1 of the third range by template matching, and calculates the similarity (first similarity) between the post-judgment image drawing and the image drawing 1. Also, the comparison unit 108, for example, compares the extracted post-judgment image drawing of the second range with the extracted image drawing 1 of the fourth range by template matching, and calculates the similarity (second similarity) between the post-judgment image drawing and the image drawing 1. The comparison unit 108 may compare the feature amounts of the extracted post-judgment image drawing and the image drawing 1, and calculate the similarity between the post-judgment image drawing and the image drawing 1. The comparison unit 108 determines whether the symbol on the post-determination image drawing and the attribute associated with the symbol are also present in the same position on the image drawing 1, depending on whether the calculated first similarity and second similarity are equal to or greater than a predetermined threshold (hereinafter referred to as the "similarity determination threshold").

[0061] For example, when the calculated first similarity and second similarity are both equal to or greater than the similarity judgment threshold, comparison unit 108 judges that the symbol on the post-judgment image drawing and the attribute associated with the symbol are also present at the same positions on image drawing 1. In other words, comparison unit 108 judges that the symbol on the post-judgment image drawing and the attribute associated with the symbol match the symbol on image drawing 1 and the attribute associated with the symbol. In this case, comparison unit 108 records related judgment result information regarding the symbols judged to match in match data recording unit 109 as match data.

[0062] On the other hand, when either or both of the calculated first similarity or second similarity is not equal to or greater than the similarity determination threshold, the comparison unit 108 determines that the symbol on the determined image drawing and the attribute associated with the symbol do not exist at the same position on the image drawing 1. In other words, the comparison unit 108 determines that the symbol on the determined image drawing and the attribute associated with the symbol do not match the symbol on the image drawing 1 and the attribute associated with the symbol. In this case, the comparison unit 108 records the symbol recognition result for the symbol determined to be mismatched and the attribute recognition result for the attribute associated with the symbol determined to be mismatched in the mismatch data recording unit 110. For example, the comparison unit 108 may acquire the recognition result (see FIG. 5) from the recognition unit 102 via the judgment unit 104 and the generation unit 107, and extract the symbol recognition result for the symbol determined to be mismatched and the attribute recognition result for the attribute associated with the symbol determined to be mismatched from the acquired recognition result. In addition, when extracting mismatched data, the comparison unit 108 also determines that the connection lines connected to the symbols determined to be mismatched are also mismatched, and records the connection line recognition results for those connection lines in the mismatched data recording unit 110.

[0063] The comparison section 108 performs such a comparison process on all related determination result information included in the determination result output from the determination section 104 .

[0064] Here, Figure 10 is an explanatory diagram showing an image of the process in which the comparison unit 108 compares the post-determination image drawing output by the generation unit 107 with the image drawing 1 acquired by the drawing acquisition unit 101 in embodiment 1. For the sake of convenience, FIG. 10 is not related to the judgment result of FIG. 9, but is merely a diagram for explaining an image of a comparison between the post-judgment image diagram and image diagram 1. For ease of explanation, Fig. 10 shows areas including matched data and mismatched data extracted and enlarged from image drawing 1 and the post-determination image drawing. In Fig. 10, the area extracted and enlarged from image drawing 1 is indicated by 1001, and the area extracted and enlarged from the post-determination image drawing is indicated by 1002. For example, the "coil" associated with the "88TBL1C" shown in the area indicated by 1002 does not exist in the area indicated by 1001 at a position corresponding to the area indicated by 1002. Therefore, the comparison unit 108 records the symbol recognition result for the "coil" and the attribute recognition result for the "88TBL1C" in the mismatch data recording unit 110 as mismatch data.

[0065] Drawing recognition device 100 repeats re-determination by determination unit 104, generation of a post-determination image drawing by generation unit 107, and comparison processing by comparison unit 108 until a preset condition (hereinafter referred to as a "determination end condition") is satisfied. The "determination end condition" is, for example, "determination processing has been performed a preset number of times" or "there is no mismatching data and it has been determined that all data is matching." The "determination end condition" can be set appropriately by the user. Specifically, for example, when the comparison unit 108 records match data or mismatch data, the control unit of the drawing recognition device 100 determines whether or not the judgment termination condition is satisfied. If it is determined that the judgment termination condition is not satisfied, the control unit notifies the judgment unit 104 that the judgment termination condition is not satisfied. When the judgment unit 104 receives the notification from the comparison unit 108 that the judgment termination condition is not satisfied, it re-determines whether or not the symbols, attributes, and connections between symbols in the inference data have been correctly recognized, based on the mismatch data stored in the mismatch data recording unit 110, the part information stored in the part recording unit 105, and the relevance information stored in the relevance recording unit 106. The mismatched data recorded in the mismatched data recording unit 110 is obtained by removing, from the recognition results, the symbol recognition results for the symbols determined to be matching data by the comparing unit 108, the attribute recognition results for the attributes determined to be matching data by the comparing unit 108, and the connection line recognition results for the connection lines determined to be matching data by the comparing unit 108. Therefore, when re-determination is performed by the determining unit 104, the range of symbols, attributes, or connecting lines to be associated is narrowed, and the determination can be made more accurately.

[0066] When it is determined that the judgment end condition is satisfied, the control unit notifies the judgment unit 104 and the drawing display unit 111 that the judgment end condition is satisfied. When the judgment unit 104 receives the notification from the control unit that the judgment end condition is satisfied, the judgment process ends. That is, the comparison unit 108 does not perform the comparison process thereafter. Note that, here, the control unit of the drawing recognition device 100 determines whether or not the judgment end condition is satisfied and notifies based on the judgment result, but this is merely an example. For example, the judgment unit 104 or the comparison unit 108 may determine whether or not the judgment end condition is satisfied and notify based on the judgment result.

[0067] The matched data recording unit 109 records the matched data output by the comparison unit . The mismatched data recording section 110 records the mismatched data output by the comparison section .

[0068] The drawing display unit 111 displays the matched data recorded in the matched data recording unit 109 on the display 301 of the display device 300 . Specifically, the drawing display unit 111 converts the matching data into CAD data using, for example, a CAD application, and displays the CAD data on the display 301 of the display device 300. The graphic display unit 111 displays the matching data when the above-mentioned determination end condition is satisfied. For example, as described above, when the determination end condition is satisfied, the control unit notifies the drawing display unit 111 that the determination end condition is satisfied. When the drawing display unit 111 is notified by the comparison unit 108 that the determination end condition is satisfied, the drawing display unit 111 refers to the matched data recording unit 109 and displays the matched data.

[0069] The operation of the drawing recognition device 100 according to the first embodiment will be described. FIG. 11 is a flowchart for explaining the operation of drawing recognition apparatus 100 according to the first embodiment.

[0070] The drawing acquisition unit 101 acquires the image drawing 1 and the drawing characteristic information 2 inputted from the user via the operation input device 200 (step ST1101). The drawing acquisition unit 101 performs a predetermined process on the image drawing 1 to generate inference data. The drawing acquisition unit 101 outputs the inference data generated based on the image drawing 1 and the acquired drawing characteristic information 2 to the recognition unit 102.

[0071] The recognition unit 102 acquires a symbol recognition result, an attribute recognition result, or a connection line recognition result by performing inference using the first machine learning model 131, the second machine learning model 132, or the third machine learning model 133 stored in the inference device storage unit 103 based on the inference data and drawing characteristic information 2 output from the drawing acquisition unit 101 in step ST1101. As a result, the recognition unit 102 recognizes the symbol information, the attribute information, or the connection line information (step ST1102). The recognition unit 102 outputs the inference result, in other words, the symbol recognition result, the attribute recognition result, or the connection line recognition result, to the determination unit 104 .

[0072] FIG. 12 is a flowchart for explaining details of the inference operation performed by the recognition unit 102 in step ST1102 in FIG. The symbol recognition unit 1021 acquires a symbol recognition result based on the inference data and drawing characteristic information 2 output from the drawing acquisition unit 101 in step ST1101 of FIG. Specifically, the symbol recognition unit 1021 inputs the image feature amount extracted from the inference data and the feature amount extracted from the drawing characteristic information 2 to the first machine learning model 131, and obtains a symbol recognition result (step ST1201). The symbol recognition unit 1021 outputs the obtained symbol recognition result to the determination unit 104 .

[0073] The attribute information recognition unit 1022 acquires an attribute recognition result based on the inference data and drawing characteristic information 2 output from the drawing acquisition unit 101 in step ST1101 of FIG. Specifically, the attribute information recognition unit 1022 inputs the image feature amount extracted from the inference data and the feature amount extracted from the drawing characteristic information 2 to the second machine learning model 132 to obtain the attribute recognition result (step ST1202). The attribute information recognition unit 1022 outputs the acquired attribute recognition result to the determination unit 104 .

[0074] The connection line recognition unit 1023 acquires a connection line recognition result based on the inference data and drawing characteristic information 2 output from the drawing acquisition unit 101 in step ST1101 of FIG. Specifically, the connection line recognition unit 1023 inputs the image features extracted from the inference data and the features extracted from the drawing characteristic information 2 to the third machine learning model 133 to obtain the connection line recognition result (step ST1203). The connection line recognition unit 1023 outputs the acquired connection line recognition result to the determination unit 104 .

[0075] Returning to the explanation of the flowchart in FIG. The judgment unit 104 judges the correspondence between the symbols, attributes, and other symbols in the inference data based on the symbol recognition result, attribute recognition result, or connection line recognition result output by the recognition unit 102, the part information stored in the part recording unit 105, and the relevance information stored in the relevance recording unit 106, and makes a comprehensive judgment as to whether the symbols, attributes, and connections between symbols have been correctly recognized.

[0076] The determination unit 104 first refers to the part recording unit 105, compares the part information with the attribute recognition result, and determines whether or not an attribute included in the attribute recognition result matches the tool number defined in the part information (step ST1103). If an attribute included in the attribute recognition result does not match the tool number defined in the part information, the determination unit 104 discards the attribute from the attribute recognition result.

[0077] Next, the determination unit 104 refers to the symbol recognition result, the attribute recognition result, or the connection line recognition result, and infers the association between the symbol, the attribute, and the connection line based on the coordinates indicating the position of the symbol, the coordinates indicating the position of the attribute, and the coordinates indicating the position of the connection line from each recognition result, and tentatively determines the correspondence between the symbol, the attribute, and the connection line. Then, the judgment unit 104 refers to the association information stored in the association recording unit 106 and judges whether or not the association between the symbol, the attribute, and the connection line has been correctly determined based on the provisionally determined association (step ST1104). The determination unit 104 outputs to the generation unit 107 a determination result as to whether or not the correspondence between the symbols, attributes, and connection lines in the inference data has been correctly determined.

[0078] The generating unit 107 generates post-determination image drawing data from the determination result output by the determining unit 104 in step ST1104 (step ST1105). The generating unit 107 outputs the generated post-determination image drawing data to the comparing unit 108.

[0079] The comparison unit 108 compares the post-determination image drawing output by the generation unit 107 in step ST1105 with the image drawing 1 acquired by the drawing acquisition unit 101 in step ST1101 (step ST1106). Specifically, the comparison unit 108 compares, for example, the post-determination image drawing with the image drawing 1, and determines whether or not the symbols on the post-determination image drawing and the attributes associated with the symbols are also present at the same positions on the image drawing 1. Note that, at this time, the comparison unit 108 mainly compares the symbols and attributes, and excludes the connection lines.

[0080] If the comparison unit 108 determines, as a result of the comparison judgment, that the symbol on the judged image drawing and the attributes associated with the symbol match the symbol on the image drawing 1 and the attributes associated with the symbol ("YES" in step ST1106), it records the related judgment result information regarding the symbol that has been judged to match as matching data in the matching data recording unit 109 (step ST1107).

[0081] If the comparison unit 108 determines, as a result of the comparison judgment, that the symbol on the judged image drawing and the attribute associated with the symbol are inconsistent with the symbol on the image drawing 1 and the attribute associated with the symbol ("NO" in step ST1106), it records the symbol recognition result for the symbol judged to be inconsistent and the attribute recognition result for the attribute associated with the symbol judged to be inconsistent as inconsistent data in the inconsistency data recording unit 110 (step ST1108).

[0082] The control unit of the drawing recognition apparatus 100 judges whether or not a judgment end condition is satisfied (step ST1109). When it is determined that the judgment end condition is not satisfied (in the case of "NO" in step ST1109), the control unit notifies the judgment unit 104 that the judgment end condition is not satisfied. Then, the operation of the drawing recognition device 100 returns to step ST1103, and the judgment unit 104 re-determines whether or not the symbols, attributes, and connections between symbols in the inference data have been correctly recognized, based on the mismatch data stored in the mismatch data recording unit 110, the part information stored in the part recording unit 105, and the relevance information stored in the relevance recording unit 106.

[0083] When it is determined that the judgment end condition is satisfied (in the case of "YES" in step ST1109), the control unit notifies the judgment unit 104 and the image display unit 111 that the judgment end condition is satisfied. When the judgment unit 104 receives the notification that the judgment end condition is satisfied from the control unit, the judgment unit 104 ends the judgment process. That is, the comparison unit 108 does not perform the comparison process thereafter. When the control unit notifies the drawing display unit 111 in step ST1109 that the determination end condition has been satisfied, the drawing display unit 111 displays the matched data recorded in the matched data recording unit 109 on the display 301 of the display device 300 (step ST1110).

[0084] In this way, in the drawing recognition device 100, the recognition unit 102 can use the first machine learning model 131, the second machine learning model 132, and the third machine learning model 133 to recognize symbol information regarding one or more symbols contained in the image drawing 1, attribute information regarding attributes associated with the symbols, and connection line information regarding the connection lines between the symbols. Furthermore, in the drawing recognition device 100, the judgment unit 104 compares the recognition result (symbol recognition result, attribute recognition result, or connection line recognition result) output from the recognition unit 102 with part information and relevance information to determine the correspondence between the symbols, attributes, and other symbols in the image drawing 1, and makes a comprehensive judgment as to whether the symbols, attributes, and connections between the symbols have been correctly recognized, thereby improving accuracy. Furthermore, in the drawing recognition device 100, the comparison unit 108 can detect errors in the recognition of the symbol or attribute by comparing the symbols and the attributes associated with the symbols between the post-determination image drawing data generated by the generation unit 107 and the image drawing 1, and can select a recognition result with high accuracy.

[0085] In the above-described first embodiment, in drawing recognition device 100, determination unit 104 compares the recognition result output from recognition unit 102 with both the part information stored in part recording unit 105 and the relevance information recorded in relevance recording unit 106, but this is merely an example. Determination unit 104 may compare only with either the part information or the relevance information, or may determine the recognition result without comparing with either the part information or the relevance information. When no matching is performed with either the part information or the relevance information, the determination unit 104 determines the correspondence between the symbol, the attribute, and other symbols in the image drawing 1 based on the information on the symbol, the information on the attribute, and the information on the connection line recognized by the recognition unit 102. Specifically, for example, the determination unit 104 refers to the symbol recognition result, the attribute recognition result, or the connection line recognition result, calculates the distance between the symbol, the attribute, and the connection line from each recognition result, and determines the correspondence between the symbol, the attribute, and the connection line according to the calculated distance. However, by comparing the recognition result with part information or relevance information, the determination unit 104 can determine the correspondence between symbols, attributes, and connection lines with higher accuracy than when the recognition result is determined without comparing with either part information or relevance information.

[0086] Furthermore, an interface may be provided that allows the user to select the required recognition result or judgment result by referring to the symbol recognition result, attribute recognition result, or connection line recognition result output by the recognition unit 102 (see FIG. 5) or the judgment result output by the judgment unit 104 (see FIG. 9). Specifically, for example, a display control unit (not shown) of the drawing recognition device 100 causes each recognition result output by the recognition unit 102 or the judgment result output by the judgment unit 104 to be displayed on the display 301 of the display device 300. The user understands the contents of each recognition result or judgment result by checking the display 301. For example, if there is a clearly erroneous symbol, attribute, connection line, etc., the user operates the keyboard 201 or mouse 202 of the operation input device 200 to input an instruction to delete the clearly erroneous symbol, attribute, connection line, etc. The control unit of the drawing recognition device 100 accepts the instruction input by the user and deletes information about the symbol, attribute, connection line, etc. from the recognition result or judgment result based on the accepted instruction. With this configuration, the drawing recognition device 100 can select information related to symbols, attributes, connection lines, or the like that is required by the user.

[0087] In the above-described first embodiment, the inference unit storage unit 103 is provided in the drawing recognition device 100, but this is merely an example. The inference unit storage unit 103 may be provided in a location outside the drawing recognition device 100 that can be referenced by the drawing recognition device 100. Specifically, for example, a single or multiple network storage devices (not shown) arranged on a communication network may have the function of the inference unit storage unit 103 and store the first machine learning model 131, the second machine learning model 132, and the third machine learning model 133. The drawing recognition device 100 accesses the network storage device. As a result, the external network storage device can accumulate the first machine learning model 131, the second machine learning model 132, and the third machine learning model 133, and a database can be constructed outside the drawing recognition device 100.

[0088] In the above first embodiment, the part recording unit 105, the relevance recording unit 106, the matching data recording unit 109, and the mismatching data recording unit 110 are provided in the drawing recognition device 100, but this is merely an example. The part recording unit 105, the relevance recording unit 106, the matching data recording unit 109, and the mismatching data recording unit 110 may be provided in a location outside the drawing recognition device 100 that can be referenced by the drawing recognition device 100.

[0089] In the above-described first embodiment, the drawing recognition device 100 includes the drawing acquisition unit 101, the recognition unit 102, the inference unit storage unit 103, the judgment unit 104, the part recording unit 105, the relevance recording unit 106, the generation unit 107, the comparison unit 108, the matched data recording unit 109, the mismatched data recording unit 110, and the drawing display unit 111, but this is merely an example. The drawing recognition device 100 may include at least the drawing acquisition unit 101, the recognition unit 102, and the judgment unit 104. In this case, for example, the generation unit 107, the comparison unit 108, and the drawing display unit 111 may be provided in a device (a drawing generation device, not shown) external to the drawing recognition device 100, and the drawing generation system may be configured by the drawing recognition device 100 and the drawing generation device. The inference unit storage unit 103, the part recording unit 105, the matched data recording unit 109, and the mismatched data recording unit 110 are provided in, for example, a server that can be referenced by the drawing recognition device 100 and the drawing generation device.

[0090] The drawing recognition device 100 according to the first embodiment is suitable for use in, for example, restoring CAD data from a paper drawing or converting CAD data for a different CAD system. Specifically, for example, a user inputs a printout created with a specific CAD system into the drawing recognition device 100 as an image drawing 1. The drawing recognition device 100 recognizes symbols, attributes, or connection lines present on the input image drawing 1, and records the matching data. By inputting the recorded matching data into the specific CAD system, the user can restore the printout to CAD data, and can reuse the CAD data for design revision or improvement, etc. Furthermore, the user can convert the above-mentioned matching data into CAD data for use in another CAD system by, for example, separately providing a conversion program for converting the data into a data format of a different CAD system. In this manner, the drawing recognition device 100 according to the first embodiment can improve the accuracy of restoring CAD data in a CAD system by being used to restore CAD data in the CAD system, for example.

[0091] 13A and 13B are diagrams illustrating an example of a hardware configuration of the drawing recognition device 100 according to the first embodiment. In the first embodiment, the functions of the drawing acquisition unit 101, the recognition unit 102, the determination unit 104, the generation unit 107, the comparison unit 108, and the drawing display unit 111 are realized by a processing circuit 1301. That is, the drawing recognition device 100 includes the processing circuit 1301 for recognizing symbols and the like on the input image drawing 1. The processing circuit 1301 may be dedicated hardware as shown in FIG. 13A, or may be a CPU (Central Processing Unit) 1304 that executes a program stored in a memory 1305 as shown in FIG. 13B.

[0092] When the processing circuitry 1301 is dedicated hardware, the processing circuitry 1301 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0093] When the processing circuit 1301 is a CPU 1304, the functions of the drawing acquisition unit 101, the recognition unit 102, the determination unit 104, the generation unit 107, the comparison unit 108, and the drawing display unit 111 are realized by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in the memory 1305. The processing circuit 1301 executes the functions of the drawing acquisition unit 101, the recognition unit 102, the determination unit 104, the generation unit 107, the comparison unit 108, and the drawing display unit 111 by reading and executing the program stored in the memory 1305. That is, the drawing recognition device 100 includes a memory 1305 for storing a program that is executed as a result of steps ST1101 to ST1110 in FIG. 11 and steps ST1201 to ST1203 in FIG. 12 when executed by the processing circuit 1301. It can also be said that the programs stored in the memory 1305 cause the computer to execute the procedures or methods of the drawing acquisition unit 101, the recognition unit 102, the determination unit 104, the generation unit 107, the comparison unit 108, and the drawing display unit 111. Here, the memory 1305 corresponds to, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a DVD (Digital Versatile Disc), or the like.

[0094] The functions of the drawing acquisition unit 101, the recognition unit 102, the determination unit 104, the generation unit 107, the comparison unit 108, and the drawing display unit 111 may be partially realized by dedicated hardware and partially realized by software or firmware. For example, the function of the drawing acquisition unit 101 may be realized by a processing circuit 1301 as dedicated hardware, and the functions of the recognition unit 102, the determination unit 104, the generation unit 107, the comparison unit 108, and the drawing display unit 111 may be realized by the processing circuit 1301 reading out and executing a program stored in a memory 1305. Also, for example, an HDD is used for the inference unit storage unit 103, the part recording unit 105, the relevance recording unit 106, the matching data recording unit 109, and the mismatching data recording unit 110. Note that this is just one example, and the inference unit storage unit 103, the part recording unit 105, the relevance recording unit 106, the matching data recording unit 109, and the mismatching data recording unit 110 may be configured with the memory 1305, an SSD (Solid State Drive), a DVD, or the like. The drawing recognition apparatus 100 also includes an input interface device 1302 and an output interface device 1303 that perform wired or wireless communication with devices such as the operation input device 200 or the display device 300.

[0095] In the first embodiment, first machine learning model 131, second machine learning model 132, and third machine learning model 133 are generated by learning device 400 (see FIG. 14 described later) performing learning in machine learning. Hereinafter, learning device 400 will be described.

[0096] FIG. 14 is a diagram illustrating an example of a configuration of a learning device 400 according to the first embodiment. The learning device 400 is connected to the drawing recognition device 100 via a network. The learning device 400 generates a first machine learning model 131, a second machine learning model 132, and a third machine learning model 133 based on the input image drawing 1 and drawing characteristic information 2. The image drawing 1 is input together with a teacher label. The learning device 400 trains a machine learning model composed of a neural network by so-called supervised learning. The teacher label is information indicating a symbol, an attribute, and a connection line on the image drawing 1. For example, the teacher label regarding a symbol is text data in which coordinates indicating the position of the symbol on the image drawing 1 are associated with information that can identify the symbol. The position of the symbol is represented by, for example, the coordinates on the image drawing 1 of the four corners of a rectangle when the symbol is surrounded by the rectangle. The information that can identify the symbol may be, for example, the name of the symbol. The teacher label regarding the attribute and the teacher label regarding the connection line are also text data in which coordinates indicating the position of the attribute or connection line on the image drawing 1 are associated with information that can identify the attribute or connection line, similar to the teacher label regarding the symbol. The training data is generated in advance for each image drawing 1 by, for example, a user.

[0097] The learning device 400 stores the generated first machine learning model 131, second machine learning model 132, and third machine learning model 133 in the inference device storage unit 103 of the drawing recognition device 100. The learning device 400 also stores the input drawing characteristic information 2 in the relevance recording unit 106. Note that the relevance recording unit 106 stores relevance information (see FIG. 7) that has been generated in advance by the user. The user generates relevance information in advance based on the image drawing 1 and the drawing characteristic information 2. The learning device 400 stores the drawing characteristic information 2 together with the relevance information. In addition, in FIG. 14, for the sake of simplicity, only the inference device storage unit 103 and the relevance recording unit 106 are shown as components of the drawing recognition device 100, but the configuration example of the drawing recognition device 100 is the same as the configuration example of the drawing recognition device 100 described using FIG. 1.

[0098] 14, learning device 400 is assumed to be a separate device from drawing recognition device 100 and connected to drawing recognition device 100, but this is merely an example. Learning device 400 may be mounted on drawing recognition device 100.

[0099] The learning device 400 includes a learning data acquisition unit 401 , a symbol model generation unit 402 , an attribute information model generation unit 403 , and a connection line model generation unit 404 .

[0100] The learning data acquisition unit 401 performs data acquisition processing for learning. Specifically, the learning data acquisition unit 401 acquires a plurality of image drawings 1 to which teacher labels have been assigned, and drawing characteristic information 2 corresponding to the image drawings 1. The teacher labels are, for example, text data as described above. Specifically, the learning data acquisition unit 401 acquires a plurality of image drawings 1 and text data for each image drawing 1. The plurality of image drawings 1 to which teacher labels have been assigned and the drawing characteristic information 2 are input by a user. For example, the learning device 400 is connected to the operation input device 200 (see FIG. 1), and the user inputs the plurality of image drawings 1 and the drawing characteristic information 2 from the operation input device 200.

[0101] Then, the learning data acquisition unit 401 generates learning data based on the acquired image drawing 1. Specifically, the learning data acquisition unit 401 first divides the acquired image drawing 1 into data of, for example, 224×224 pixel size. The image drawing 1 may be divided vertically and horizontally at equal intervals of 224×224 pixel size, and overlapping portions may be included when the image drawing 1 is divided. Note that the learning data acquisition unit 401 does not need to perform the above-mentioned processing if, for example, the acquired image drawing 1 is within the size limit that can be used as an input for a machine learning model. Furthermore, the learning data acquisition unit 401 analyzes the acquired drawing characteristic information 2 and extracts feature amounts from the drawing characteristic information 2. The learning data acquisition unit 401 extracts feature amounts from the drawing characteristic information 2 as numerical values, such as "0" for "home appliance", "1" for "distribution board", and "2" for "car". The learning data acquisition unit 401 generates data including the divided image drawing 1 to which the teacher label has been added and the feature amount extracted from the drawing characteristic information 2 as learning data. The learning data acquisition unit 401 outputs the generated learning data to the symbol model generation unit 402 , the attribute information model generation unit 403 , and the connection line model generation unit 404 .

[0102] In this embodiment, the learning data acquisition unit 401 performs an analysis process on the drawing characteristic information 2, extracts a feature amount, and includes the feature amount in the learning data, but this is merely an example. For example, the learning data acquisition unit 401 may use the divided image drawing 1 to which a teacher label is added and the drawing characteristic information 2 as the learning data. In this case, the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404 described below each perform the above-mentioned analysis process on the drawing characteristic information 2 included in the learning data when generating a machine learning model. Furthermore, the division of the image drawing 1 as described above may be performed by the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404 described below. In this case, the learning data acquisition unit 401 includes the acquired image drawing 1 as it is in the learning data, and the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404 described below each divide the image drawing 1 when generating a machine learning model.

[0103] Furthermore, the learning data acquiring unit 401 stores the acquired drawing characteristic information 2 in the relevance recording unit 106. As described above, the relevance recording unit 106 stores relevance information in advance. In addition to the relevance information and the drawing characteristic information 2, the relevance recording unit 106 may store various information related to symbols, attributes, or connection lines.

[0104] The symbol model generation unit 402 learns to judge symbols based on the learning data output by the learning data acquisition unit 401, and generates a first machine learning model 131 that uses the image drawing 1 and drawing characteristic information 2 as input and outputs information regarding the number of symbols on the image drawing 1, the position information of each symbol, the type of each symbol, and the accuracy of each symbol as a symbol recognition result. In the first embodiment, the symbol model generating unit 402 performs learning related to object detection of a neural network using the learning data output by the learning data acquiring unit 401. That is, learning is performed by extracting image features, classifying them, and correcting coordinates. As a learning method, a well-known learning method such as SSD (Single Shot Detector) or YOLO (You Only Look Once) may be used. The symbol model generation unit 402 stores the generated first machine learning model 131 in the inference unit storage unit 103.

[0105] The attribute information model generation unit 403 learns to judge attributes based on the learning data output by the learning data acquisition unit 401, and generates a second machine learning model 132 that uses the image drawing 1 and drawing characteristic information 2 as input and outputs information regarding the number of attributes on the image drawing 1, the position information of each attribute, the type of each attribute, and the accuracy of each attribute as attribute recognition results. In the first embodiment, the attribute information model generating unit 403 performs learning related to object detection of a neural network using the learning data output by the learning data acquiring unit 401. That is, learning is performed by extracting image features, classifying them, and correcting coordinates. As a learning method, a well-known learning method such as VGSL (Variable-size Graph Specification Language) may be used. The attribute information model generation unit 403 stores the generated second machine learning model 132 in the inference unit storage unit 103.

[0106] The connection line model generation unit 404 learns to determine connection lines based on the learning data output by the learning data acquisition unit 401, and generates a third machine learning model 133 that uses the image drawing 1 and drawing characteristic information 2 as input and outputs information regarding the number of connection lines on the image drawing 1, the position information of each connection line, the type of each connection line, and the accuracy of each connection line, etc., as a connection line recognition result. In the first embodiment, the connection line model generating unit 404 performs learning related to object detection of a neural network using the learning data output by the learning data acquiring unit 401. That is, learning is performed by extracting image features, classifying them, and correcting coordinates. As a learning method, a well-known learning method such as SSD (Single Shot Detector) or YOLO (You Only Look Once) may be used. The connection line model generation unit 404 stores the generated third machine learning model 133 in the inference unit storage unit 103.

[0107] The operation of learning device 400 according to the first embodiment will be described. FIG. 15 is a flowchart for explaining the operation of learning device 400 according to the first embodiment.

[0108] The learning data acquiring unit 401 performs a process of acquiring data for learning (step ST1501). Specifically, the learning data acquisition unit 401 acquires a plurality of image drawings 1 to which teacher labels have been added, and the drawing characteristic information 2 corresponding to the image drawings 1. Then, the learning data acquisition unit 401 generates learning data based on the acquired image drawing 1. The learning data acquisition unit 401 outputs the generated learning data to the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404. Moreover, the learning data acquisition unit 401 stores the acquired drawing characteristic information 2 in the relevance recording unit 106 .

[0109] The symbol model generation unit 402 learns symbol determination based on the learning data output by the learning data acquisition unit 401 in step ST1501, and generates the first machine learning model 131 (step ST1502). The symbol model generation unit 402 stores the generated first machine learning model 131 in the inference unit storage unit 103.

[0110] The attribute information model generation unit 403 learns attribute determination based on the learning data output by the learning data acquisition unit 401 in step ST1501, and generates the second machine learning model 132 (step ST1503). The attribute information model generation unit 403 stores the generated second machine learning model 132 in the inference unit storage unit 103.

[0111] The connection line model generation unit 404 learns the determination of connection lines based on the learning data output by the learning data acquisition unit 401 in step ST1501, and generates the third machine learning model 133 (step ST1504). The connection line model generation unit 404 stores the generated third machine learning model 133 in the inference unit storage unit 103.

[0112] In this way, the learning device 400 can generate a first machine learning model 131, a second machine learning model 132, and a third machine learning model 133 based on multiple image drawings 1 and drawing characteristic information 2 through so-called supervised learning.

[0113] The learning device 400 can perform weighting based on the drawing characteristic information 2 by generating a first machine learning model 131, a second machine learning model 132, and a third machine learning model 133 based on the image drawing 1 and the drawing characteristic information 2. For example, if the feature extracted from the drawing characteristic information 2 indicates "home appliances", the learning device 400 will preferentially recognize symbols related to "home appliances" when training the machine learning model. In this way, by using the drawing characteristic information 2 in addition to the image drawing 1 as input to the machine learning model, the learning device 400 can weight the symbols, attributes, or connecting lines on the image drawing 1 based on the drawing characteristic information 2, and can generate a machine learning model that outputs recognition results that take into account the purpose of the image drawing 1.

[0114] In the above first embodiment, the learning device 400 generates a machine learning model based on a plurality of image drawings 1 and drawing characteristic information 2, but the learning device 400 does not require the drawing characteristic information 2 in learning. In this case, the learning data acquisition unit 401 of the learning device 400 does not need to acquire the drawing characteristic information 2. In addition, the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404 of the learning device 400 generate the first machine learning model 131, the second machine learning model 132, and the third machine learning model 133 based on the image drawing 1, respectively. However, by generating a machine learning model using the drawing characteristic information 2, the learning device 400 can generate a machine learning model with higher accuracy than when a machine learning model is generated without using the drawing characteristic information 2. In addition, if the learning device 400 does not use drawing characteristic information 2 in learning, the drawing recognition device 100 does not require drawing characteristic information 2 when performing inference based on the first machine learning model 131, the second machine learning model 132, and the third machine learning model 133.

[0115] 16A and 16B are diagrams illustrating an example of a hardware configuration of learning device 400 according to the first embodiment. In the first embodiment, the functions of the learning data acquisition unit 401, the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404 are realized by a processing circuit 1601. That is, the learning device 400 includes the processing circuit 1601 for generating a first machine learning model 131, a second machine learning model 132, and a third machine learning model 133 based on an image drawing 1 and drawing characteristic information 2. The processing circuit 1601 may be dedicated hardware as shown in FIG. 16A, or may be a CPU (Central Processing Unit) 1604 that executes a program stored in a memory 1605 as shown in FIG. 16B.

[0116] When the processing circuit 1601 is dedicated hardware, the processing circuit 1601 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0117] When the processing circuit 1601 is a CPU 1604, the functions of the learning data acquisition unit 401, the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404 are realized by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in the memory 1605. The processing circuit 1601 executes the functions of the learning data acquisition unit 401, the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404 by reading and executing the program stored in the memory 1605. That is, the learning device 400 includes a memory 1605 for storing a program that, when executed by the processing circuit 1601, results in the execution of steps ST1501 to ST1504 in FIG. 15 described above. It can also be said that the programs stored in the memory 1605 cause the computer to execute the procedures or methods of the learning data acquisition unit 401, the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404. Here, the memory 1605 corresponds to, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a DVD (Digital Versatile Disc), or the like.

[0118] The functions of the learning data acquisition unit 401, the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404 may be partly realized by dedicated hardware and partly realized by software or firmware. For example, the function of the learning data acquisition unit 401 can be realized by a processing circuit 1601 as dedicated hardware, and the functions of the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404 can be realized by the processing circuit 1601 reading out and executing a program stored in a memory 1605. The learning device 400 also includes an input interface device 1602 and an output interface device 1603 that perform wired or wireless communication with devices such as the drawing recognition device 100.

[0119] As described above, the drawing recognition device 100 of the first embodiment is configured to include the drawing acquisition unit 101 that acquires drawing data (image drawing 1), the recognition unit 102 that recognizes information about symbols in the drawing data acquired by the drawing acquisition unit 101, information about attributes in the drawing data acquired by the drawing acquisition unit 101, and information about connection lines in the drawing data acquired by the drawing acquisition unit 101 based on the drawing data acquired by the drawing acquisition unit 101 and a machine learning model that receives the drawing data as input and outputs a recognition result about symbols, attributes, and connection lines connecting the symbols, and the determination unit 104 that determines the correspondence between the symbols, attributes, and other symbols in the drawing data acquired by the drawing acquisition unit 101 based on the information about the symbols, information about the attributes, and information about the connection lines recognized by the recognition unit 102. Therefore, the drawing recognition device 100 can determine the correspondence between characters and symbols without requiring placement knowledge.

[0120] In addition, in the present disclosure, any of the components of the embodiments may be modified or any of the components of the embodiments may be omitted. [Explanation of symbols]

[0121] 1 Image drawing, 2 Drawing characteristic information, 100 Drawing recognition device, 101 Drawing acquisition unit, 102 Recognition unit, 1021 Symbol recognition unit, 1022 Attribute information recognition unit, 1023 Connection line recognition unit, 103 Inference device storage unit, 131 First machine learning model, 132 Second machine learning model, 133 Third machine learning model, 104 Judgment unit, 105 Part recording unit, 106 Association degree recording unit, 107 Generation unit, 108 Comparison unit, 109 Matching data recording unit, 110 Mismatching data recording unit, 111 Drawing display unit, 200 Operation input device, 201 Keyboard, 202 Mouse, 300 Display device, 301 Display, 400 Learning device, 401 Learning data acquisition unit, 402 Symbol model generation unit, 403 Attribute information model generation unit, 404 Connection line model generation unit, 1301, 1601 Processing circuit, 1302, 1602 input interface device, 1303, 1603 output interface device, 1304, 1604 CPU, 1305, 1605 memory.

Claims

1. A drawing acquisition unit that acquires drawing data; a recognition unit that recognizes, based on the drawing data acquired by the drawing acquisition unit and a machine learning model that receives the drawing data as an input and outputs recognition results related to symbols, attributes, and connection lines connecting the symbols on the drawing data, information related to the symbols in the drawing data acquired by the drawing acquisition unit, including position information of the symbols, information related to the attributes in the drawing data acquired by the drawing acquisition unit, including position information of the attributes, and information related to the connection lines in the drawing data acquired by the drawing acquisition unit, including position information of the connection lines; a determination unit that determines a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit, based on the information about the symbol, the information about the attribute, and the information about the connection line recognized by the recognition unit, The determination unit is Based on information on the symbol, the attribute, and the connection line recognized by the recognition unit, and on the degree of association between the symbol and the attribute shown on the drawing data, which is represented by the number of times the symbol and the attribute related to the symbol are shown together on the drawing data, or on the degree of association between the symbols, which is represented by the number of times the symbol and the connection destination symbol to which the symbol is connected are shown connected on the drawing data, the corresponding relationship between the symbol, the attribute, and other symbols in the drawing data acquired by the drawing acquisition unit is determined. A drawing recognition device characterized by the above.

2. The recognition unit is Based on the drawing data acquired by the drawing acquisition unit, a first machine learning model that receives the drawing data as an input and outputs a symbol recognition result related to the symbol on the drawing data, a second machine learning model that receives the drawing data as an input and outputs an attribute recognition result related to the attribute on the drawing data, and a third machine learning model that receives the drawing data as an input and outputs a connection line recognition result related to the connection line connecting the symbols on the drawing data, information related to the symbol, information related to the attribute, and information related to the connection line connecting the symbols in the drawing data acquired by the drawing acquisition unit are recognized.

2. The drawing recognition device according to claim 1.

3. The recognition unit is Based on the drawing data acquired by the drawing acquisition unit, drawing characteristic information related to characteristics of the drawing data, and the machine learning model, information related to the symbols, information related to the attributes, and information related to the connection lines connecting the symbols in the drawing data acquired by the drawing acquisition unit are recognized.

2. The drawing recognition device according to claim 1.

4. The recognition unit is Based on the drawing data acquired by the drawing acquisition unit, drawing characteristic information related to characteristics of the drawing data, and the machine learning model, information related to the symbols, information related to the attributes, and information related to the connection lines connecting the symbols in the drawing data acquired by the drawing acquisition unit are recognized.

3. The drawing recognition device according to claim 2.

5. The relevance information is The data includes at least one of symbol-attribute information in which a degree of association between the symbol and the attribute is defined for each symbol, symbol-connection information in which a degree of association between the symbol and other symbols connected to the symbol is defined for each symbol, usage association information in which a degree of association between the symbol and the attribute and a degree of association between the symbol and other symbols are defined for each usage in which the drawing data acquired by the drawing acquisition unit is used, and model association information in which a degree of association between the symbol and the attribute and a degree of association between the symbol and other symbols are defined for each model of equipment in which the drawing data acquired by the drawing acquisition unit is used.

2. The drawing recognition device according to claim 1.

6. a generating section for generating determined image drawing data based on a result of the determination by the determining section of the correspondence relationship between the symbol, the attribute, and another symbol in the drawing data; 6. The drawing recognition device according to claim 1, further comprising:

7. a comparison unit that calculates a similarity between the determined image drawing data generated by the generation unit and the drawing data acquired by the drawing acquisition unit, and records the similarity as matched data when the similarity is high, and records the similarity as non-matched data when the similarity is low.

7. The drawing recognition device according to claim 6, further comprising:

8. The determination unit is Based on the mismatch data recorded by the comparison unit, a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit is re-determined.

8. The drawing recognition device according to claim 7.

9. A drawing acquisition unit that acquires drawing data; a recognition unit that recognizes, based on the drawing data acquired by the drawing acquisition unit and a machine learning model that receives the drawing data as an input and outputs recognition results related to symbols, attributes, and connection lines connecting the symbols on the drawing data, information related to the symbols in the drawing data acquired by the drawing acquisition unit, including position information of the symbols, information related to the attributes in the drawing data acquired by the drawing acquisition unit, including position information of the attributes, and information related to the connection lines in the drawing data acquired by the drawing acquisition unit, including position information of the connection lines; a determination unit that determines a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit, based on the information about the symbol, the information about the attribute, and the information about the connection line recognized by the recognition unit, The determination unit is determining a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit based on information related to the symbol, the attribute, and the connection line recognized by the recognition unit, and on a degree of association between the symbol and the attribute shown on the drawing data, or on a degree of association between the connections between the symbols; The relevance information is The data includes at least one of symbol-attribute information in which a degree of association between the symbol and the attribute is defined for each symbol, symbol-connection information in which a degree of association between the symbol and other symbols connected to the symbol is defined for each symbol, usage association information in which a degree of association between the symbol and the attribute and a degree of association between the symbol and other symbols are defined for each usage in which the drawing data acquired by the drawing acquisition unit is used, and model association information in which a degree of association between the symbol and the attribute and a degree of association between the symbol and other symbols are defined for each model of equipment in which the drawing data acquired by the drawing acquisition unit is used. A drawing recognition device characterized by the above.

10. The recognition unit is Based on the drawing data acquired by the drawing acquisition unit, a first machine learning model that receives the drawing data as an input and outputs a symbol recognition result related to the symbol on the drawing data, a second machine learning model that receives the drawing data as an input and outputs an attribute recognition result related to the attribute on the drawing data, and a third machine learning model that receives the drawing data as an input and outputs a connection line recognition result related to the connection line connecting the symbols on the drawing data, information related to the symbol, information related to the attribute, and information related to the connection line connecting the symbols in the drawing data acquired by the drawing acquisition unit are recognized.

10. The drawing recognition device according to claim 9.

11. The recognition unit is Based on the drawing data acquired by the drawing acquisition unit, drawing characteristic information related to characteristics of the drawing data, and the machine learning model, information related to the symbols, information related to the attributes, and information related to the connection lines connecting the symbols in the drawing data acquired by the drawing acquisition unit are recognized.

10. The drawing recognition device according to claim 9.

12. a generating section for generating determined image drawing data based on a result of the determination by the determining section of the correspondence relationship between the symbol, the attribute, and another symbol in the drawing data; 12. The drawing recognition device according to claim 9, further comprising:

13. a comparison unit that calculates a similarity between the determined image drawing data generated by the generation unit and the drawing data acquired by the drawing acquisition unit, and records the similarity as matched data when the similarity is high, and records the similarity as non-matched data when the similarity is low. The drawing recognition device according to claim 12, further comprising:

14. The determination unit is Based on the mismatch data recorded by the comparison unit, a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit is re-determined.

14. The drawing recognition device according to claim 13.

15. A drawing acquisition unit that acquires drawing data; a recognition unit that recognizes, based on the drawing data acquired by the drawing acquisition unit and a machine learning model that receives the drawing data as an input and outputs recognition results related to symbols, attributes, and connection lines connecting the symbols on the drawing data, information related to the symbols in the drawing data acquired by the drawing acquisition unit, including position information of the symbols, information related to the attributes in the drawing data acquired by the drawing acquisition unit, including position information of the attributes, and information related to the connection lines in the drawing data acquired by the drawing acquisition unit, including position information of the connection lines; a determination unit that determines a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit, based on the information about the symbol, the information about the attribute, and the information about the connection line recognized by the recognition unit; a generating unit that generates determined image drawing data based on a result of the determination by the determining unit of the correspondence relationship between the symbol, the attribute, and another symbol in the drawing data; a comparison unit that calculates a similarity between the determined image drawing data generated by the generation unit and the drawing data acquired by the drawing acquisition unit, and records the similarity as matched data when the similarity is high, and records the similarity as non-matched data when the similarity is low. A drawing recognition device equipped with the above.

16. The recognition unit is Based on the drawing data acquired by the drawing acquisition unit, a first machine learning model that receives the drawing data as an input and outputs a symbol recognition result related to the symbol on the drawing data, a second machine learning model that receives the drawing data as an input and outputs an attribute recognition result related to the attribute on the drawing data, and a third machine learning model that receives the drawing data as an input and outputs a connection line recognition result related to the connection line connecting the symbols on the drawing data, information related to the symbol, information related to the attribute, and information related to the connection line connecting the symbols in the drawing data acquired by the drawing acquisition unit are recognized.

16. The drawing recognition device according to claim 15.

17. The recognition unit is Based on the drawing data acquired by the drawing acquisition unit, drawing characteristic information related to characteristics of the drawing data, and the machine learning model, information related to the symbols, information related to the attributes, and information related to the connection lines connecting the symbols in the drawing data acquired by the drawing acquisition unit are recognized.

16. The drawing recognition device according to claim 15.

18. The determination unit is Based on the information on the symbol, the attribute, and the connection line recognized by the recognition unit, and on the degree of association between the symbol and the attribute shown on the drawing data, or on the degree of association between the connection between the symbols, the attribute, and other symbols in the drawing data acquired by the drawing acquisition unit, a correspondence relationship is determined.

18. The drawing recognition device according to claim 15,

19. The determination unit is Based on the mismatch data recorded by the comparison unit, a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit is re-determined.

16. The drawing recognition device according to claim 15.

20. Computer, A drawing acquisition unit that acquires drawing data; a recognition unit that recognizes, based on the drawing data acquired by the drawing acquisition unit and a machine learning model that receives the drawing data as an input and outputs recognition results related to symbols, attributes, and connection lines connecting the symbols on the drawing data, information related to the symbols in the drawing data acquired by the drawing acquisition unit, including position information of the symbols, information related to the attributes in the drawing data acquired by the drawing acquisition unit, including position information of the attributes, and information related to the connection lines in the drawing data acquired by the drawing acquisition unit, including position information of the connection lines; a program for causing the program to function as a determination unit that determines a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit, based on information about the symbol, information about the attribute, and information about the connection line recognized by the recognition unit, the program comprising: The determination unit is Based on information on the symbol, the attribute, and the connection line recognized by the recognition unit, and on a degree of association between the symbol and the attribute shown on the drawing data, which is represented by the number of times the symbol and the attribute related to the symbol are shown together on the drawing data, or on a degree of association between the symbols, which is represented by the number of times the symbol and a destination symbol to which the symbol is connected are shown connected on the drawing data, the corresponding relationship between the symbol, the attribute, and other symbols in the drawing data acquired by the drawing acquisition unit is determined. A drawing recognition program comprising:

21. A computer comprising: A drawing acquisition unit that acquires drawing data; a recognition unit that recognizes, based on the drawing data acquired by the drawing acquisition unit and a machine learning model that receives the drawing data as an input and outputs recognition results related to symbols, attributes, and connection lines connecting the symbols on the drawing data, information related to the symbols in the drawing data acquired by the drawing acquisition unit, including position information of the symbols, information related to the attributes in the drawing data acquired by the drawing acquisition unit, including position information of the attributes, and information related to the connection lines in the drawing data acquired by the drawing acquisition unit, including position information of the connection lines; a program for causing the program to function as a determination unit that determines a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit, based on information about the symbol, information about the attribute, and information about the connection line recognized by the recognition unit, the program comprising: The determination unit is determining a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit based on information related to the symbol, the attribute, and the connection line recognized by the recognition unit, and on a degree of association between the symbol and the attribute shown on the drawing data, or on a degree of association between the connections between the symbols; The relevance information is The data includes at least one of symbol-attribute information in which a degree of association between the symbol and the attribute is defined for each symbol, symbol-connection information in which a degree of association between the symbol and other symbols connected to the symbol is defined for each symbol, usage association information in which a degree of association between the symbol and the attribute and a degree of association between the symbol and other symbols are defined for each usage in which the drawing data acquired by the drawing acquisition unit is used, and model association information in which a degree of association between the symbol and the attribute and a degree of association between the symbol and other symbols are defined for each model of equipment in which the drawing data acquired by the drawing acquisition unit is used. A drawing recognition program comprising:

22. A computer comprising: A drawing acquisition unit that acquires drawing data; a recognition unit that recognizes, based on the drawing data acquired by the drawing acquisition unit and a machine learning model that receives the drawing data as an input and outputs recognition results related to symbols, attributes, and connection lines connecting the symbols on the drawing data, information related to the symbols in the drawing data acquired by the drawing acquisition unit, including position information of the symbols, information related to the attributes in the drawing data acquired by the drawing acquisition unit, including position information of the attributes, and information related to the connection lines in the drawing data acquired by the drawing acquisition unit, including position information of the connection lines; a determination unit that determines a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit, based on the information about the symbol, the information about the attribute, and the information about the connection line recognized by the recognition unit; a generating unit that generates determined image drawing data based on a result of the determination by the determining unit of the correspondence relationship between the symbol, the attribute, and another symbol in the drawing data; a comparison unit that calculates a similarity between the determined image drawing data generated by the generation unit and the drawing data acquired by the drawing acquisition unit, and records the similarity as matched data when the similarity is high, and records the similarity as non-matched data when the similarity is low. A drawing recognition program to function as a.

23. The determination unit: Based on the mismatch data recorded by the comparison unit, a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit is re-determined.

23. The drawing recognition program according to claim 22.

Images