Image classification system, image classification method, and program

Abstract

The present disclosure addresses the problem of inhibiting, in a multi-label classification, a combination of two or more labels associated with an image from becoming an essentially improbable combination. An image classification system (1) comprises a classifier (23), an attribute determination unit (24), and a label setting unit (25). The classifier (23) associates, with a target image, two or more labels among a plurality of labels as association labels. The attribute determination unit (24) assigns the two or more association labels to one of a plurality of attributes. The label setting unit (25): sets, for one of the plurality of attributes that is assigned with one of the association labels, said one association label; and sets, for one of the plurality of attributes that is assigned with at least two of the association labels, one association label selected from said at least two association labels on the basis of a predetermined selection condition.

Description

Image classification system, image classification method, and program

[0001] The present disclosure relates to an image classification system, an image classification method, and a program.

[0002] The image classification model in Patent Document 1 is a model based on a convolutional neural network, and outputs a score for each label for an input image. The score indicates the level of confidence that the image has been correctly classified into a specific label. The image classification model can then associate multiple labels (multi-label) with an image based on the score for each label.

[0003] Also, the multi-label used to classify images is a hierarchical label. In this case, the score of each label is calculated from top to bottom. The image classification model calculates the score of the higher-level label, and the score of the lower-level label is multiplied by the score of the corresponding higher-level label.

[0004] As disclosed in Patent Literature 1, there is an image classification system that performs multi-label classification in which multiple labels are associated with one image (target image). However, when the number of labels used for image classification increases, the number of teacher images associated with some labels may be insufficient, making it difficult to accurately classify images that should be associated with those labels.

[0005] As a result, the combination of two or more labels associated with one image in multi-label classification may be an impossible combination.

[0006] Special table 2019-505063 publication

[0007] An object of the present disclosure is to provide an image classification system, an image classification method, and a program that can prevent combinations of two or more labels associated with an image from becoming combinations that are not actually possible when performing multi-label classification.

[0008] An image classification system according to one aspect of the present disclosure classifies target images through image processing. The image classification system includes a classifier, an attribute discrimination unit, and a label setting unit. The classifier associates two or more labels from a plurality of labels with the target image as related labels. The attribute discrimination unit assigns the two or more related labels to one of a plurality of attributes. The label setting unit sets one related label to an attribute from the plurality of attributes that has been assigned one of the related labels. The label setting unit assigns one related label selected from at least two related labels based on predetermined selection conditions to an attribute from the plurality of attributes that has been assigned at least two of the related labels.

[0009] An image classification method according to one aspect of the present disclosure classifies target images by image processing. The image classification method includes a classification step, an attribute determination step, and a label setting step. The classification step associates two or more labels from a plurality of labels with the target image as related labels. The attribute determination step assigns the two or more related labels to any of a plurality of attributes. The label setting step sets one related label for an attribute from the plurality of attributes that has been assigned one of the related labels. The label setting step sets one related label selected from at least two related labels based on predetermined selection conditions for an attribute from the plurality of attributes that has been assigned at least two of the related labels.

[0010] A program according to one aspect of the present disclosure causes a computer system to execute the image classification method described above.

[0011] FIG. 1 is a block diagram showing the configuration of an image classification system according to an embodiment. FIG. 2 is a plan view showing an original image used by the image classification system according to the embodiment. FIG. 3 is a diagram showing a symbol for an electrical outlet used by the image classification system according to the embodiment. FIG. 4 is a plan view showing a target image used by the image classification system according to the embodiment. FIG. 5 is a diagram showing related labels associated with target images by a classifier of the image classification system according to the embodiment. FIG. 6 is a diagram showing attribute information used by the image classification system according to the embodiment. FIG. 7 is a diagram showing the result of related label allocation by an attribute discrimination unit of the image classification system according to the embodiment. FIG. 8 is a diagram showing the result of related label setting by a label setting unit of the image classification system according to the embodiment. FIG. 9 is a flowchart showing an image classification method according to an embodiment. FIG. 10 is a diagram showing combination information used in the image classification system according to a first modified example. FIG. 11 is a block diagram showing the configuration of an image processing device provided in the image classification system according to a second modified example. FIG. 12 is a diagram showing characters recognized by a character recognition unit of the image classification device according to the embodiment. FIG. 13 is a diagram showing aggregated data output by the image classification system according to the embodiment.

[0012] The following embodiments relate to an image classification system, an image classification method, and a program, and more particularly to an image classification system, an image classification method, and a program that perform multi-label classification of target images.

[0013] The embodiment described below is merely an example of an embodiment of the present disclosure. The present disclosure is not limited to the following embodiment, and various modifications are possible depending on the design, etc., as long as the effects of the present disclosure can be achieved.

[0014] (Embodiments) (1) Overview The image classification system of this embodiment uses image processing technology with a computer system to perform multi-label classification, associating multiple labels with a target image. The target image is an image of an object such as a sign, symbol, character, person, vehicle, or animal. However, the object appearing in the target image may be other than those described above and is not limited to a specific object.

[0015] FIG. 1 shows a block diagram of an image classification system 1 according to this embodiment. The image classification system 1 classifies target images through image processing. The image classification system 1 includes a classifier 23, an attribute discrimination unit 24, and a label setting unit 25. The classifier 23 associates two or more labels from a plurality of labels with the target image as related labels. The attribute discrimination unit 24 assigns the two or more related labels to one of a plurality of attributes. The label setting unit 25 assigns one related label to an attribute that has been assigned one related label among the plurality of attributes. The label setting unit 25 assigns one related label selected from the at least two related labels based on predetermined selection conditions to an attribute that has been assigned at least two related labels among the plurality of attributes.

[0016] When performing multi-label classification, the image classification system 1 having the above-described configuration can prevent combinations of two or more labels associated with an image from becoming combinations that are essentially impossible.

[0017] (2) Details As shown in Fig. 1, the image classification system 1 includes an image processing device 2, an input device 3, and an output device 4. The image classification system 1 extracts a target image D2 (see Fig. 4) from an original image D1 (see Fig. 2) and performs multi-label classification by associating multiple labels with the target image D2. In this embodiment, image data of an architectural drawing is taken as data of the original image D1, and a symbol B (see Fig. 3) included in the architectural drawing is taken as the target image D2.

[0018] (2.1) Original Image FIG. 2 shows an example of an original image D1.

[0019] The original image D1 is an architectural drawing. Specifically, the original image D1 is a single-line diagram that shows the wiring of electrical outlets, lighting fixtures, and the like in a building with single lines. The original image D1 includes symbols such as the symbol B for an electrical outlet. In addition to the symbol B for an electrical outlet, the original image D1 also includes symbols for lighting fixtures, distribution boards, and the like.

[0020] The symbol B for an outlet varies depending on the type of outlet. Figure 3 shows symbols B1-B4 as examples of the symbol B for an outlet. Symbol B1 is the symbol for a [single-phase, 100V, wall-mounted] outlet. Symbol B2 is the symbol for a [single-phase, 100V, floor-mounted] outlet. Symbol B3 is the symbol for a [single-phase, 100V, ceiling-mounted] outlet. Symbol B4 is the symbol for a [single-phase, 200V, wall-mounted] outlet.

[0021] Data of the original image D1 is stored in a database 5. The database 5 is communicatively connected to a network NT, such as a local area network (LAN), established within an organization such as a company, factory, office, or department. The network NT is preferably a dedicated line for the organization, but may also include public communication networks such as a mobile phone network, the Internet, and a landline network. For example, the original image D1 is computer-aided design (CAD) data, and is stored in the database 5 from a CAD system (not shown).

[0022] (2.2) Input Device, Output Device The input device 3 has a user interface function that accepts user operations. The input device 3 has at least one user interface such as a touch panel display, a keyboard, and a mouse. The user operates the input device 3 to cause the image processing device 2 to execute the image classification method. The input device 3 outputs an operation signal corresponding to the user operation to the image processing device 2, and the image processing device 2 executes the image classification method in response to the operation signal.

[0023] The output device 4 includes a liquid crystal display device or an organic EL display device, and receives image data from the image processing device 2 and displays the image data. The image data is displayed, for example, as an operation screen for executing and stopping the image classification method, as well as a notification screen for the execution process and execution results of the image classification method. The user operates the input device 3 while looking at the screen displayed on the output device 4 to perform operations such as executing and stopping the image classification method, and to check the process and results of the image classification method. The input device 3 and output device 4 may be the operation unit and screen of a smartphone, tablet terminal, or the like. Furthermore, a display device is basically used as the output device 4, but a printer may also be used in combination.

[0024] (2.3) Image Processing Device The image processing device 2 includes an image acquisition unit 21, an extraction unit 22, a classifier 23, an attribute discrimination unit 24, a label setting unit 25, and a result output unit 26. The image processing device 2 performs multi-label classification on a target image D2 (see FIG. 4) included in an original image D1 (see FIG. 2).

[0025] The image processing device 2 preferably includes a computer system that executes a program to realize some or all of the functions of the image processing device 2. The computer system's main hardware configuration is a processor that operates according to the program. The processor may be of any type, as long as it can realize its functions by executing the program. The processor may be composed of one or more electronic circuits, including a semiconductor integrated circuit (IC) or a large-scale integration (LSI). While ICs and LSIs are used here, the names may vary depending on the degree of integration, and may also be called system LSIs, very large-scale integrations (VLSIs), or ultra-large-scale integrations (ULSIs). Field-programmable gate arrays (FPGAs), which are programmed after the LSI is manufactured, or reconfigurable logic devices that can reconfigure the connections within the LSI or set up circuit partitions within the LSI, can also be used for the same purpose. Multiple electronic circuits may be integrated on a single chip or provided on multiple chips. Multiple chips may be integrated into a single device or provided on multiple devices. The program is recorded on a non-transitory recording medium such as a computer system-readable ROM, an optical disk, a hard disk drive, etc. The program may be stored in the non-transitory recording medium in advance, or may be supplied to the non-transitory recording medium via a wide area communication network including the Internet.

[0026] Furthermore, some or all of the functions of the image processing device 2 may be implemented by a single computer device or by multiple computers linked to each other. For example, some or all of the functions of the image processing device 2 may be implemented as a cloud computing system. Furthermore, some or all of the functions of the image processing device 2 may be configured by a single device such as a personal computer.

[0027] (2.3.1) Image Acquisition Unit The image acquisition unit 21 functions as a communication interface for communicating via the network NT. The image acquisition unit 21 can acquire data of the original image D1 from the database 5 via the network NT.

[0028] (2.3.2) Extraction Unit The extraction unit 22 extracts a target image D2 from an original image D1. As shown in FIG. 2 , the original image D1 is an architectural drawing that includes at least one symbol B for an electrical outlet. In this embodiment, the image of symbol B extracted from the original image D1 is defined as the target image D2. Therefore, the extraction unit 22 extracts the target image D2, which is an image of symbol B, from the original image D1. Therefore, the image classification system 1 can classify all target images D2 included in the original image D1.

[0029] FIG. 4 shows an example of the target image D2, which is an image obtained by extracting the area in which the symbol B3 appears in the original image D1.

[0030] In this embodiment, the extraction unit 22 includes a learning model M22 that uses a neural network. The learning model M22 uses a large number of images of architectural drawings that include the symbol B as first teacher images, and constructs a neural network for extracting the symbol B from the original image D1 by learning using these first teacher images. When the learning model M22 receives the data of the original image D1 acquired by the image acquisition unit 21 as input, it recognizes the image of the symbol B included in the original image D1 and extracts a target image D2, which is an image of the symbol B, from the original image D1.

[0031] (2.3.3) Classifier The classifier 23 performs multi-label classification, associating two or more labels from a plurality of labels with the target image D2 as related labels.

[0032] In this embodiment, the classifier 23 includes a learning model M23 that uses a neural network. The learning model M23 uses a large number of images containing the symbol B as second teacher images, and by learning using these second teacher images, constructs a neural network for multi-label classification of the target image D2. Therefore, the image classification system 1 can easily realize the classifier 23 that performs multi-label classification.

[0033] In particular, it is preferable that the learning model M23 uses, as the neural network, a single neural network that performs multi-label classification trained using second teacher images in which a label is associated with each of a plurality of attributes. That is, the second teacher images are associated with one label corresponding to each of a plurality of attributes, and the single neural network trained using such second teacher images can perform multi-label classification in which two or more labels are associated with the target image D2. Therefore, the image classification system 1 can realize the classifier 23 that performs multi-label classification using a single neural network.

[0034] The multi-label classification by the classifier 23 will be described below.

[0035] The symbol B for an outlet varies depending on the type of outlet. The symbol B1 for an outlet shown in FIG. 3 is the symbol for a single-phase, 100V, wall-mounted outlet. The symbol B2 is the symbol for a single-phase, 100V, floor-mounted outlet. The symbol B3 is the symbol for a single-phase, 100V, ceiling-mounted outlet. The symbol B4 is the symbol for a single-phase, 200V, wall-mounted outlet. In other words, the shape of the symbol B for an outlet varies depending on the phase, voltage, and installation type of the outlet.

[0036] Therefore, in this embodiment, "phase," "voltage," and "installation type" are set as attributes of the symbol B for the outlet. The attribute "phase" corresponds to the phase of the outlet, the attribute "voltage" corresponds to the voltage of the outlet, and the attribute "installation type" corresponds to the installation type of the outlet. Labels belonging to the attribute "phase" are "single-phase" and "three-phase," labels belonging to the attribute "voltage" are "100V," "200V," and "250V," and labels belonging to the attribute "installation type" are "wall-mounted," "floor-mounted," and "ceiling-mounted."

[0037] The classifier 23 uses all labels belonging to the attributes "phase," "voltage," and "installation type," namely, "single phase," "three phase," "100V," "200V," "250V," "wall-mounted," "floor-mounted," and "ceiling-mounted," as multiple labels that can be associated with the target image D2.

[0038] Then, when the classifier 23 (learning model M23) receives the data of the target image D2 extracted by the extraction unit 22, it calculates the score for each of the above-mentioned multiple labels using the target image D2 as a classification target. The score indicates the reliability of the target image D2 for that label. In other words, it indicates the degree to which the target image D2 is correctly associated with that label. The classifier 23 associates, as related labels, labels among the multiple labels whose scores are equal to or greater than a threshold with the target image D2. Specifically, the related labels associated with the target image D2 are those of the multiple labels "single phase," "three phase," "100V," "200V," "250V," "wall-mounted," "floor-mounted," and "ceiling-mounted" whose scores are equal to or greater than a threshold. If there are two or more labels whose scores are equal to or greater than the threshold, the classifier 23 performs multi-label classification in which the classifier 23 calculates the score for each of the multiple labels for the target image D2 and extracts two or more related labels from the multiple labels based on the scores.

[0039] Here, if the number of second teacher images used for training the learning model M23 of the classifier 23 is insufficient, it becomes difficult to accurately classify the target image D2. As a result, the two or more related labels associated with one target image D2 in multi-label classification may include at least two related labels that belong to the same attribute.

[0040] However, even if the related labels associated with the target image D2 include two or more labels belonging to the same attribute, the classifier 23 associates any label with the target image D2 as long as the label has a score equal to or greater than a threshold. For example, as shown in FIG. 5, the classifier 23 associates related labels L1-L4 with the target image D2. The related label L1 is "single phase," the related label L2 is "100V," the related label L3 is "200V," and the related label L4 is "ceilinged." In this case, the related label L2 "100V" and the related label L3 "200V" belong to the same attribute "voltage," and the combination of the related label L2 and the related label L3 is essentially impossible.

[0041] Therefore, the attribute determining unit 24 and the label setting unit 25 perform the following processing to eliminate impossible combinations of related labels from among two or more related labels associated with the target image D2.

[0042] (2.3.4) Attribute Discrimination Unit The attribute discrimination unit 24 assigns the related labels L1-L4 associated with the target image D2 to one of a plurality of attributes. In particular, the attribute discrimination unit 24 preferably assigns the related labels L1-L4 associated with the target image D2 to one of a plurality of attributes based on attribute information TB1 (see FIG. 6 ) that associates a plurality of labels with one of a plurality of attributes. In this case, the image classification system 1 can achieve label assignment by the attribute discrimination unit 24 with a simple configuration.

[0043] Specifically, the image processing device 2 stores attribute information TB1 shown in FIG. 6 in advance in the storage unit of the image processing device 2. The attribute information TB1 is a list indicating labels belonging to each attribute of "phase," "voltage," and "installation type." The attribute determination unit 24 can determine to which attribute a related label belongs by referring to the attribute information TB1. In the attribute information TB1, the labels belonging to the attribute "phase" are "single-phase" and "three-phase," the labels belonging to the attribute "voltage" are "100V," "200V," and "250V," and the labels belonging to the attribute "installation type" are "wall-mounted," "floor-mounted," and "ceiling-mounted."

[0044] 7, the attribute discrimination unit 24 assigns the related label L1 "single phase" to the attribute "phase" based on the attribute information TB1. The attribute discrimination unit 24 assigns the related label L2 "100V" and the related label L3 "200V" to the attribute "voltage". The attribute discrimination unit 24 assigns the related label L4 "ceiling-mounted" to the attribute "installation type".

[0045] (2.3.5) Label Setting Unit The label setting unit 25 sets one related label to an attribute that has been assigned one related label among a plurality of attributes. The label setting unit 25 sets one related label selected from at least two related labels based on predetermined selection conditions to an attribute that has been assigned at least two related labels among a plurality of attributes.

[0046] 7, the attribute discrimination unit 24 assigns one related label L1 "single-phase" to the attribute "phase" and one related label L4 "ceiling-mounted" to the attribute "installation type." Therefore, the label setting unit 25 sets the related label L1 "single-phase" to the attribute "phase" and the related label L4 "ceiling-mounted" to the attribute "installation type."

[0047] Meanwhile, the attribute discrimination unit 24 assigns two related labels to the attribute "voltage," namely, related label L2 "100V" and related label L3 "200V." Therefore, the label setting unit 25 selects one of the related label L2 "100V" and related label L3 "200V" based on predetermined selection conditions, and sets the selected related label to the attribute "voltage."

[0048] In this embodiment, the selection condition is to select the related label with the highest score from at least two related labels assigned to the same attribute. The scores are the scores of the related label L2 "100V" and the related label L3 "200V" obtained by the classifier 23 (learning model M23) using the target image D2 as a classification target. In this embodiment, the score of the related label L2 "100V" is higher than the score of the related label L3 "200V". Therefore, the label setting unit 25 selects the related label L2 "100V" from the related label L2 "100V" and the related label L3 "200V" and sets the related label L2 "100V" to the attribute "voltage". By using the above selection condition, the label setting unit 25 can accurately determine an appropriate combination of multiple related labels.

[0049] That is, as shown in FIG. 8, the label setting unit 25 sets the related label L1 "single phase" for the attribute "phase," the related label L2 "100V" for the attribute "voltage," and the related label L3 "ceiling-mounted" for the attribute "installation type" as related labels to be associated with the target image D2 containing the symbol B3.

[0050] Therefore, the combination of the related label L2 and the related label L3 that belong to the same attribute "voltage" is resolved in the setting result of the label setting unit 25. In other words, when performing multi-label classification, the image classification system 1 can prevent the combination of two or more labels associated with the target image D2 from becoming an impossible combination.

[0051] (2.3.6) Result Output Unit The result output unit 26 outputs screen data including the setting results of the label setting unit 25 to the output device 4 as the classification result of the target image D2. The output device 4 displays the classification result of the target image D2 on a display device to present the classification result to the user.

[0052] (3) Image Classification Method The image classification method by the above-described image classification system 1 can be summarized as shown in the flowchart of Fig. 9. The image classification method may be realized by a computer system executing a program.

[0053] The image classification method includes an image acquisition step S1, an extraction step S2, a classification step S3, an attribute determination step S4, a label setting step S5, and a result output step S6.

[0054] In the image acquisition step S1, the image acquisition unit 21 acquires data of the original image D1 (see FIG. 2) from the database 5 via the network NT.

[0055] In the extraction step S2, the extraction unit 22 extracts a target image D2 (see FIG. 4) from the original image D1.

[0056] In the classification step S3, the classifier 23 associates two or more labels from the plurality of labels with the target image D2 as related labels.

[0057] In the attribute determination step S4, the attribute determination unit 24 assigns two or more related labels associated with the target image D2 to one of a plurality of attributes.

[0058] In label setting step S5, the label setting unit 25 sets one related label to an attribute that has been assigned one related label among the plurality of attributes, and sets one related label selected from the at least two related labels based on predetermined selection conditions to an attribute that has been assigned at least two related labels among the plurality of attributes.

[0059] In the result output step S6, the result output unit 26 outputs screen data including the setting result of the label setting unit 25 to the output device 4 as the classification result of the target image D2.

[0060] The image classification method described above can prevent the combination of two or more labels associated with the target image D2 from becoming an inherently impossible combination when performing multi-label classification.

[0061] (4) First Modification The label setting unit 25 of the first modification uses the following selection conditions when selecting one related label from at least two related labels assigned to the same attribute: From at least two related labels assigned to the same attribute, select a related label that allows for a combination of related labels for multiple attributes.

[0062] By using the above selection conditions, the image classification system 1 can accurately determine an appropriate combination of multiple related labels.

[0063] Specifically, the image processing device 2 stores combination information TB2 shown in FIG. 10 in advance in the storage unit of the image processing device 2. The combination information TB2 is a list showing possible combinations among all combinations of labels for each of a plurality of attributes. The attribute determination unit 24 can determine possible label combinations by referring to the combination information TB2. Possible label combinations are combinations that are actually possible for the outlet symbol B among all combinations of labels for each of a plurality of attributes. The combination information TB2 lists combinations Y in which one label is assigned to each of the attributes "phase," "voltage," and "installation type." Combination Y is a combination that is actually possible and is a possible combination.

[0064] For example, as shown in Fig. 7, the attribute discrimination unit 24 assigns the related label L1 "single phase" to the attribute "phase." The attribute discrimination unit 24 assigns the related label L2 "100V" and the related label L3 "200V" to the attribute "voltage." The attribute discrimination unit 24 assigns the related label L4 "ceiling-mounted" to the attribute "installation type."

[0065] In this case, the label setting unit 25 sets the related label L1 "single-phase" to the attribute "phase" and the related label L4 "ceiling-mounted" to the attribute "installation type." However, the attribute determination unit 24 assigns two related labels to the attribute "voltage," the related label L2 "100V" and the related label L3 "200V." Therefore, based on the above selection conditions, the label setting unit 25 selects a related label that allows a combination of related labels for multiple attributes from the related label L2 "100V" and the related label L3 "200V" assigned to the same attribute "voltage."

[0066] Specifically, in the combination information TB2, of the labels "100V," "200V," and "250V" that belong to the attribute "voltage," only "100V" can be combined with the related label L1 "single-phase" and the related label L4 "ceiling-mounted" (see combination Y1 in FIG. 10 ). Therefore, the label setting unit 25 refers to the combination information TB2, selects the related label L2 "100V" from the related label L2 "100V" and the related label L3 "200V" that are assigned to the same attribute "voltage," and sets the related label of the attribute "voltage" to the related label L2 "100V."

[0067] Therefore, the combination of the related label L2 and the related label L3 that belong to the same attribute "voltage" is resolved in the setting result of the label setting unit 25. In other words, when performing multi-label classification, the image classification system 1 can prevent the combination of two or more labels associated with the target image D2 from becoming an impossible combination.

[0068] (5) Second Modification An image classification system 1 of a second modification includes an image processing device 2A shown in Fig. 11. The image processing device 2A further includes a character recognition unit 27. Moreover, the image processing device 2A includes a result output unit 26A instead of the result output unit 26.

[0069] The character recognition unit 27 extracts and recognizes the character C (see FIG. 2) added near the symbol B included in the original image D1, and associates information corresponding to the recognized character with the symbol B. The character recognition unit 27 preferably includes a learning model M27 using a neural network. The learning model M27 uses a large number of images of architectural drawings including the character C as third teacher images, and constructs a neural network for extracting and recognizing the character C from the original image D1 by learning using these third teacher images. When the data of the original image D1 acquired by the image acquisition unit 21 is input, the learning model M27 extracts the image of the character C included in the original image D1 and recognizes the character C.

[0070] Specifically, the character recognition unit 27 extracts images of characters C1-C3 shown in Fig. 12 from the original image D1 and recognizes the characters C1-C3. The character C1 is "1EET," which indicates a "single-port outlet with a grounding electrode and a grounding terminal." The character C2 is "2EWP," which indicates a "two-port waterproof outlet with a grounding electrode." The character C3 is "2," which indicates a "two-port outlet."

[0071] The result output unit 26A then creates the aggregated data G1 shown in Fig. 13 based on the information on the label set in the target image D2 by the label setting unit 25 and the information on the character C recognized by the character recognition unit 27. Specifically, the result output unit 26A determines the type, quantity, etc. of the outlets based on the label corresponding to the outlet symbol B included in the original image D1 and the information on the character C attached to the symbol B. The result output unit 26A then creates the aggregated data G1 including the determination results of the type, quantity, etc. of the outlets. The aggregated data G1 is used, for example, for estimation work.

[0072] In the second modification, the aggregate data G1 including the determination results of the type and quantity of the outlets is automatically created from the original image D1, thereby improving the efficiency of the estimation work and the like.

[0073] (6) Third Modification The learning model M23 of the classifier 23 may use, as the neural network, a plurality of neural networks that perform single-label classification and are trained using a teacher image in which a label can be associated with one of a plurality of attributes.

[0074] That is, the second teacher image is associated with one label corresponding to one of the multiple attributes, and a neural network trained using such a second teacher image can perform single-label classification in which one label is associated with the target image D2. Therefore, by constructing a neural network that performs single-label classification for each attribute and using multiple neural networks constructed for each attribute, it is possible to perform multi-label classification in which two or more labels are associated with the target image D2.

[0075] For example, the learning model M23 uses a neural network that performs single-label classification of labels belonging to the attribute "phase," a neural network that performs single-label classification of labels belonging to the attribute "voltage," and a neural network that performs single-label classification of labels belonging to the attribute "installation form."

[0076] In this case, by using a neural network that performs single-label classification, which is easy to construct, it becomes relatively easy to construct the classifier 23 that performs multi-label classification.

[0077] (7) Fourth Modification At least one of the learning models M22, M23, and M27 is preferably constructed by deep learning using a convolutional neural network (CNN) or a fully convolutional network (FCN). However, the algorithm executed by the learning models M22, M23, and M27 is not limited to a specific algorithm. The learning model may be a model using another algorithm, such as a support vector machine. Furthermore, at least one of the learning models M22, M23, and M27 may be a model generated by a method other than machine learning (e.g., a linear prediction model based on statistical data).

[0078] Furthermore, at least one of the extraction unit 22, the classifier 23, and the character recognition unit 27 is not limited to a configuration that uses a learning model, and may be a configuration that uses pattern recognition processing, for example.

[0079] The target image D2 is not limited to the image of the outlet symbol B included in the original image D1. The target image D2 may be, for example, an image of a symbol for a switch, a lighting fixture, a communication facility, an air conditioning facility, a drainage facility, or the like included in the original image D1.

[0080] The original image D1 is not limited to an architectural drawing, but may be a drawing of an electric circuit, a mechanical drawing, or the like.

[0081] The targets of multi-label classification performed by the image classification system 1 are not limited to symbols contained in drawings. The targets of multi-label classification may also be people, mountains, structures, etc. (target images) contained in photographs (original images). Furthermore, the targets of multi-label classification may also be people, cars, signs, traffic lights, etc. (target images) contained in videos (original images).

[0082] The functions of the image classification system 1 may be embodied in an image classification method, a computer program, or a recording medium on which a computer program is recorded.

[0083] (8) Summary An image classification system (1) according to a first aspect of the present embodiment classifies a target image (D2) through image processing. The image classification system (1) includes a classifier (23), an attribute discrimination unit (24), and a label setting unit (25). The classifier (23) associates two or more labels from a plurality of labels with the target image (D2) as related labels (L1-L4). The attribute discrimination unit (24) assigns the two or more related labels (L1-L4) to one of a plurality of attributes. The label setting unit (25) assigns one related label (L1 or L4) to an attribute that has been assigned one related label (L1-L4) among the plurality of attributes, and assigns one related label (L2) selected from the at least two related labels (L2, L3) based on predetermined selection conditions to an attribute that has been assigned at least two related labels (L2, L3) among the plurality of attributes.

[0084] The image classification system (1) described above can prevent combinations of two or more labels associated with an image from becoming combinations that are essentially impossible when performing multi-label classification.

[0085] In the image classification system (1) of the second aspect according to the embodiment, in the first aspect, the classifier (23) includes a learning model (M23) using a neural network.

[0086] The image classification system (1) described above can easily realize a classifier (23) that performs multi-label classification.

[0087] In the image classification system (1) of the third aspect of the embodiment, in the second aspect, it is preferable that the learning model (M23) uses, as the neural network, one neural network that performs multi-label classification trained using teacher images in which a label can be associated with each of a plurality of attributes.

[0088] The above-mentioned image classification system (1) can realize a classifier (23) that performs multi-label classification using a single neural network.

[0089] In the image classification system (1) of the fourth aspect of the embodiment, in the second aspect, the learning model (M23) preferably uses, as neural networks, a plurality of neural networks that perform single-label classification and are trained using teacher images in which a label can be associated with one of a plurality of attributes.

[0090] The image classification system (1) described above uses a neural network that performs single-label classification, which is easy to construct, making it relatively easy to construct a classifier (23) that performs multi-label classification.

[0091] In the image classification system (1) of the fifth aspect of the embodiment, in any one of the first to fourth aspects, it is preferable that the attribute discrimination unit (24) assigns at least two related labels (L1-L4) to any one of a plurality of attributes based on attribute information that associates the plurality of labels with any one of a plurality of attributes.

[0092] The image classification system (1) described above can realize label assignment by the attribute discrimination unit (24) with a simple configuration.

[0093] In the image classification system (1) of a sixth aspect according to any one of the first to fifth aspects, the classifier (23) calculates scores for each of a plurality of labels for the target image (D2) and extracts two or more related labels (L1-L4) from the plurality of labels based on the scores. Preferably, the selection condition is to select the related label (L2) with the highest score from at least two related labels (L2, L3).

[0094] The image classification system (1) described above can accurately determine an appropriate combination of multiple related labels.

[0095] In the seventh aspect of the image classification system (1) according to the embodiment, in any one of the first to fifth aspects, it is preferable that the selection condition is to select, from among at least two related labels (L2, L3), a related label (L2) that allows for a combination of related labels for each of multiple attributes.

[0096] The image classification system (1) described above can accurately determine an appropriate combination of multiple related labels.

[0097] It is preferable that the image classification system (1) of the eighth aspect of the embodiment, in any one of the first to seventh aspects, further comprises an extraction unit (22) that extracts a target image (D2) from the original image (D1).

[0098] The image classification system (1) described above can classify all target images (D2) included in the original image (D1).

[0099] An image classification method according to a ninth aspect of the present embodiment classifies a target image (D2) by image processing. The image classification method includes a classification step (S3), an attribute determination step (S4), and a label setting step (S5). The classification step (S3) associates two or more labels from a plurality of labels with the target image (D2) as related labels (L1-L4). The attribute determination step (S4) assigns the two or more related labels (L1-L4) to one of a plurality of attributes. The label setting step (S5) assigns one related label (L1 or L4) to an attribute that has been assigned one of the related labels (L1-L4) among the plurality of attributes. The label setting step (S5) assigns one related label (L2) selected from at least two related labels (L2, L3) based on predetermined selection conditions to an attribute that has been assigned at least two related labels (L1-L4) among the plurality of attributes.

[0100] The above-described image classification method can prevent the combination of two or more labels associated with an image from becoming an impossible combination when performing multi-label classification.

[0101] A program according to a tenth aspect of the present embodiment causes a computer system to execute the image classification method according to the ninth aspect.

[0102] When performing multi-label classification, the above-described program can prevent combinations of two or more labels associated with an image from becoming combinations that are inherently impossible.

[0103] 1 Image classification system 22 Extraction unit 23 Classifier 24 Attribute discrimination unit 25 Label setting unit M23 Learning model D1 Original image D2 Target image L1-L4 Related labels S3 Classification step S4 Attribute discrimination step S5 Label setting step

Claims

1. An image classification system that classifies target images through image processing, comprising: a classifier that associates two or more labels from a plurality of labels with the target image as related labels; an attribute discrimination unit that assigns the two or more related labels to one of a plurality of attributes; and a label setting unit that sets one related label for an attribute of the plurality of attributes that has been assigned one of the related labels, and sets one related label selected from at least two related labels based on predetermined selection conditions for an attribute of the plurality of attributes that has been assigned at least two of the related labels.

2. The image classification system of claim 1, wherein the classifier includes a learning model using a neural network.

3. The image classification system of claim 2, wherein the learning model uses, as the neural network, one neural network that performs multi-label classification trained using training images in which a label can be associated with each of the multiple attributes.

4. The image classification system of claim 2, wherein the learning model uses, as the neural network, a plurality of neural networks that perform single-label classification trained using training images in which a label can be associated with one of the plurality of attributes.

5. An image classification system according to any one of claims 1 to 4, wherein the attribute discrimination unit assigns the two or more related labels to one of the plurality of attributes based on attribute information that associates the plurality of labels with one of the plurality of attributes.

6. The image classification system according to any one of claims 1 to 5, wherein the classifier calculates a score for each of the plurality of labels for the target image, and extracts the two or more related labels from the plurality of labels based on the score, and the selection condition is to select the related label with the highest score from the at least two related labels.

7. The image classification system according to any one of claims 1 to 5, wherein the selection condition is to select, from the at least two related labels, related labels that enable combinations of the related labels for each of the multiple attributes.

8. The image classification system according to any one of claims 1 to 7, further comprising an extraction unit that extracts the target image from the original image.

9. An image classification method for classifying a target image by image processing, comprising: a classification step for associating two or more labels from a plurality of labels with the target image as related labels; an attribute discrimination step for assigning the two or more related labels to any of a plurality of attributes; and a label setting step for setting one related label for an attribute from the plurality of attributes that has been assigned one of the related labels, and for setting one related label selected from at least two related labels based on predetermined selection conditions for an attribute from the plurality of attributes that has been assigned at least two of the related labels.

10. A program for causing a computer system to execute the image classification method according to claim 9.

Images