Component search device, computer-readable recording medium, and component search method

Abstract

A component search device (10) includes a component image recognition section (11) that recognizes an image of a component from an image of a control panel, an element recognition section (12) that recognizes a plurality of elements that constitute the component from the image of the component, a category determination section (13) that determines a category of the component in which the plurality of elements have been recognized, based on the plurality of elements that have been recognized, using negative element definition information (72) in which a plurality of negative elements that represent a plurality of elements that cannot constitute the component are defined in units of component categories, and a search section (14) that searches for a component from a component DB (20) in which a plurality of components used for generation of a graphical control screen are stored, based on the plurality of elements that have been recognized and the category that has been determined.

Description

TECHNICAL FIELD

[0001] The present application relates to a component search device, a component search program, and a component search method for searching for components used for creation of a graphical control screen from an image of a control panel. BACKGROUND

[0002] In the related art of creating a graphical control screen from an image of a control panel, an image of a component possessed by the control panel is extracted from an image obtained by photographing the control panel. Then, by performing pattern recognition based on image information of a component registered in advance with respect to the extracted image of the component, the category and the type of the component are determined. After the category and the type of the component are determined, a component corresponding to the determined category and type is selected from among components registered for the graphical control screen (for example, Patent Literature 1).

[0003] In Patent Literature 2, a technique of determining an electronic component or a mechanical component within a circuit diagram by mapping with an expression of an electronic component or a mechanical component stored in a database is disclosed. The following (1) to (4) are disclosed in Patent Literature 2.

[0004] (1) An element is defined as an object or a part of an object. An element database contains a type of the element, an element name, a visual attribute of the element, and a description attribute. As the visual attribute, a shape, a size, a color, and the like can be contained.

[0005] (2) An element is recognized from a circuit diagram based on a group of attributes of a common visual attribute. Characteristic information (size, shape, color, and the like) of the recognized element is extracted. The extracted characteristic information is compared with a description in the element database, and a consistent element is determined.

[0006] (3) Although it is generally preferable that all the visual attributes are consistent, a spatial distance of several features among the visual attributes of a shape, an orientation, a color, a line type, and the like can be used for consistency determination.

[0007] (4) In addition to pattern matching of a shape, a visual attribute is used, whereby further distinction of elements is possible.

[0008] Patent Literature 1: Japanese Patent Application Publication No. 2012-160134

[0009] Patent Literature 2: U.S. Patent Application Publication No. 2015 / 0161301 SUMMARY

[0010] In Patent Literature 1, a category and a type of a component having a similar appearance are determined based on pattern recognition of images with each other. Therefore, there is a problem that, for example, in the case of components having similar shapes such as a rotary switch and an analog meter, an analog meter having similar image feature amounts is erroneously searched for a rotary switch image included in an image of a control panel in pattern recognition using only images.

[0011] In addition, the similar component search method using characteristic information such as size, shape, color, and the like in Patent Literature 2 can be applied to similar component search for components whose shapes are standardized. However, there is a problem that Patent Literature 2 is difficult to apply to components whose shapes are not standardized.

[0012] An object of the present application is to provide a component search device which does not erroneously search for components having similar shapes even in the case of components having similar shapes such as an analog meter with respect to a rotary switch.

[0013] In addition, an object of the present application is to provide a device to which similar component search can be applied to components whose shapes are not standardized.

[0014] The component search device according to the present application has:

[0015] a component image recognition section which recognizes an image of a component from an image of a control panel;

[0016] an element recognition section which recognizes a plurality of elements constituting the component from the image of the component;

[0017] a category determination section which determines a category of the component of which the plurality of elements have been recognized, based on the plurality of recognized elements and negative element definition information in which a plurality of negative elements indicating a plurality of elements which cannot constitute a component are defined in units of component categories; and

[0018] a search section which searches for a component from a database in which a plurality of components used for generation of a graphical control screen are stored, based on the plurality of recognized elements and the determined category.

[0019] Effects of the Invention

[0020] The component search device according to the present application has a category determination section, and thus can provide a component search device which does not erroneously search for components having similar shapes even in the case of components having similar shapes such as an analog meter with respect to a rotary switch.

[0021] The component search device according to the present application can provide a device to which similar component search can be applied to components whose shapes are not standardized. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 FIG. 1 is a diagram of Embodiment 1, which is a configuration diagram of the component search device 10.

[0023] Figure 2 FIG. 2 is a diagram of Embodiment 1, which is a flowchart showing the operation of the component search device 10.

[0024] Figure 3 FIG. 3 is a diagram of Embodiment 1, which is a hardware configuration diagram of the component search device 10.

[0025] Figure 4 FIG. 4 is a diagram of Embodiment 1, which is a diagram showing the element definition information 51.

[0026] Figure 5 FIG. 5 is a diagram of Embodiment 1, which is a diagram showing the image 40 of the control panel.

[0027] Figure 6 FIG. 6 is a diagram of Embodiment 1, which is a diagram showing the recognition of the elements in the image 40b of the component.

[0028] Figure 7 FIG. 7 is a diagram of Embodiment 1, which is a diagram showing the recognition of the elements in the image 40d of the component.

[0029] Figure 8 FIG. 9 is a diagram of Embodiment 2, which is a configuration diagram of the component search device 10.

[0030] Figure 9 FIG. 10 is a diagram of Embodiment 2, which is a flowchart showing the operation of the component search device 10.

[0031] Figure 10 FIG. 11 is a diagram of Embodiment 2, which is a diagram showing the attribute definition information 52.

[0032] Figure 11 FIG. 12 is a diagram of Embodiment 2, which is a diagram showing the recognition of the attribute features of the elements of the image 40b of the component.

[0033] Figure 12 FIG. 13 is a diagram of Embodiment 2, which is a diagram showing the recognition of the attribute features of the elements of the image 40d of the component.

[0034] Figure 13 FIG. 15 is a diagram of Embodiment 3, which is a configuration diagram of the component search device 10.

[0035] Figure 14 FIG. 16 is a diagram of Embodiment 3, which is a flowchart showing the operation of the component search device 10.

[0036] Figure 15 FIG. 17 is a diagram of Embodiment 3, which is a diagram showing the image 40-1 of the control panel.

[0037] Figure 16 FIG. 40-2 is a diagram that represents an image 40-2 of the control panel taken at a different timing from the image 40-1, according to Embodiment 3.

[0038] Figure 17 FIG. 40-3 is a diagram that represents dynamic attribute information identified from the images 40-1, 40-2, according to Embodiment 3.

[0039] Figure 18 FIG. 40-4 is another diagram that represents dynamic attribute information identified from the images 40-1, 40-2, according to Embodiment 3.

[0040] Figure 19 FIG. 41 is a diagram that represents error search example 1 of the component, according to Embodiment 4.

[0041] Figure 20 FIG. 42 is a diagram that represents error search example 2 of the component, according to Embodiment 4.

[0042] Figure 21 FIG. 43 is a diagram that represents error search example 3 of the component, according to Embodiment 4.

[0043] Figure 22 FIG. 44 is a diagram that represents a configuration of the component search device 10, according to Embodiment 4.

[0044] Figure 23 FIG. 45 is a diagram that represents a hardware configuration of the component search device 10, according to Embodiment 4.

[0045] Figure 24 FIG. 46 is a flowchart of an operation of the component search device 10, according to Embodiment 4.

[0046] Figure 25 FIG. 47 is a flowchart that represents details of the step 63 of Figure 24 , according to Embodiment 4.

[0047] Figure 26 FIG. 48 is a diagram that represents the element definition information 51, according to Embodiment 4.

[0048] Figure 27 FIG. 49 is a diagram that represents the negative element definition information 72, according to Embodiment 4. DETAILED DESCRIPTION

[0049] In the description of the embodiments and the drawings, the same reference numerals are assigned to the same elements and corresponding elements. The description of the elements assigned with the same reference numerals is appropriately omitted or simplified. In the following embodiments, “unit” can be appropriately replaced with “circuit”, “step”, “flow”, “process”, “processing circuit”, or “circuitry”.

[0050] In the following embodiments, a database is denoted as DB.

[0051] Embodiment 1

[0052] Reference Figures 1 to 7 The component search device 10 of Embodiment 1 is described. Figure 1 is a block diagram of the component search device 10. Figure 2 is a flowchart showing the operation of the component search device 10. Figure 3 is a hardware configuration diagram of the component search device. Figure 4 is the element definition information 51. The element definition information 51 will be described later. Figure 5 shows the image 40 of the control panel. Figure 6 shows the recognition of the elements in the image 40b of the component. Figure 7 shows the recognition of the elements in the image 40d of the component.

[0053] ***Explanation of the Configuration***

[0054] In Figure 1 , the component search device 10 has a component image recognition section 11, an element recognition section 12, a definition information DB 15, a category determination section 13, a search section 14, and a component DB 20. In addition, an image input section 1 is connected to the component image recognition section 11. The image input section 1 inputs the image of the control panel to the component image recognition section 11.

[0055] (1) The component image recognition section 11 recognizes the image of the component from the image of the control panel input by the image input section 1.

[0056] (2) The element recognition section 12 recognizes the structural elements of the component from the image of the component recognized by the component image recognition section 11. Hereinafter, the structural elements of the component are referred to as elements.

[0057] Further, "the element recognition section 12 recognizes the elements" means "the element recognition section 12 speculates the categories of the elements". In Embodiment 1, the element recognition section 12 speculates the categories of the elements, and the category determination section 13 determines the category of the component based on the result of the speculation of the categories of the elements by the element recognition section 12 and the element definition information 51.

[0058] On the contrary, in Embodiment 2 described later, the element determination section 17 determines the categories of the elements using the attributes. "The element determination section 17 determines the categories of the elements" means "the element determination section 17 adjudicates the categories of the elements".

[0059] (3) The definition information DB 15 stores the element definition information 51 which defines the elements of the component in units of the categories of the components. The element definition information 51 will be described later. Figure 4To be described later.

[0060] (4) The category determination unit 13 determines the category of the component identified by the component image recognition unit 11 based on the element and definition information 51 stored in the element definition information DB 15 identified by the element recognition unit 12. That is, the category determination unit 13 determines the category of the component identified by the component image recognition unit 11 based on the element category inferred by the element recognition unit 12 and the element definition information 51 stored in the element definition information DB 15.

[0061] (5) The retrieval unit 14 retrieves similar components from the component DB 20 based on the elements identified by the element identification unit 12 and the component categories determined by the category determination unit 13. Furthermore, DB refers to a database.

[0062] (6) Component DB 20 stores multiple components used to generate the graphical control screen.

[0063] Component retrieval device 10 consists of Figure 3 The hardware configuration is shown. Processing by the image input unit 1 is performed via the image interface 911. Processing by the component image recognition unit 11 and the element recognition unit 12 is performed by the image recognition processor 912. Processing by the definition information DB 15 and the component DB 20 is performed by the storage device 920. Processing by the category determination unit 13 and the retrieval unit 14 is executed by the processor 901 reading the program stored in the memory 902.

[0064] Furthermore, processor 901 and image recognition processor 912 can also be implemented by a single processor. In this case, a component retrieval program that implements the functions of component image recognition unit 11, element recognition unit 12, category determination unit 13, and retrieval unit 14 is stored in storage device 920. A processor loads the component retrieval program from storage device 920 into memory 902 and executes it.

[0065] The component retrieval program can be provided stored on a computer-readable recording medium or as a program product. The component retrieval program is a program that causes a computer to execute processes, processes, or procedures that replace the "section" in the component image recognition unit 11, element recognition unit 12, category determination unit 13, and retrieval unit 14 with "process," "flow," or "operation." The operation flow of the component retrieval device 10 is equivalent to the component retrieval method. The program that implements the operation of the component retrieval device 10 is equivalent to the component retrieval program.

[0066] ***Instructions for Action***

[0067] Next, refer to Figure 2 The operation of the component retrieval device 10 will be explained. Figure 2The contents in the parentheses indicate the subject of the action of the step.

[0068] The element definition information 51 indicating the presence or absence of elements of components used in the control panel is defined in advance in units of categories of the components, and is stored in the definition information DB 15. As shown in Figure 4 The element definition information 51 associates an element category such as a pointer, a knob, a scale, and the like with a component category such as an analog meter, a rotary switch, and the like, respectively. In Figure 4 In the drawing, a circular mark indicates the presence of the element, and a blank indicates the absence of the element. In the case where the component category is an analog meter, there are a pointer, a scale, a numerical display, a center circle, and the like as categories of the elements. In the case where the component category is a rotary switch, there are a knob, a scale, a numerical display, a center circle as categories of the elements.

[0069] <Step S10>

[0070] When the components of the control panel are recognized, and the components used in the graphical control screen are searched, first, the image 40 of the control panel is input to the image input section 1. In Figure 5 The image 40 of the control panel is shown in the drawing. If the image input section 1 is input with the image 40, the image 40 is input to the component image recognition section 11.

[0071] <Step S11>

[0072] The component image recognition section 11 recognizes the images of the components from the image 40 of the control panel. Specifically, as described below, the component image recognition section 11 recognizes the images 40a, 40b, 40c, 40d, 40e, 40f, 40g, 40h, 40i of the components included in the image 40 from the image 40 of the control panel input by the image input section 1 using a method such as a contour extraction method.

[0073] <Step S12>

[0074] The element recognition section 12 recognizes a plurality of elements constituting the components from the images of the components. That is, the element recognition section 12 estimates the categories of the elements constituting the components from the images of the components.

[0075] Specifically, as described below, the element recognition section 12 recognizes the elements constituting the components from the nine images of the images 40a to 40i of the components recognized by the component image recognition section 11. For the recognition of the elements, a method such as a feature amount extraction method is used.

[0076] Figure 6 The recognition of the elements constituting the rotary switch from the image 40b of the rotary switch as the component is shown. The element recognition section 12 recognizes a total of four elements of the knob 41b, the scale 42b, the numerical display 43b, and the center circle 44b.Figure 7 The recognition of the elements constituting the analog meter from the image 40d of the analog meter as a component is shown. The element recognition section 12 recognized the pointer 41d, the scale 42d, the numerical display 43d, and the center circle 44d, a total of four elements.

[0077] Further, in Figure 6 and Figure 7 , the category of the element is described as, for example, a knob, a memory, and the like. This is the result of the element recognition section 12 inferring the category of the element as an element. As described above, the "recognition" of the element recognition section 12 in Embodiment 1 corresponds to inferring the category of the element. In Embodiment 2, the category of the element is determined using the attribute.

[0078] <Step S13>

[0079] The category determination section 13 determines the category of the component from which the plurality of elements have been recognized, based on the plurality of elements that have been recognized and the element definition information 51 that defines the plurality of elements constituting the component in units of component categories. This is described below. The category determination section 13 compares the element definition information 51 stored in the definition information DB 15 with the elements recognized by the element recognition section 12, and determines the component category. For example, Figure 6 the case of the rotary switch is described below. In Figure 4 the element definition information 51 and Figure 6 the recognized elements shown in FIG. 4B, in Figure 6 , the element recognition section 12 recognized the knob 41b, the scale 42b, the numerical display 43b, and the center circle 44b as a result of inferring the category of the element. On the other hand, in the element definition information 51, the component category of the element having the knob, the scale, the numerical display, and the center circle is the rotary switch. Therefore, the category determination section 13 determined that the component category of the image 40b is the rotary switch, based on the recognition result of the elements by the element recognition section 12 and the element definition information 51.

[0080] Figure 7 the case of the analog meter is described below. In Figure 4 the element definition information 51 and Figure 7 the recognized elements shown in FIG. 4D, in Figure 7 , the element recognition section 12 recognized the pointer 41d, the scale 42d, the numerical display 43d, and the center circle 44d as a result of inferring the category of the element. On the other hand, in the element definition information 51, the component category of the element having the pointer, the scale, the numerical display, and the center circle is the analog meter. Therefore, the category determination section 13 determined that the component category of the image 40d is the analog meter, based on the recognition result of the elements by the element recognition section 12 and the element definition information 51.

[0081] <Step S14>

[0082] The search section 14 searches for a component from a database in which a plurality of components used for generating a graphical control screen are stored, based on the plurality of elements recognized and the category determined. This is described below. The search section 14 searches for a similar component for a graphical control screen from the component DB 20 in which component images used for a graphical control screen are stored, based on the elements recognized by the element recognition section 12 and the component category determined by the category determination section 13.

[0083] Explanation of Effects of Embodiment 1

[0084] As described above, the component search device 10 recognizes elements of a component from an image of the component recognized based on the image 40 of the control panel. The component search device 10 determines a category of the component by comparing the presence or absence of the recognized elements with the element definition information 51. The component search device 10 searches for a component for a graphical control screen based on the determined component category and the recognized elements. Thus, according to the component search device 10, it is possible to prevent the existing problem of erroneously searching for a component for a graphical control screen that is similar in image feature amount to an image of a component photographed in the image 40 of the control panel but differs in component category.

[0085] Embodiment 2

[0086] Reference Figures 8 to 12 The component search device 10 of Embodiment 2 is described. The component search device 10 of Embodiment 2 further has an element attribute recognition section 16 and an element determination section 17, relative to the component search device 10 of Embodiment 1. Figure 8 is a block diagram showing the configuration of the component search device 10. Figure 9 is a flowchart showing the operation of the component search device 10. Figure 10 Attribute definition information 52 is shown. As shown in Figure 10 , the attribute definition information 52 defines the characteristics of the attributes of the elements of the component in units of the elements of the component. In the attribute definition information 52, a pointer, a knob, a scale, a numerical value display, and the like are defined as elements. In addition, a shape, a size, the presence or absence of a numerical value, and the like are defined as attributes in units of the elements of the component. For example, in terms of the shape, characteristics such as a triangle, a "rectangle or ellipse", and the like are defined as the characteristics of the attributes. Figure 11 Recognition of an element of a rotary switch 40b of the image 40 of the control panel is shown. Figure 12 Recognition of an element of an analog meter 40d of the image 40 of the control panel is shown. In Embodiment 2, if each element is a knob or a pointer, determination of the category of the element is performed by the element determination section 17. In Figure 11 , Figure 12In the present embodiment, it is not clear what kind of elements the images 41b, 41d are.

[0087] The element attribute identifying section 16 identifies the features of the attributes of the elements identified by the element identifying section 12. The element determining section 17 determines the categories of the elements based on the features of the attributes of the elements identified by the element attribute identifying section 16, the attribute definition information 52 stored in the definition information DB 15.

[0088] Next, the operation of the component search device 10 is described using the flowchart of FIG. 8. The bracketed contents of each step of the flowchart indicate the operation subject of the step. The element definition information 51 of FIG. 7 and the attribute definition information 52 of FIG. 8 are stored in the definition information DB 15 in advance. Figure 9 Figure 4 Figure 10

[0089] In Embodiment 2, the operation from the step S10 of the registration processing of the element definition information 51 and the processing of inputting the image 40 of the control panel to the step S12 of the processing of identifying the elements from the image of the component is the same as that of Embodiment 1, and thus the description is omitted.

[0090] <Step S41>

[0091] The element attribute identifying section 16 identifies the features of the attributes of each of the plurality of elements identified. Specifically, as described below, the element attribute identifying section 16 identifies the features of the attributes from the elements identified by the element identifying section 12. For example, in the case of the element 41b identified from the image 40b of the rotary switch, the element attribute identifying section 16 identifies each of the features of the shape, the size, and the numerical value as the attributes, such as the shape being a rectangular shape, the size being short and not reaching the scale, and the numerical value being absent. Figure 11 Figure 12 In the case of the element 41d identified from the image 40d of the analog meter, the element attribute identifying section 16 identifies each of the features of the shape, the size, and the numerical value as the attributes, such as the shape being a triangular shape, the size being long and reaching the scale, and the numerical value being absent.

[0092] <Step S42>

[0093] The element determining section 17 determines the plurality of elements identified based on the features of the attributes identified for each of the elements, the attribute definition information 52 in which the features of the attributes are defined in units of elements. Specifically, as described below, the element determining section 17 compares the attribute definition information 52 stored in the definition information DB 15 with the features of the attributes of the elements identified by the element attribute identifying section 16, and determines the elements. In the case of the element 41b identified from the image 40b of the rotary switch, the element determining section 17 determines the element 41b as a rotary switch based on the features of the attributes identified by the element attribute identifying section 16 and the attribute definition information 52 in which the features of the attributes are defined in units of elements. Figure 11 ​​​​In the case of the element recognized in the image 40b of the rotary switch shown, the features of each attribute coincide with the knobs of the attribute definition information 52. Therefore, the element determination section 17 determines that the element is a knob. Also, in the case of the element recognized in the image 40c of the analog meter shown, the features of each attribute coincide with the needle of the attribute definition information 52. Therefore, the element determination section 17 determines that the element is a needle. Figure 12 In the case of the element recognized in the image 40d of the analog meter shown, the features of each attribute coincide with the needle of the attribute definition information 52. Therefore, the element determination section 17 determines that the element is a needle.

[0094] <Step S43>

[0095] The category determination section 13 determines the category of the component based on the plurality of elements determined and the element definition information 51. Specifically, as described below. Similarly to step S13, the category determination section 13 compares the element definition information 51 stored in the definition information DB 15 with the presence or absence of the elements determined by the element determination section 17, and determines the category of the component.

[0096] <Step S14>

[0097] The search section 14 searches for a similar component for a graphical control screen from the component DB 20 storing components for a graphical control screen, based on the elements recognized by the element recognition section 12 and the category of the component determined by the category determination section 13.

[0098] Explanation of Effects of Embodiment 2

[0099] As described above, the component search device 10 infers the category of the element from the image of the component recognized from the image 40 of the control panel. In the component search device 10 of Embodiment 2, the element determination section 17 compares the features of the attributes of the element category inferred by the element recognition section 12 with the attribute definition information 52 to determine the category of the element. Therefore, it is possible to more reliably prevent erroneous recognition of the category of the element in the image of the component than in Embodiment 1.

[0100] Embodiment 3

[0101] Reference Figures 13 to 18 The component search device 10 of Embodiment 3 will be described. Figure 13 is a block diagram showing the configuration of the component search device 10. Figure 14 is a flowchart showing the operation of the component search device 10. Figure 15 , Figure 16 are images of a control panel taken at different timings. Figure 17 , Figure 18 shows dynamic attribute information 53 recognized from the images of the control panel taken at different timings.

[0102] Embodiment 3 relates to dynamic attribute information 53. This dynamic attribute information 53 is reflected in the attribute possessed by the "component used for generation of the graphical control screen" obtained as a result of the search of the component DB 20 made by the search section 14 of the component search device 10. That is, the dynamic attribute information 53 is reflected in the component of the component DB 20. The reflection of the dynamic attribute information 53 to the component can be implemented by the dynamic attribute recognition section 18 of the component search device 10, or can be implemented by another device different from the component search device 10. The "component used for generation of the graphical control screen" stored in the component DB 20 has not only the information of the pattern, but also, as the "dynamic attribute information 53", information such as the respective colors at the time of ON / OFF if it is a push button, or the range of action or the range of values if it is a component such as an analog meter. By reflecting the "dynamic attribute information 53" obtained in Embodiment 3 in the component of the component DB 20, it is possible to generate the graphical control screen using the component in which the dynamic attribute information 53 is reflected.

[0103] With respect to Figure 1 The component search device 10 of Embodiment 3 further has the dynamic attribute recognition section 18, in addition to the component search device 10 of Embodiment 1. In addition, the image input section 1 is inputted a plurality of images of the same control panel taken at different timings, and inputs these images to the component image recognition section 11. The dynamic attribute recognition section 18 recognizes the features of the attributes of the elements recognized by the element recognition section 12 from the images of the plurality of control panels taken at different timings, respectively, and stores the features as the dynamic attribute information 53 if there is a change.

[0104] Next, the operation of the component search device 10 will be described. The process of storing the element definition information 51 in the definition information DB 15, the process of determining the component class by comparing the element definition information 51 stored in the definition information DB 15 with the presence or absence of the element recognized by the element recognition section 12 (step S13), and the operation thereafter are the same as in Embodiment 1.

[0105] <Step S51>

[0106] The image input section 1 is inputted a plurality of images of the control panel taken at different timings, and inputs the plurality of images to the component image recognition section 11.

[0107] <Step S11>

[0108] The component image recognition section 11 can recognize the images of the same component from each of the images of the same control panel taken at different timings. This will be described specifically as follows. The component image recognition section 11 recognizes the images of the components from the images of each of the control panels inputted by the image input section 1 by a contour extraction method or the like.

[0109] <Step S12>

[0110] The element recognition section 12 recognizes a plurality of elements constituting the same component from the images of the same component recognized from the respective images of the control panel. This is described in detail below. The element recognition section 12 recognizes the elements from the respective images of the component recognized by the component image recognition section 11 by a feature amount extraction method or the like.

[0111] <Step S52>

[0112] The dynamic attribute recognition section 18 stores a change in the feature of the attribute in the case where the feature of the attribute changes among the same elements from among the plurality of elements recognized from the images of the same component recognized from the respective images of the control panel. This is described in detail below. The dynamic attribute recognition section 18 inputs the elements recognized by the element recognition section 12 from the element recognition section 12 in units of the components of the respective images of the control panel input to the image input section 1. The dynamic attribute recognition section 18 recognizes the feature of the attribute of each element. If the feature of the attribute changes in the same element of the same component, the dynamic attribute recognition section 18 judges this as dynamic attribute information 53 and stores the feature. For example, in the images 40-1, 40-2 of the control panel shown in Figs. 40-1, 40-2 taken at different timings, the feature of the color matching attribute changes at the switches 40e-1, 40e-2. The dynamic attribute recognition section 18 stores the color matching attribute as dynamic attribute information 53, and the display color at the time of ON, the display color at the time of OFF as dynamic attribute information of the color matching. Similarly, in the images 40-1, 40-2 of the control panel shown in Figs. 40-1, 40-2 taken at different timings, the feature of the display angle attribute changes at the analog meters 40d-1, 40d-2. The dynamic attribute recognition section 18 stores the display angle at the time of display of the minimum value and the display angle at the time of display of the maximum value as dynamic attribute information 53. Figure 15 Figure 16 Figure 17 Figure 15 Figure 16 Figure 18

[0113] ***Explanation of Effects of Embodiment 3***

[0114] ​​​​​​As described above, the component retrieval device 10 identifies the same elements of the same component as input using a plurality of images of the control panel captured at different timings. In the component retrieval device 10, if there is a change in the feature of the attribute identified from the same elements, the dynamic attribute identification section 18 judges it as the dynamic attribute information 53 and stores the dynamic attribute information 53 in the storage device 920. Thus, by reflecting the dynamic attribute information 53 stored by the dynamic attribute identification section 18 in the components of the component DB 20, it is possible to generate the graphical control screen using the components that reflect the dynamic attribute information 53.

[0115] Embodiment 4

[0116] Reference Figures 19 to 27 The component retrieval device 10 of Embodiment 4 will be described. The component retrieval device 10 of Embodiment 4 is characterized in that the category determination section 13 determines the component category using the negative structure element definition information 72.

[0117] Hereinafter, the negative structure element definition information 72 is written as the negative element definition information 72.

[0118] Figure 19 Error retrieval example 1 of components. In Figure 19 the following state is shown, that is, for the rotary switch image 91 in the control panel image, since the rotary switch and the button have similar characteristic information, the analog meter 92 is erroneously retrieved as a retrieval result of similar components. Here, the characteristic information refers to information of the size, shape, color, and the like of the identified elements as described in the background art.

[0119] Figure 20 Error retrieval example 2 of components. In Figure 20 the following state is shown, that is, for the rotary switch image 93 in the control panel image, since the rotary switch and the button have similar characteristic information, the button 94 is erroneously retrieved as a retrieval result of similar components.

[0120] Figure 21 Error retrieval example 3 of components. In Figure 21 the following state is shown, that is, for the buzzer image 95 in the control panel image, since the buzzer and the button have similar characteristic information, the button 96 is erroneously retrieved as a retrieval result of similar components.

[0121] Figure 22The structure of the component retrieval device 10 is shown. Compared to the component retrieval device 10 of Embodiment 1, the component retrieval device 10 of Embodiment 4 further includes a second DB, namely, negative element definition information DB 62, and a display device 930. The component retrieval device 10 includes a component image recognition unit 11, an element recognition unit 12, a category determination unit 13, a retrieval unit 14, definition information DB 15, a component DB 20, a display device 930, and negative element definition information DB 62.

[0122] Negative element definition information DB 62 is sometimes referred to as second DB 62. Image input unit 1 is connected to component image recognition unit 11. Image input unit 1 inputs the image of the control panel to component image recognition unit 11. Component DB 20 and display device 930, which displays the search results for similar components, are connected to search unit 14. Display device 930 displays the search results for components that are the same as or similar to components in the component category determined by category determination unit 13.

[0123] Figure 23 This is a hardware structure diagram showing the component retrieval device 10. (Compared to Embodiment 1) Figure 3 , Figure 23 The hardware structure diagram also includes a display device 930. The component retrieval device 10 consists of... Figure 23 The hardware configuration is shown. The processing performed by the image input unit 2 is performed via the image interface 911. The processing of the component image recognition unit 11 and the element recognition unit 12 is performed by the image recognition processor 912. The processing of definition information DB 15, negative element definition information DB 62, and component DB 20 is performed by the storage device 920. The processing of the category determination unit 13 and the retrieval unit 14 is performed by the processor 901 reading the program stored in the memory 902.

[0124] Figure 24 This is a flowchart of the operation of the component retrieval device 10.

[0125] Figure 25 It means Figure 24 The flowchart details step 63.

[0126] Figure 26 The element definition information 51 represents implementation method 4.

[0127] Figure 27 Information 72 indicates the definition of the negative element.

[0128] ***Instructions for Action***

[0129] Figure 24is a flowchart explaining the operation of the component search device 10 of Embodiment 4. The operation of the component search device 10 corresponds to a component search method. In addition, the operation of the component search device 10 corresponds to processing of a component search program. As described in Embodiment 1, the component search program is stored in the storage device 920. The component search program can be provided stored in a recording medium readable by a computer, or can be provided as a program product. The component search program causes the computer to execute each processing, each flow, or each procedure of the "parts" of the component image recognition part 11, the element recognition part 12, the category determination part 13, and the search part 14, replacing the "parts" with "processing", "flow", or "procedure".

[0130] Figure 24 The bracketed contents in each step of Figure 24 The operation of the component search device 10 will be described. Figure 24 Steps S60 to S64 in Figure 2 correspond to steps S10 to S14 of

[0131] <Step S60>

[0132] In step S60, the image input part 1 inputs the image of the control panel to the component image recognition part 11.

[0133] <Step S61>

[0134] In step S61, the component image recognition part 11 recognizes one or more component images from the image of the control panel input by the image input part 1.

[0135] <Step S62>

[0136] In step S62, the element recognition part 12 recognizes one or more elements of the component from each of the component images recognized by the component image recognition part 11. The element, as described in "Explanation of Structure" of Embodiment 1, refers to a structural element of the component.

[0137] <Step S63>

[0138] As described in Embodiment 1, the definition information DB 15 stores the structural element definition information 51 defining the elements possessed by the component in units of component categories. The definition information DB 15 is sometimes referred to as the first DB 15.

[0139] <Negative element definition information 72>

[0140] The second DB 62 stores information defining elements that the component does not have, that is, negative element definition information, in units of component categories. The negative element definition information 72 is information defining negative elements that the component does not have, in units of element categories. The element definition information 51 is information defining elements that the component has, in units of component categories. Therefore, the negative element definition information 72 is information completely opposite to the element definition information 51.

[0141] In step S63, the category determination section 13 determines the category of the component from which the plurality of elements are recognized by the element recognition section 12, based on the plurality of recognized elements and the negative element definition information 72. The negative element definition information 72 defines a plurality of negative elements indicating a plurality of elements that cannot constitute the component, in units of component categories. The negative element definition information 72 will be described later with the description of the element definition information 51. Figure 27

[0142] The category determination section 13 determines the "candidates" of the category of the component from which the plurality of elements are recognized by the element recognition section 12, based on the plurality of recognized elements and the element definition information 51.

[0143] The category determination section 13 decides the candidates of the category of the component, based on the negative element definition information 72, and determines the category of the component as the category of the component decided. Here, the element definition information 51 is information defining a plurality of elements constituting the component, in units of component categories.

[0144] The following will be described in detail. The category determination section 13 determines the category of the component recognized by the component image recognition section 11, in units of components, from the element definition information 51 stored in the element definition information DB 15 and the negative element definition information 72 stored in the second DB 62, based on the one or more elements recognized by the element recognition section 12 from the component image. The category determination section 13 gives priority to the negative element definition information 72 over the element definition information 51 when determining the category of the component. The category determination section 13 determines that the category of the component determined based on the element definition information 51 is incorrect when the element recognition section 12 recognizes an element that the component does not have based on the comparison between the element and the negative element definition information 72. Alternatively, the category determination section 13 first performs the comparison between the element recognized by the element recognition section 12 and the negative element definition information 72 when determining the category of the component. The category determination section 13 can not perform the comparison using the element definition information 51 and can not adopt the category of the component when the element recognition section 12 recognizes an element that the component does not have based on the comparison result.

[0145] < Element Definition Information 51 >

[0146] Figure 26 The element definition information 51 in Embodiment 4 is indicated.​Figure 26 The element definition information 51 shows analog instruments, rotary switches, and buttons as component categories. Elements listed as component categories include pointers, knobs, scales, numerical displays, center circles, indicator lights, indicators, and holes. Figure 26 In the diagram, a circle indicates that the component category can have that element. When the component category is analog instrument, the analog instrument can have pointer, scale, numerical display, and center circle as elements. When the component category is rotary switch, the rotary switch can have seven elements as elements, ranging from pointer to indicator. When the component category is button, the button can have numerical display, center circle, and indicator light as elements.

[0147] <Definition Information of Negation Elements 72>

[0148] Figure 27 This indicates the definition information 72 for the negative element in implementation method 4. Figure 27 The negative element definition information 72 shows analog instruments, rotary switches, and buttons as component categories. The negative element definition information 72 also shows pointers, knobs, scales, numerical displays, center circles, indicator lights, indicators, and holes as components. Figure 27 In the diagram, an "×" symbol indicates that the component category cannot have that feature. When the component category is analog instrument, the categories of features that cannot be present are knobs, indicator lights, indicators, and holes. These features are listed in... Figure 27 The element definition information 51 is left blank. Rotary switches and buttons are the same as analog instruments, so their descriptions are omitted.

[0149] <Step S64>

[0150] The retrieval unit 14 retrieves components that are identical or similar to components of the component category determined by the category determination unit 13 from the component DB 20, which stores components for the graphical control screen, based on multiple elements identified by the element identification unit 12 and the component category determined by the category determination unit 13, for use in the graphical control screen.

[0151] Component DB 20 stores multiple components used for generating the graphical control screen. This is the same as that described in Embodiment 1.

[0152] <Step S65>

[0153] In step S65, the retrieval unit 14 displays the retrieval results for component DB 20 on the display device 930.

[0154] <Details of step S63>

[0155] Figure 26This is a flowchart illustrating the detailed actions of the category determination unit 13 in step 63. (See attached diagram.) Figure 25 The details of step S63 performed by the category determination unit 13 will be explained.

[0156] <Step S631>

[0157] In step S631, the category determination unit 13 compares multiple elements identified by the element recognition unit 12 from the component image with the element definition information 51. Based on the comparison results, the category determination unit 13 determines component category candidates for the components in the component image. The category determination unit 13 determines component category candidates on a per-component basis, using the component image identified by the component image recognition unit 11 as a unit. Figure 25 The processing of a component image by the category determination unit 13 is shown.

[0158] <Step S632>

[0159] (1) There are no alternative component categories.

[0160] If the comparison in step S631 results in no candidate component category among the components in the component image (NO in step S632), the process proceeds to step S636. In step S636, the category determination unit 13 sets the component category of the components in component image 81 to unclassified.

[0161] (2) There is a candidate component category.

[0162] The existence of a candidate component category refers to the situation described below. For example, multiple elements identified by the element recognition unit 12 from the component image 81 are designated as "numerical display, center circle, indicator light". The category determination unit 13 determines the category based on the multiple elements and... Figure 25 Based on the comparison result of the element definition information 51, the candidate component category is determined to be "button" (step S631). If there is one candidate component category for the component in the component image, the process proceeds to step S634 (YES in step S632, NO in step S633). In step S634, the category determination unit 13 checks whether there is a candidate "button" among the multiple elements "numerical display, center circle, indicator light". Figure 26 The negative element definition information 72 is used to determine the elements included. In this example, the "numerical display, center circle, and indicator light" of the "button" are not included. Figure 27 The negative element definition information 72 is in the "button" element. Therefore, in this example, the process proceeds to step S635. In step S635, the category determination unit 13 sets the "button" which is a candidate for component category from "candidate" to "decided", and determines the component category of the component of component image 81 as the decided "button".

[0163] If the element identified by the element recognition unit 12 is "scale, numerical display, center circle, indicator light", then in step S631, "button" is determined as a candidate. In this case, "scale" is included in the negative element of "button" in the negative element definition information 72, so in this case, the process proceeds from step S634 to step S636.

[0164] (3) There are multiple candidate component categories.

[0165] The situation where multiple component category candidates exist refers to the following scenario. For example, suppose the multiple elements identified by the element recognition unit 12 from the component image are "pointer, knob, scale, numerical display, center circle". The category determination unit 13 determines the component category candidates as "analog instrument" and "rotary switch" based on the comparison results between the multiple elements and the element definition information 51 (step S631). Although in Figure 27 In the element definition information 51, the analog instrument does not have a "knob" as an element. However, the category determination unit 13 determines the component category of the component in the component image 81 as "analog instrument" and "rotary switch" by comparing the multiple elements "pointer, knob, scale, numerical display, center circle" identified by the element recognition unit 12 with the element definition information 51. If there are multiple candidate component categories for the component in the component image, the process proceeds to step S637 (YES in step S632, YES in step S633).

[0166] In step S637, the category determination unit 13 determines whether each of the candidate "analog instrument" and "rotary switch" exists. Figure 26 The elements included in the negative element definition information 72. In this example, the element "knob" of the candidate "analog instrument" is included in the negative elements of "analog instrument" in the negative element definition information 72. On the other hand, the elements of the candidate "rotary switch" are not included in the negative elements of "analog instrument" in the negative element definition information 72. Therefore, in this example, the process proceeds to step S638. In step S638, the category determination unit 13 excludes "analog instrument" whose elements are included in the negative element definition information 72 from the candidates. Then, the process proceeds to step S632. The candidate is "rotary switch", so the process proceeds to steps S633, S634, and S635. In step S635, the category determination unit 13 sets the "rotary switch" as a candidate component category from "candidate" to "determined", and determines the component category of the component of component image 81 as the determined "rotary switch".

[0167] Further, in a case where the element included in the negative element definition information 72 is not present in the component class candidate in step S637, the component has 2 candidates and is in an undetermined state, and the process proceeds to step S636.

[0168] Explanation of Effects of Embodiment 4

[0169] Conventionally, a similar component is retrieved on the basis of characteristic information such as the size, shape, color, and the like of an element in a component image. Thus, there is a problem in that, for example, in the case of components whose shapes of elements are similar to each other such as a rotary switch and an analog meter, an analog meter whose characteristic information in a control panel image is similar to that of a rotary switch image is erroneously retrieved.

[0170] Regarding this problem, as shown in the negative element definition information 72 of the Figure 27 Figure 27 "knob" is registered as a negative element of an analog meter.

[0171] In a case where the "knob" is present in the rotary switch image in the control panel image, the analog meter is excluded from the component class at the time of determination of the component class by the class determination section 13.

[0172] Thus, there is an effect that a rotary switch similar to a component in a control panel image can be correctly retrieved by the negative element definition information 72.

[0173] Further, although the component retrieval device 10 has the first DB 15, the component DB 20, the second DB 62, and the like in Embodiments 1 to 4, these DBs can also exist outside the component retrieval device 10, for example, like a cloud server.

[0174] Further, in step S63, the class determination section 13 determines the component class using the element definition information 51 and the negative element definition information 72. However, the class determination section 13 can also determine the component class on the basis of a plurality of elements and the negative element definition information 72 without using the element definition information 51.

[0175] The above describes Embodiments 1 to 4. Two or more of these embodiments can also be combined and implemented. Alternatively, one of a plurality of technical matters included in one embodiment can also be partially implemented. Alternatively, two or more of a plurality of technical matters included in one embodiment can also be partially combined and implemented.

[0176] For example, as described below.

[0177] The element attribute identifying section 16 and the element determining section 17 of Embodiment 2 can also be applied to the component search device 10 of Embodiment 4. Specifically, the element attribute identifying section 16 identifies the attribute of each of the plurality of elements identified by the element identifying section 12. The element determining section 17 determines the plurality of elements identified based on the attribute identified by the element attribute identifying section 16 for each element, the attribute definition information 52 in which the attribute is defined in units of elements. Then, in step S63, the category determining section 13 determines the category of the component based on the plurality of elements determined and the negative element definition information 72 by using the plurality of elements determined instead of the plurality of elements identified.

[0178] In addition, the dynamic attribute identifying section of Embodiment 3 can also be applied to the component search device 10 of Embodiment 4. Specifically, the component image identifying section 11 identifies images of the same component from each of the images of the same control panel taken at different timings. The element identifying section 12 identifies a plurality of elements constituting the same component from the images of the same component identified from each of the images of the control panel. Then, the dynamic attribute identifying section 18 stores a change in the attribute for the same element among the plurality of elements identified from the images of the same component identified from each of the images of the control panel in the case where there is a change in the attribute.

[0179] Explanation of Reference Signs

[0180] 1 image input section, 10 component search device, 11 component image identifying section, 12 element identifying section, 13 category determining section, 14 search section, 15 definition information DB, 16 element attribute identifying section, 17 element determining section, 18 dynamic attribute identifying section, 20 component DB, 40 image, 51 element definition information, 52 attribute definition information, 53 dynamic attribute information, 62 second DB, 72 negative element definition information, 901 processor, 902 memory, 911 image interface, 912 image recognition processor, 920 storage device.

Claims

1. A component retrieval device, comprising: The component image recognition unit identifies the images of components from the images on the control panel; An element recognition unit identifies multiple elements constituting the component from the image of the component; The category determination unit determines the category of the component whose multiple elements have been identified based on the identified multiple elements and the negative element definition information that defines multiple negative elements representing multiple elements that cannot constitute a component, on a component category basis. as well as The retrieval unit retrieves components from a database containing multiple components used in the generation of the graphical control screen, based on the identified multiple elements and the determined categories.

2. The component retrieval device according to claim 1, wherein, The category determination unit determines candidates for the category of the component that have been identified based on the multiple elements identified by the element identification unit and the element definition information of the multiple elements constituting the component, in units of component category. Based on the negative element definition information, the unit decides the candidate for the category of the component as the category of the component and determines the decided category of the component as the category of the component.

3. The component retrieval device according to claim 1 or 2, wherein, The component retrieval device also has: The feature attribute identification unit identifies the characteristics of the attributes of each of the identified plurality of features; and The element determination unit determines the identified multiple elements based on the characteristics of the attributes identified for each element and attribute definition information that defines the characteristics of attributes on an element-by-element basis. The category determination unit determines the category of the component by using the determined plurality of elements instead of the identified plurality of elements, thereby determining the category of the component based on the determined plurality of elements and the negative element definition information.

4. The component retrieval device according to claim 3, wherein, The component image recognition unit identifies the same component from images captured at different times on the same control panel. The element recognition unit identifies multiple elements constituting the same component from the images of the same component identified based on the images of the control panel. The component retrieval device further includes a dynamic attribute recognition unit, which stores changes in the characteristics of the same element among the plurality of elements identified from the images of the same component identified based on the images of the control panel, when the characteristics of the attribute change.

5. A computer-readable recording medium having a component retrieval program recorded thereon, the component retrieval program causing the computer to perform the following processes: Component image recognition processing identifies component images from images on the control panel; Element recognition processing identifies multiple elements constituting the component from the image of the component; The category determination process determines the category of the component with the identified multiple elements based on the identified multiple elements and defines negative element definition information for multiple negative elements that cannot be composed of components by component category, in units of component category. as well as The retrieval process retrieves components from a database containing multiple components used in the generation of the graphical control screen, based on the identified multiple elements and the determined categories.

6. A component retrieval method, wherein a computer performs the following processing: Identify component images from the control panel image. Identify multiple elements constituting the component from the image of the component. Based on the identified multiple elements, and negative element definition information that defines multiple negative elements representing multiple elements that cannot constitute a component by component category, the category of the component for which the identified multiple elements have been determined is determined. Based on the identified multiple elements and determined categories, components are retrieved from a database containing multiple components used in the generation of graphical control screens.

Images