Information processing device, grouping method, and grouping program

Abstract

An information processing device (100) has an acquisition unit (120) that acquires an image that includes a plurality of components, a detection unit (130) that detects the plurality of components on the basis of the image, an identification unit (140) that identifies similar components on the basis of respective feature quantities for the plurality of components and the respective sizes of the plurality of components, and a grouping unit (150) that performs grouping on the basis of the center coordinates of similar components.

Description

Information processing device, grouping method, and grouping program

[0001] The present disclosure relates to an information processing device, a grouping method, and a grouping program.

[0002] There is known a technique for identifying an object included in an image. For example, when an image includes a control panel, there is known a technique for identifying components of the control panel (see Patent Document 1).

[0003] International Publication No. 2023 / 032087

[0004] However, there are cases where an image contains multiple parts. There are also cases where the relationship between the parts needs to be known. In order to identify the relationship between the parts, it is necessary to group multiple parts that have a relationship among the multiple parts. The above technology can identify the parts, but cannot group them. Therefore, the problem is how to perform grouping.

[0005] The purpose of this disclosure is to perform grouping.

[0006] According to one aspect of the present disclosure, there is provided an information processing device including: an acquisition unit that acquires an image including a plurality of parts; a detection unit that detects the plurality of parts based on the image; an identification unit that identifies similar parts based on feature amounts of the plurality of parts and sizes of the plurality of parts; and a grouping unit that performs grouping based on center coordinates of the similar parts.

[0007] According to the present disclosure, grouping can be performed.

[0008] 1 is a diagram showing hardware included in an information processing device of embodiment 1. FIG. 2 is a block diagram showing functions of the information processing device of embodiment 1. FIG. 3 is a diagram showing a specific example (part 1) of processing executed by the information processing device of embodiment 1. FIG. 4 is a diagram showing a specific example (part 2) of processing executed by the information processing device of embodiment 1. FIG. 5 is a block diagram showing functions of an information processing device of embodiment 3. FIG. 6 is a diagram showing a specific example (part 1) of processing executed by the information processing device of embodiment 3. FIG. 7 is a diagram showing a specific example (part 2) of processing executed by the information processing device of embodiment 3. FIG. 8 is a block diagram showing functions of an information processing device of embodiment 4. FIG. 9 is a diagram showing a specific example (part 1) of processing executed by the information processing device of embodiment 4. FIG. 10 is a diagram showing a specific example (part 2) of processing executed by the information processing device of embodiment 4.

[0009] Hereinafter, embodiments will be described with reference to the drawings. The following embodiments are merely examples, and various modifications are possible within the scope of the present disclosure.

[0010] Embodiment 1.

[0011] 1 is a diagram showing hardware included in an information processing device according to embodiment 1. The information processing device 100 is a device that executes a grouping method. The information processing device 100 includes a processor 101, a volatile storage device 102, and a non-volatile storage device 103.

[0012] The processor 101 controls the entire information processing device 100. For example, the processor 101 is a central processing unit (CPU) or a field programmable gate array (FPGA). The processor 101 may be a multiprocessor. The information processing device 100 may also include a processing circuit.

[0013] The volatile storage device 102 is a main storage device of the information processing device 100. For example, the volatile storage device 102 is a random access memory (RAM). The nonvolatile storage device 103 is an auxiliary storage device of the information processing device 100. For example, the nonvolatile storage device 103 is a hard disk drive (HDD) or a solid state drive (SSD).

[0014] Next, a description will be given of the functions of the information processing device 100. Fig. 2 is a block diagram showing the functions of the information processing device of embodiment 1. The information processing device 100 has a storage unit 110, an acquisition unit 120, a detection unit 130, an identification unit 140, a grouping unit 150, a setting unit 160, and an output unit 170.

[0015] The storage unit 110 may be realized as a storage area secured in the volatile storage device 102 or the non-volatile storage device 103. Some or all of the acquisition unit 120, detection unit 130, identification unit 140, grouping unit 150, setting unit 160, and output unit 170 may be realized by a processing circuit. Furthermore, some or all of the acquisition unit 120, detection unit 130, identification unit 140, grouping unit 150, setting unit 160, and output unit 170 may be realized as program modules executed by the processor 101. For example, the program executed by the processor 101 is also referred to as a grouping program. For example, the grouping program is recorded on a recording medium.

[0016] The storage unit 110 stores various information.

[0017] The acquisition unit 120 acquires an image. For example, the acquisition unit 120 acquires the image from the storage unit 110. Also, for example, the acquisition unit 120 acquires the image from an external device. Note that the external device is a device that exists outside the information processing device 100. For example, the external device is a cloud server, an external memory, etc. The external device is not shown in the drawing.

[0018] The image will be described. For example, the image may be an image showing a control panel, a display screen of a programmable display, or the like. The image includes a plurality of components. For example, if the image is an image showing a control panel, the plurality of components are components of the control panel. If the image is an image showing a display screen of a programmable display, the plurality of components are components included in the display screen.

[0019] The detection unit 130 detects a plurality of parts included in an image based on the image. Specifically, the detection unit 130 detects the plurality of parts using image recognition technology. Specifically, the detection unit 130 detects the plurality of parts using a trained model. Note that the trained model is acquired by the acquisition unit 120 from the storage unit 110 or an external device.

[0020] The identification unit 140 identifies similar parts based on the feature values ​​of each of the multiple parts and the sizes of each of the multiple parts. The process will be described in detail. The identification unit 140 identifies the feature values ​​of each of the multiple parts (e.g., AKAZE (Accelerated KAZE) feature values). The identification unit 140 identifies the size of each of the multiple parts. For example, the size may be the width, height, and aspect ratio of the part. The identification unit 140 identifies similar parts based on the feature values ​​of each of the multiple parts and the sizes of each of the multiple parts. The identification unit 140 may identify a certain part and a part that is tilted as similar parts. The identification unit 140 may also identify a certain part and a part that is inverted as similar parts.

[0021] The grouping unit 150 performs grouping based on the center coordinates of similar parts. The grouping process will be described in detail below. The grouping unit 150 identifies the center coordinates of similar parts. When the X or Y coordinates of the center coordinates of similar parts are the same, the grouping unit 150 groups parts with the same X or Y coordinate. The horizontal direction of the image is the X axis. The vertical direction of the image is the Y axis.

[0022] The setting unit 160 assigns an ID (identifier) ​​or a name to the group. The output unit 170 outputs the ID or the name and information indicating the components belonging to the group. For example, the output unit 170 outputs this information to a display of the information processing device 100.

[0023] Next, a specific example will be used to explain the processing executed by the information processing device 100. Fig. 3 is a diagram showing a specific example (part 1) of the processing executed by the information processing device of the first embodiment.

[0024] The acquisition unit 120 acquires an image. The image includes buttons 10a to 10d, lamps 11a to 11d, buttons 12a to 12d, and lamps 13a to 13d on the control panel. The detection unit 130 detects buttons 10a to 10d, lamps 11a to 11d, buttons 12a to 12d, and lamps 13a to 13d based on the image.

[0025] The identification unit 140 identifies similar parts based on the feature amounts of each of the multiple parts and the sizes of each of the multiple parts. As a result, buttons 10a, 10b, 12a, and 12b are identified as similar parts. Also, buttons 10c, 10d, 12c, and 12d are identified as similar parts. Furthermore, lamps 11a to 11d and 13a to 13d are identified as similar parts.

[0026] The grouping unit 150 identifies the center coordinates of the buttons 10a, 10b, 12a, and 12b (i.e., similar components). The buttons 10a, 10b, 12a, and 12b have the same Y coordinate. Therefore, the grouping unit 150 groups the buttons 10a, 10b, 12a, and 12b.

[0027] The grouping unit 150 identifies the center coordinates of the buttons 10c, 10d, 12c, and 12d (i.e., similar components). The buttons 10c, 10d, 12c, and 12d have the same Y coordinate. Therefore, the grouping unit 150 groups the buttons 10c, 10d, 12c, and 12d.

[0028] The grouping unit 150 identifies the center coordinates of the lamps 11a to 11d and 13a to 13d (i.e., similar components). The Y coordinates of the lamps 11a to 11d and 13a to 13d are the same. Therefore, the grouping unit 150 groups the lamps 11a to 11d and 13a to 13d.

[0029] The setting unit 160 sets "Group #1" to the group to which buttons 10a, 10b, 12a, and 12b belong. The setting unit 160 sets "Group #2" to the group to which buttons 10c, 10d, 12c, and 12d belong. The setting unit 160 sets "Group #3" to the group to which lamps 11a to 11d and 13a to 13d belong. The output unit 170 outputs this information to the display of the information processing device 100.

[0030] According to the first embodiment, as described above, the information processing device 100 can perform grouping.

[0031] The grouping unit 150 may perform the following processing. When grouping similar parts based on the central coordinates of the parts, the grouping unit 150 does not group parts that are separated by a predetermined threshold or more. The processing of the grouping unit 150 will be specifically described with reference to FIG. 4 .

[0032] 4 is a diagram showing a specific example (part 2) of the process executed by the information processing device of embodiment 1. It is assumed that the distance between button 10b and button 12a is equal to or greater than a threshold. The grouping unit 150 does not group button 10b and button 12a. The grouping unit 150 then groups buttons 10a and 10b. The grouping unit 150 also groups buttons 12a and 12b.

[0033] It is assumed that the distance between button 10d and button 12c is equal to or greater than a threshold value. The grouping unit 150 does not group button 10d and button 12c. The grouping unit 150 then groups buttons 10c and 10d. The grouping unit 150 also groups buttons 12c and 12d.

[0034] It is assumed that the distance between lamp 11d and lamp 13a is equal to or greater than a threshold value. The grouping unit 150 does not group lamp 11d with lamp 13a. The grouping unit 150 then groups lamps 11a to 11d. The grouping unit 150 also groups lamps 13a to 13d.

[0035] In this way, the information processing device 100 does not group parts that are far apart. When parts are far apart, the relationship between the parts is considered to be weak. Therefore, by not grouping parts that are far apart, the information processing device 100 can improve the accuracy of grouping.

[0036] The grouping unit 150 may also perform the following process. When three or more similar parts have the same X or Y coordinate and the same spacing (i.e., distance between parts), the grouping unit 150 groups the three or more similar parts. The relationship between such parts is considered to be strong. Therefore, by performing grouping, the information processing device 100 can improve the accuracy of grouping.

[0037] Second Embodiment Next, a second embodiment will be described. In the second embodiment, differences from the first embodiment will be mainly described. Furthermore, in the second embodiment, descriptions of the commonalities between the first embodiment and the second embodiment will be omitted.

[0038] In the second embodiment, the grouping method is different from that in the first embodiment, so only the processing of the grouping unit 150 will be described.

[0039] When the number of similar parts is three or more, the grouping unit 150 performs a regression analysis based on the center coordinates of the similar parts. The grouping unit 150 performs grouping based on the results of the regression analysis. For example, the grouping unit 150 calculates a regression line based on the center coordinates of each of three or more similar parts. The grouping unit 150 groups parts for which the distance between the center coordinates of each of the three or more similar parts and the regression line is equal to or less than a threshold. Furthermore, for example, the grouping unit 150 calculates a regression curve based on the center coordinates of each of the three or more similar parts. The grouping unit 150 groups parts for which the distance between the center coordinates of each of the three or more similar parts and the regression curve is equal to or less than a threshold. Note that, for example, when grouping parts of a clock face, the grouping unit 150 can use the regression curve to group parts arranged on the curve.

[0040] In the first embodiment, a rule that the X coordinate or the Y coordinate must be the same is set in advance. On the other hand, when regression analysis is used, it is not necessary to set a rule in advance. Therefore, according to the second embodiment, grouping is realized without setting a rule in advance.

[0041] Embodiment 3 Next, embodiment 3 will be described. In embodiment 3, differences from embodiment 1 will be mainly described. Furthermore, in embodiment 3, descriptions of matters common to embodiment 1 will be omitted.

[0042] 5 is a block diagram showing the functions of an information processing device according to embodiment 3. The information processing device 100a includes a grouping unit 150a. The functions of the grouping unit 150a will be described later.

[0043] Next, the processing executed by the information processing device 100a will be described using a specific example. Fig. 6 is a diagram showing a specific example (part 1) of the processing executed by the information processing device of embodiment 3. The acquisition unit 120 acquires an image 200. The image 200 includes buttons 21a to 21f and lamps 22a to 22f on a control panel. The buttons 21a to 21f and lamps 22a to 22f are also surrounded by a frame 20.

[0044] The detection unit 130 detects the buttons 21a to 21f and the lamps 22a to 22f based on the image 200. The detection unit 130 detects the frame 20 based on the edges of the image 200.

[0045] The identification unit 140 identifies similar parts based on the feature amounts of each of the multiple parts and the sizes of each of the multiple parts. As a result, buttons 21a to 21d are identified as similar parts. Buttons 21e and 21f are also identified as similar parts. Furthermore, lamps 22a to 22f are identified as similar parts. The grouping unit 150a groups similar parts included in frame 20 together. The grouping process will be described using FIG. 7.

[0046] 7 is a diagram showing a specific example (part 2) of the processing executed by the information processing device of embodiment 3. The grouping unit 150a groups the buttons 21a to 21d included in the frame 20. The grouping unit 150a groups the buttons 21e and 21f included in the frame 20. The grouping unit 150a groups the lamps 22a to 22f included in the frame 20.

[0047] The setting unit 160 sets "Group #1" to the group to which buttons 21a to 21d belong. The setting unit 160 sets "Group #2" to the group to which buttons 21e and 21f belong. The setting unit 160 sets "Group #3" to the group to which lamps 22a to 22f belong. The output unit 170 outputs this information to the display of the information processing device 100a.

[0048] According to the third embodiment, the information processing device 100a groups similar parts included in the frame 20. When similar parts exist in the same frame, it can be said that the similar parts have a strong relationship. Therefore, the information processing device 100a can achieve highly accurate grouping by grouping similar parts that exist in the same frame.

[0049] Embodiment 4 Next, embodiment 4 will be described. In embodiment 4, differences from embodiments 1 to 3 will be mainly described. Furthermore, in embodiment 4, descriptions of matters common to embodiments 1 to 3 will be omitted.

[0050] 8 is a block diagram showing the functions of the information processing device of embodiment 4. The information processing device 100 further includes an evaluation unit 180. Part or all of the evaluation unit 180 may be realized by a processing circuit. Alternatively, part or all of the evaluation unit 180 may be realized as a program module executed by the processor 101. The functions of the evaluation unit 180 will be described later.

[0051] Next, the processing executed by the information processing device 100 will be described using a specific example. Fig. 9 is a diagram showing a specific example (part 1) of the processing executed by the information processing device of embodiment 4. Fig. 9 shows an image 300. The image 300 includes buttons 30 to 32. The buttons 30 to 32 are grouped by the processing of the grouping unit 150. In other words, the buttons 30 to 32 are multiple components that belong to a group that has been set by the grouping processing.

[0052] FIG. 10 is a diagram showing a specific example (part 2) of processing executed by the information processing device of embodiment 4. The acquisition unit 120 acquires information indicating the positions of multiple parts belonging to a group established by a previous grouping process from the storage unit 110 or an external device. The information acquisition process will be described using a specific example. FIG. 10 shows an image 400. The image 400 includes buttons 40 to 42. The buttons 40 to 42 were previously grouped by processing by the grouping unit 150. Information indicating the positions of the buttons 40 to 42 belonging to the group is stored in the storage unit 110 or the external device. The acquisition unit 120 acquires the information from the storage unit 110 or the external device.

[0053] The evaluation unit 180 compares the positions of the buttons 30 to 32 with the positions of the buttons 40 to 42. The evaluation unit 180 evaluates the accuracy of the grouping based on the difference between the positions of the buttons 30 to 32 and the positions of the buttons 40 to 42. For example, if the difference is equal to or less than a threshold, the evaluation unit 180 evaluates that the grouping of the buttons 30 to 32 is correct. If the difference is greater than the threshold, the evaluation unit 180 evaluates that the grouping of the buttons 30 to 32 is incorrect.

[0054] The output unit 170 outputs the evaluation result. If the difference is equal to or smaller than a threshold, the output unit 170 may output the ID or name and information indicating the component belonging to the group. If the difference is greater than the threshold, the output unit 170 does not output this information.

[0055] The reason why the information processing device 100 performs the above processing will be explained. The information processing device 100 checks the accuracy of the grouping of the buttons 30 to 32 by using information on past groupings. Note that image 300 includes two meters. On the other hand, image 400 includes one meter. As such, there are differences between image 300 and image 400. However, as long as buttons are normally arranged in this manner, it is acceptable to use images containing different items.

[0056] According to the fourth embodiment, the information processing device 100 can check the accuracy of the grouping. Moreover, the fourth embodiment can be applied to the third embodiment.

[0057] The features of the above-described embodiments can be combined with each other as appropriate.

[0058] 10a to 10d buttons, 11a to 11d lamps, 12a to 12d buttons, 13a to 13d lamps, 20 frame, 21a to 21f buttons, 22a to 22f lamps, 30 to 32 buttons, 40 to 42 buttons, 100 information processing device, 100a information processing device, 101 processor, 102 volatile storage device, 103 non-volatile storage device, 110 storage unit, 120 acquisition unit, 130 detection unit, 140 identification unit, 150 grouping unit, 150a grouping unit, 160 setting unit, 170 output unit, 180 evaluation unit, 200 image, 300 image, 400 image.

Claims

1. An acquisition unit that acquires an image including a plurality of components, a detection unit that detects the plurality of components based on the image, a specification unit that specifies similar components based on the respective feature amounts of the plurality of components and the respective sizes of the plurality of components, and a grouping unit that performs grouping based on the center coordinates of the similar components. An information processing apparatus having the above components.

2. When performing grouping based on the center coordinates of the similar components, the grouping unit does not group components that are separated by a predetermined threshold or more. The information processing apparatus according to claim 1.

3. When the X coordinates or Y coordinates of three or more similar components are the same and the intervals are the same, the grouping unit groups the three or more similar components. The information processing apparatus according to claim 1.

4. When the number of similar components is three or more, the grouping unit performs regression analysis based on the center coordinates of the similar components and performs grouping based on the result of the regression analysis. The information processing apparatus according to claim 1.

5. An acquisition unit that acquires an image including a plurality of components surrounded by a frame, a detection unit that detects the plurality of components based on the image, a specification unit that specifies similar components based on the respective feature amounts of the plurality of components and the respective sizes of the plurality of components, and a grouping unit that groups similar components included in the frame. An information processing apparatus having the above components.

6. Further having an evaluation unit, the acquisition unit acquires information indicating the positions of a plurality of components belonging to a group, and the evaluation unit evaluates the accuracy of grouping based on the difference between the positions of a plurality of components belonging to a group set by performing the grouping process and the positions of the plurality of components indicated by the acquired information. The information processing apparatus according to any one of claims 1 to 5.

7. A grouping method in which an information processing apparatus acquires an image including a plurality of components, detects the plurality of components based on the image, specifies similar components based on the respective feature amounts of the plurality of components and the respective sizes of the plurality of components, and performs grouping based on the center coordinates of the similar components.

8. A grouping method in which an information processing apparatus acquires an image including a plurality of components surrounded by a frame, detects the plurality of components based on the image, identifies similar components based on respective feature amounts of the plurality of components and respective sizes of the plurality of components, and groups similar components included in the frame.

9. A grouping program that causes an information processing apparatus to execute a process of acquiring an image including a plurality of components, detecting the plurality of components based on the image, identifying similar components based on respective feature amounts of the plurality of components and respective sizes of the plurality of components, and performing grouping based on center coordinates of the similar components.

10. A grouping program that causes an information processing apparatus to execute a process of acquiring an image including a plurality of components surrounded by a frame, detecting the plurality of components based on the image, identifying similar components based on respective feature amounts of the plurality of components and respective sizes of the plurality of components, and grouping similar components included in the frame.

Images