Classifier generation device, classifier generation method, classifier generation program, and image classification device

Abstract

To easily generate a region classifier.SOLUTION: A classifier generation apparatus 10 includes a patch image acquisition unit 40A, a patch classifier generation unit 40B, a region classification result image generation unit 40C, and a region classifier generation unit 40F. The patch image acquisition unit 40A acquires a partial region consisting of a plurality of pixels in a teacher image which is an image of a state-variable object 20, as a patch image. The patch classifier generation unit 40B generates a patch classifier that takes a patch image as an input and outputs a classification result of a state of the object captured in the patch image. The region classification result image generation unit 40C generates, using the patch classifier, a region classification result image that specifies a classification result for each element that is an element constituting of an image and is a region smaller than the patch image. The region classifier generation unit 40F generates, using learning data constituting of a pair of the teacher image and the region classification result image, a region classifier that takes an image as an input and outputs a region classification result that is the region classification result image.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] Embodiments of the present invention relate to a classifier generation device, a classifier generation method, a classifier generation program, and an image classification device.

Background Art

[0002] There is disclosed a method of generating a region classifier that outputs a region classification result in which classification results are defined for each pixel from an image using learning data in which a correct class is taught for each of all pixels in the image (see, for example, Patent Document 1, Patent Document 2, and Patent Document 3, etc.). Further, there is disclosed a method of generating a region classifier by performing teaching of the type of object captured in the image or teaching of surrounding the region of the object captured in the image with a rectangular frame (see, for example, Non-Patent Document 1, Non-Patent Document 2, and Non-Patent Document 3). Further, there is disclosed a method of generating a region classifier using learning data in which a correct class is taught for some pixels in the image (see Non-Patent Document 4, etc.).

[0003] However, the method of teaching a correct class for each of all pixels in the image has a high work load for the user performing the teaching work. Also, when an object in which a plurality of regions representing different classification results are mixed is captured in the image, it may be difficult for the user to teach the correct class of pixels near the boundary of the regions. Further, in the method of teaching the type of object captured in the image, it is necessary to prepare a large amount of images in which various types of objects are captured for the generation of the region classifier. Further, in the method of performing teaching of surrounding the region of the object captured in the image with a rectangular frame, it is necessary to prepare an image in which the boundaries between a plurality of regions representing different classification results are clear. Further, in the method of teaching a correct class for some pixels in the image, it is necessary to prepare a large amount of images in which various types of objects are captured at various positions for the generation of the region classifier. That is, in the prior art, it has been difficult to easily generate a region classifier.

Prior Art Documents

Patent Documents

[0004] [License 1] Patent No. 6309663 [License 2] Special Announcement No. 2021-509713 [License 3] Patent No. 6800901 [Non-licensed literature]

[0005] [Non-licensed Document 1] Guolei Sun, Wenguan Wang, Jifeng Dai, and Luc Van Gool. Mining cross-image semantics for weakly supervised semantic segmentation. In ECCV, 2020. [Non-licensed Document 2] Ahn, J., Cho, S., Kwak, S.: Weakly supervised learning of instance segmentation with inter-pixel relations. In: CVPR (2019) [Non-licensed Document 3] A. Khoreva, R. Benenson, J. Hosang, M. Hein, and B. Schiele. Simple does it: Weakly supervised instance and semantic segmentation. In CVPR, 2017. [Non-licensed Document 4] D. Lin, J. Dai, J. Jia, K. He, and J. Sun. Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. In (CVPR), 2016. [Non-licensed Document 5] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” in MICCAI. Springer, 2015, pp. 234-241. [Non-Patent Document 6] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilateral segmentation network for real-time semantic segmentation. In Proceedings of the European Conference on Computer Vision(ECCV), pages 325-341, 2018. [Non-Patent Document 7] P. Krahenb ¨ uhl and V. Koltun, “Efficient inference in fully connected crfs with gaussian edge potentials,” in NIPS, 2011. [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] The present invention has been made in view of the above, and aims to provide a classifier generation device, a classifier generation method, a classifier generation program, and an image classification device that can easily generate a region classifier. [Means for solving the problem]

[0007] The classifier generation device of the embodiment comprises a patch image acquisition unit, a patch classifier generation unit, a region classification result image generation unit, and a region classifier generation unit. The patch image acquisition unit acquires a partial region consisting of multiple pixels in a training image, which is an image of an object whose state is variable, as a patch image. The patch classifier generation unit takes the patch image as input and generates a patch classifier that outputs the classification result of the state of the object captured in the patch image. The region classification result image generation unit uses the patch classifier to generate a region classification result image in which the classification result is defined for each element that constitutes the image and is a region smaller than the patch image. The region classifier generation unit uses training data consisting of pairs of the training image and the region classification result image to generate a region classifier that takes the image as input and outputs the region classification result, which is the region classification result image. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a schematic diagram showing an example of the overall configuration of the generation system. [Figure 2] Figure 2 is a block diagram showing an example of the functional configuration of the generation system. [Figure 3] Figure 3 is an explanatory diagram illustrating an example of patch image acquisition. [Figure 4] Figure 4 is a schematic diagram showing an example of a patch image. [Figure 5] Figure 5 is an explanatory diagram illustrating an example of patch classifier generation. [Figure 6A] Figure 6A is an explanatory diagram illustrating an example of the generation of region classification result images. [Figure 6B] Figure 6B is an explanatory diagram illustrating an example of the generation of a region classification result image. [Figure 7] Figure 7 is an explanatory diagram illustrating an example of the generation of a region classifier. [Figure 8] Figure 8 is a flowchart showing an example of an information processing flow. [Figure 9A] Figure 9A is an explanatory diagram illustrating an example of patch classifier generation. [Figure 9B] Figure 9B is an explanatory diagram illustrating an example of patch classifier generation. [Figure 9C] FIG. 9C is an explanatory diagram of an example of generation of a patch classifier. [Figure 10] FIG. 10 is a hardware configuration diagram of an example of a classifier generation device. **Embodiments for Carrying Out the Invention**

[0009] Hereinafter, a classifier generation device, a classifier generation method, a classifier generation program, and an image classification device will be described in detail with reference to the accompanying drawings.

[0010] (First Embodiment) FIG. 1 is a schematic diagram showing an example of the overall configuration of a generation system 1 according to the present embodiment. In the present embodiment, an example in which the generation system 1 is applied to a facility for storing garbage will be described as an example. Note that the application target of the generation system 1 is not limited to a facility for storing garbage.

[0011] The generation system 1 includes a classifier generation device 10, a photographing unit 12, and an operation mechanism 14. The classifier generation device 10, the photographing unit 12, and the operation mechanism 14 are communicably connected.

[0012] The classifier generation device 10 is an information processing device that takes an image of an object 20 as an input and generates a region classifier that outputs a region classification result of the object 20. Details of the region classification result and the region classifier will be described later.

[0013] The object 20 is an object to be region-classified. The object 20 is an object whose state can change. The object 20 may be any of an object whose state changes when a force is applied from the outside, an object whose state changes when a force is applied from the inside, and an object whose state changes autonomously. The object 20 is, for example, garbage, sludge, soil, gravel, minerals, wood, pruning branches, a fluid such as a liquid, a person, a vehicle, an animal, and the like.

[0014] In this embodiment, the object 20 is described as waste as an example. Waste is an aggregate of multiple substances, and when an external force is applied, the state of the substances, the positional relationship of the substances, the types of substances, etc., change.

[0015] In this embodiment, the object 20 is housed within the structure 22.

[0016] The structure 22 is an object in which the object 20 is in contact with at least a portion of its area in real space. The structure 22 is fixed in shape and position. However, the shape of a portion of the structure 22 may be variable.

[0017] The structure 22 is, for example, a box-shaped member that holds or supports the object 20. In this embodiment, the structure 22 is described as a storage tank that stores the object 20, which is garbage, inside. A storage tank that stores garbage is sometimes called a garbage pit. In this embodiment, the structure 22 is described as a garbage pit installed in an incineration facility or the like for burning garbage. In this embodiment, the shape of the structure 22 is described as a substantially rectangular parallelepiped as an example. However, the shape of the structure 22 is not limited to a rectangular parallelepiped.

[0018] The structure 22, which is a waste pit, stores the waste 20 at its inner bottom. The walls of the structure 22 are provided with, for example, an entrance 22A for bringing in the waste 20, and an entrance 22D for discharging the waste 20 to an incinerator or the like. Below the entrance 22A on the inner wall of the structure 22, there is a ramp 22B for guiding the waste 20 to the bottom of the structure 22. The waste 20 brought into the structure 22 through the entrance 22A is stored at the bottom of the structure 22 via the ramp 22B. The ramp 22B is provided with, for example, a protective member 22C to prevent the waste 20 from scattering and to guide it to the bottom. The waste 20 flows along the slope of the ramp 22B to the bottom of the structure 22, with scattering suppressed by the protective member 22C, and is stored at the bottom. Furthermore, the structure 22 may be any box-shaped member that holds or supports the object 20, and is not limited to the above configuration.

[0019] The imaging unit 12 outputs captured image data through imaging. The imaging unit 12 is, for example, a video camera, a network camera, a single-lens reflex camera, or a camera mounted on a smartphone. The imaging unit 12 is positioned to be able to photograph the object 20 and the structure 22. The imaging unit 12 acquires captured image data including the object 20 and the structure 22 through imaging. In the following, the captured image data will be simply referred to as an image.

[0020] The image captured by the camera unit 12 includes at least the object 20. The image may be either a color image or a grayscale image.

[0021] The operating mechanism 14 is a mechanism that performs operations on an object 20 in real space. The operation on the object 20 may be an operation on the entire object 20 or an operation on a part of the object 20. The operating mechanism 14 may be a mechanism that performs operations by physically contacting the object 20, or a mechanism that performs operations on the object 20 without physical contact using light, liquid, etc. Mechanisms that perform operations by physically contacting the object 20 include, for example, a crane that grips, stirs, moves, releases from gripping, hooks, etc., a drill that drills holes in the object 20, a shovel that excavates, etc., the object 20, etc. Mechanisms that perform operations on the object 20 without contact include, for example, a mechanism that irradiates light, a mechanism that sprays liquid, etc. If the object 20 is magnetic, the operating mechanism 14 may be a mechanism that holds or releases the object 20 by magnetic force.

[0022] In this embodiment, one example described is a configuration in which the operating mechanism 14 is a crane that performs operations such as gripping, stirring, moving, releasing from gripping, and hooking of the object 20.

[0023] The operating mechanism 14 is supported by a support part 16 fixed to the structure 22. The support part 16 supports the operating mechanism 14 so that it can move in the vertical direction (arrow Y direction) and in directions perpendicular to the vertical direction (arrow X direction, arrow Z direction). The arrow Y direction is the direction that coincides with the vertical direction. The arrow Y direction, arrow X direction, and arrow Z direction are perpendicular to each other. The vertical direction, arrow Y direction, may be described as the Y-axis or Y-axis direction, the arrow X direction as the X-axis or X-axis direction, and the arrow X direction as the Z-axis or Z-axis direction.

[0024] By being supported so as to be movable in the X, Y, and Z axes, the operating mechanism 14 is configured to be able to manipulate the object 20 at various positions in the space within the structure 22. Furthermore, the tip of the operating mechanism 14 is configured, for example, as a drivable claw, and operations such as gripping and releasing the object 20 are performed by driving the claw. In addition, the operating mechanism 14 is supported so as to be movable by the support part 16 and is equipped with a drivable claw, so that operations such as stirring and moving are possible with respect to the object 20.

[0025] The classifier generation device 10 generates a region classifier using an image that includes the object 20 captured by the imaging unit 12.

[0026] Figure 2 is a block diagram showing an example of the functional configuration of the generation system 1. For illustrative purposes, Figure 2 also shows a portion of the object 20 and the structure 22 together.

[0027] The classifier generation device 10 comprises a storage unit 30, a UI (user interface) unit 32, a communication unit 34, and a control unit 40. The storage unit 30, UI unit 32, communication unit 34, and control unit 40 are communicated together via a bus 36 or the like.

[0028] The storage unit 30 stores various types of data. The storage unit 30 may be, for example, a semiconductor memory element such as RAM (Random Access Memory) or flash memory, a hard disk, or an optical disk. The storage unit 30 may also be a storage device located outside the classifier generation device 10.

[0029] The communication unit 34 is a communication interface that communicates with the imaging unit 12 and the operating mechanism 14. The communication unit 34 may also communicate with an external information processing device via a network or the like.

[0030] The UI unit 32 has a reception function for receiving user input and a display function for displaying various information. The reception function is implemented, for example, by a pointing device such as a mouse or a keyboard. The display function is implemented, for example, by a display. The UI unit 32 may be a touch panel that integrates the reception function and the display function.

[0031] The control unit 40 performs various information processing in the classifier generation device 10.

[0032] The control unit 40 includes a patch image acquisition unit 40A, a patch classifier generation unit 40B, a region classification result image generation unit 40C, a specific unit 40D, a correction unit 40E, a region classifier generation unit 40F, and a viewpoint transformation image generation unit 40G.

[0033] The patch image acquisition unit 40A, the patch classifier generation unit 40B, the region classification result image generation unit 40C, the identification unit 40D, the correction unit 40E, the region classifier generation unit 40F, and the viewpoint transformation image generation unit 40G are implemented by, for example, one or more processors. For example, each of the above units may be implemented by having a processor such as a CPU (Central Processing Unit) execute a program, i.e., by software. Each of the above units may be implemented by a dedicated IC or other processor, i.e., by hardware. Each of the above units may be implemented by using both software and hardware. When multiple processors are used, each processor may implement one of the above units, or two or more of the above units. Furthermore, at least one of the above units may be mounted on an external information processing device connected to the control unit 40 via the communication unit 34 and a network, etc.

[0034] The patch image acquisition unit 40A acquires a patch image from a training image, which is an image of the target object 20, consisting of a subregion of multiple pixels.

[0035] Figure 3 is an explanatory diagram illustrating an example of acquiring patch images 54. In Figure 3, images 50 captured by the imaging unit 12 are shown as training images 52. Training images 52 are images 50 from the images 50 captured by the imaging unit 12 that are used for training the patch classifier and the region classifier, which will be described later.

[0036] A patch image 54 is an image of a subregion consisting of multiple pixels in the training image 52. In other words, a patch image 54 is an image of a subregion extracted from a part of the training image 52 at an arbitrary position in the training image 52. Figure 3 shows a patch image 54 as an area extracted from within an image frame 56 placed at an arbitrary position in the training image 52. Furthermore, a patch image 54 is an area consisting of a group of multiple pixels. The size of the patch image 54 is smaller than the training image 52, larger than the size of one pixel that makes up the training image 52, and corresponding to a group of multiple pixels. The size of the patch image 54 can be any size that satisfies the above conditions, but it is preferable that the size of the patch image 54 is such that the user can clearly identify the state of the object 20 captured within the patch image 54.

[0037] The shape of the patch image 54 is not limited. The shape of the patch image 54 may be rectangular, circular, or any other shape other than rectangular and circular. In this embodiment, the case in which the shape of the patch image 54 is rectangular will be described as an example.

[0038] Returning to Figure 2, the explanation continues. The patch image acquisition unit 40A acquires the image 50 of the object 20 captured by the imaging unit 12 as the training image 52.

[0039] For example, each time a new image 50 is captured by the imaging unit 12, the patch image acquisition unit 40A acquires the captured image 50 as a training image 52 from the imaging unit 12.

[0040] Alternatively, the patch image acquisition unit 40A may acquire the training image 52 by acquiring the image 50 from the storage unit 30. In this case, the control unit 40 only needs to store one or more images 50 captured by the imaging unit 12 in the storage unit 30. Then, the patch image acquisition unit 40A can acquire the image 50 stored in the storage unit 30 as the training image 52.

[0041] The patch image acquisition unit 40A acquires the patch image 54 by cutting out a region within the image frame 56, which is the outer frame of the patch image 54, at an arbitrary position in the training image 52. The patch image acquisition unit 40A acquires multiple patch images 54 by cutting out regions within the image frame 56 from each of several different positions in a single training image 52. Alternatively, the patch image acquisition unit 40A may acquire patch images 54 by cutting out one or more patch images 54 from each of several training images 52.

[0042] Figure 4 is a schematic diagram showing an example of a patch image 54 extracted from a training image 52. For example, the patch image acquisition unit 40A acquires multiple patch images 54 (for example, patch images 54A to 54E) by extracting regions within each image frame 56 at arbitrary positions in the training image 52.

[0043] Returning to Figure 2, we continue the explanation.

[0044] The patch classifier generation unit 40B generates a patch classifier.

[0045] Figure 5 is an explanatory diagram of an example of the generation of a patch classifier 60. The patch classifier 60 is a classifier that takes a patch image 54 as input and outputs a classification result 58 of the state of the object 20 captured in the patch image 54.

[0046] The classification result 58 is information representing the classification result of the state of the object 20. The classification result 58 may be referred to as the correct label, teaching data, etc. If the object 20 is garbage, the state of the object 20 is represented, for example, by the type of substance that makes up the object 20, the degree of mixing of the substance that makes up the object 20, etc. For the state of the object 20, one of several types obtained by classifying the state of the object 20 according to a predetermined classification rule may be used. The state of the object 20 may include information indicating that it is not the object 20. In this embodiment, the form in which the classification result 58 represents one of the following states will be explained as an example. The structure corresponds to structure 22. Note that the classification result 58 may be determined according to the type of object 20 and the classification rule for the state of the object 20, and is not limited to mixed, unmixed, sludge, pruned branches, structure, etc.

[0047] This will be explained using Figure 4. The patch classifier generation unit 40B acquires multiple pairs of patch images 54 acquired by the patch image acquisition unit 40A and classification results 58 taught to the patch images 54.

[0048] For example, the user operates the UI unit 32 to teach the correct label, classification result 58, to the patch image 54 acquired by the patch image acquisition unit 40A. More specifically, the patch classifier generation unit 40B displays a list of patch images 54 acquired by the patch image acquisition unit 40A on the UI unit 32. The user operates the UI unit 32 to teach the correct label, classification result 58, to each of the patch images 54 displayed on the UI unit 32.

[0049] Figure 4 shows an example of a situation in which each of patch images 54A to 54E was taught as a classification result 58 (classification result 58A to classification result 58E), with categories such as mixed, unmixed, sludge, pruned branches, and structures, respectively.

[0050] As described above, the patch image 54 is an image obtained by cutting out a sub-region of a part of the training image 52 at an arbitrary position in the training image 52, and is a region consisting of a group of pixels. Therefore, the user can easily and accurately teach the classification result 58 for each patch image 54 compared to teaching the classification result 58 for each pixel that makes up the training image 52. In addition, by teaching the classification result 58 for each patch image 54, the user can reduce the workload of the teaching work compared to teaching the classification result 58 for each pixel.

[0051] The patch classifier generation unit 40B may also obtain a classification result 58 as output from the patch classifier 60 by inputting the patch image 54 into the generated patch classifier 60. The patch classifier generation unit 40B may then teach the patch image 54 input to the patch classifier 60 the classification result 58 output from the patch classifier 60.

[0052] This will be explained using Figure 5.

[0053] The patch classifier generation unit 40B uses pairs of patch images 54 and classification results 58 taught for those patch images 54 as first training data, and trains the patch classifier 60 using machine learning with multiple sets of first training data. Through this training process, the patch classifier generation unit 40B generates a patch classifier 60 that takes patch images 54 as input and outputs classification results 58. Deep learning or similar methods can be used for machine learning.

[0054] Furthermore, it is preferable that the patch classifier generation unit 40B generates the patch classifier 60 using a patch image 54 acquired by the patch image acquisition unit 40A that contains an object 20 in only one state.

[0055] A patch image 54 in which only one type of object 20 is captured means that the patch image 54 does not contain a mixture of multiple types of states. In other words, it means that the patch image 54 contains only objects 20 in a state represented by a single classification result 58. Specifically, a patch image 54 in which only one type of object 20 is captured means that the patch image 54 does not contain a mixture of two or more states represented by two or more classification results 58 from among mixed, unmixed, sludge, pruned branches, and structures, and only captures one type of state.

[0056] In this case, the patch classifier generation unit 40B displays, for example, a list of patch images 54 acquired by the patch image acquisition unit 40A on the UI unit 32. By operating the UI unit 32, the user is instructed with a classification result 58 for patch images 54 that contain an object 20 in only one state from the list of patch images 54 displayed on the UI unit 32. For example, the user is instructed with "stirred" as the classification result 58 for a patch image 54 in which an object 20 in the "stirred" state is captured throughout the entire area of ​​the patch image 54. On the other hand, the user is not instructed with a classification result 58 for a patch image 54 in which an object 20 in both "stirred" and "unstirred" states is captured within the patch image 54.

[0057] The patch classifier generation unit 40B then generates the patch classifier 60 by training the patch classifier 60 using machine learning with multiple sets of first training data, with the pair of the taught patch image 54 and the classification result 58 being used as the first training data.

[0058] The patch classifier generation unit 40B generates a patch classifier 60 using a patch image 54 in which only one type of object 20 is captured, thereby enabling the generation of a patch classifier 60 capable of outputting a more accurate classification result 58 for the input patch image 54. Furthermore, the user does not need to perform the teaching work of teaching the classification result 58 for a patch image 54 in which multiple types of objects 20 are captured. This reduces the teaching workload for the user.

[0059] Furthermore, the patch classifier generation unit 40B may input the patch image 54 acquired by the patch image acquisition unit 40A into a pre-generated region classifier (described later), and identify patch images 54 in which the classification results 58 defined for each pixel constituting the image output from the region classifier are the same within the patch image 54 as patch images 54 containing an object 20 in only one state. The patch classifier generation unit 40B may then use the identified patch image 54 and the identical classification results 58 defined for each pixel within the patch image 54 as first training data.

[0060] As described above, the operating mechanism 14 is a mechanism such as a crane that performs operations on the object 20, such as gripping, stirring, moving, releasing from gripping, and hooking. Therefore, the state of the object 20 changes depending on the movement of the operating mechanism 14.

[0061] Therefore, the patch classifier generation unit 40B may teach the patch image 54 a classification result 58 corresponding to the movement information of the operating mechanism 14. The movement information is information that represents the movement of the operating mechanism 14 on the object 20. For example, the movement information represents stirring of the object 20, moving the object 20, putting the object 20 in, etc.

[0062] In this case, the patch classifier generation unit 40B acquires motion information of the operating mechanism 14 for performing operations on the object 20. The patch classifier generation unit 40B acquires motion information representing the movement of the operating mechanism 14 and motion date and time information indicating the date and time the movement represented by the motion information was performed from the operating mechanism 14 via the communication unit 34. Then, the patch classifier generation unit 40B acquires the patch image 54 acquired by the patch image acquisition unit 40A and shooting date and time information indicating the shooting date and time of the training image 52 from which the patch image 54 was extracted.

[0063] The patch classifier generation unit 40B identifies motion information corresponding to motion date and time information that matches the shooting date and time information as motion information of the operating mechanism 14 that corresponds to the patch image 54 extracted from the training image 52 taken at the shooting date and time represented by the shooting date and time information.

[0064] The patch classifier generation unit 40B then reads the classification result 58 corresponding to the identified motion information from the storage unit 30 and automatically teaches the classification result 58 to the patch image 54 without receiving any operation instructions from the user. The storage unit 30 only needs to store in advance the motion information of the operating mechanism 14 and the classification result 58 that represents the state of the object 20 when the operating mechanism 14 performs the motion represented by the motion information, in an associated manner.

[0065] Furthermore, the operating mechanism 14 may be visible in the training image 52. As mentioned above, the operating mechanism 14 is a mechanism such as a crane that performs operations on the object 20. Therefore, the operating mechanism 14 may be visible in the image 50 taken during the period when the operating mechanism 14 is performing operations on the object 20.

[0066] Therefore, it is preferable that the patch classifier generation unit 40B excludes from the patch images 54 acquired by the patch image acquisition unit 40A any patch images 54 that include the operating mechanism 14, from the patch images 54 used to generate the patch classifier 60.

[0067] In detail, for example, the patch classifier generation unit 40B displays a list of patch images 54 acquired by the patch image acquisition unit 40A on the UI unit 32. By operating the UI unit 32, the user can choose not to teach the classification result 58 to patch images 54 in which the operating mechanism 14 is visible among the list of patch images 54 displayed on the UI unit 32. The patch classifier generation unit 40B then generates the patch classifier 60 by training using the pairs of patch images 54 that have been taught the classification result 58 and the classification result 58 as first training data.

[0068] Furthermore, the patch classifier generation unit 40B inputs the patch image 54 acquired by the patch image acquisition unit 40A to a region classifier, which will be described later, and acquires the image output from the region classifier. Then, if the patch classifier generation unit 40B finds that the classification result 58 defined for each pixel constituting the image includes a classification result 58 representing the operation mechanism 14, it should exclude the patch image 54 from the first training data of the patch classifier 60.

[0069] The patch classifier generation unit 40B excludes the patch image 54 in which the operating mechanism 14 is reflected from the patch images 54 used to generate the patch classifier 60, thereby enabling the generation of a patch classifier 60 capable of outputting highly accurate classification results 58 for each patch image 54.

[0070] Returning to Figure 2, we continue the explanation.

[0071] The region classification result image generation unit 40C uses the patch classifier 60 generated by the patch classifier generation unit 40B to generate a region classification result image in which a classification result 58 is defined for each element constituting the image 50.

[0072] An element is an element that constitutes the image 50 and is a region smaller than the patch image 54. More specifically, an element is a region consisting of one or more pixel groups and is a region smaller than the patch image 54. In this embodiment, a form in which one element corresponds to one pixel is described as an example. For this reason, in this embodiment, a form in which the region classification result image is an image in which a classification result 58 is defined for each pixel is described as an example.

[0073] Figures 6A and 6B are explanatory diagrams illustrating an example of the generation of a region classification result image 62.

[0074] As shown in Figure 6A, the region classification result image generation unit 40C slides the position of the image frame 56 of the patch image 54 within the training image 52 by a slide width smaller than the size of the patch image 54. The slide width only needs to be smaller than the size of the patch image 54. For example, the slide width is the width of one or more pixels.

[0075] The region classification result image generation unit 40C then inputs the images within the frame 56 at each slid position to the patch classifier 60. The region classification result image generation unit 40C generates a region classification result image 62 using the classification results 58 for each frame image output from the patch classifier 60.

[0076] The region classification result image generation unit 40C may input the image within the frame of the image frame 56 at the slid position to the patch classifier 60 each time the image frame 56 is slid, and obtain the classification result 58 of the image within the frame. The region classification result image generation unit 40C may then set a position closer to or further away from the previous slide position as the position for extracting the next image within the frame, according to the score (confidence level) of the classification result 58 of the image within the frame extracted at the previous slide position, and repeat the above process.

[0077] In detail, the classification result 58 of the framed image cropped at the previous slide position is equal to or greater than a threshold, or the difference between the highest score and the next highest score among the scores of multiple types of states output from the patch classifier 60 is equal to or greater than a predetermined difference. In this case, the region classification result image generation unit 40C should set the position where the next framed image is cropped to a position that is slid by a predetermined slide width greater than the previous slide position. These thresholds and predetermined differences can be predetermined. Furthermore, these thresholds and predetermined differences may be changed as appropriate by user instructions on the UI unit 32.

[0078] Furthermore, if the classification result 58 of the framed image cropped at the previous slide position is below a threshold, or if the difference between the highest score and the next highest score among the scores of multiple types of states output from the patch classifier 60 is less than a predetermined difference, the region classification result image generation unit 40C can set the next framed image cropping position to a position that is slid by a slide width smaller than a predetermined slide width relative to the previous slide position. These thresholds and predetermined differences can be predetermined. These thresholds and predetermined differences can also be changed as appropriate by user instructions on the UI unit 32.

[0079] Through the above processing, the region classification result image generation unit 40C obtains a classification result 58 for each image region within the image frame 56 at the slid position within the training image 52. The region classification result image generation unit 40C then identifies the classification result 58 for each of the multiple image frames 56 that overlap with the position of each element constituting the training image 52. Furthermore, the region classification result image generation unit 40C calculates the average score or the maximum score for each type of state using the scores of each of the multiple classification results 58 identified for each element. The region classification result image generation unit 40C then defines the type of state with the largest average score or maximum score calculated for each type of state as the classification result 58 for that element. The region classification result image generation unit 40C generates a region classification result image 62 in which the classification result 58 is defined for each element by performing these processes for each element constituting the training image 52.

[0080] Figure 6B is a schematic diagram of an example of a region classification result image 62. As shown in Figure 6B, the region classification result image generation unit 40C generates a region classification result image 62 in which one of the classification results 58A to classification result 58E is defined for each element.

[0081] Returning to Figure 2, we continue the explanation.

[0082] The identification unit 40D identifies a group of elements consisting of multiple elements from the region classification result image 62 generated by the region classification result image generation unit 40C, such that the pixel values ​​of the corresponding pixel positions in the region classification result image 62 and the training image 52 used to generate the region classification result image 62 are similar.

[0083] The position in the region classification result image 62 refers to the pixel position in the region classification result image 62. The group of elements with similar positions in the region classification result image 62 refers to a predetermined number of elements whose pixel positions are adjacent to each other in the region classification result image 62. This predetermined number can be set in advance. Furthermore, this predetermined number may be changed as appropriate by user instructions on the UI unit 32.

[0084] The training image 52 used to generate the region classification result image 62 is the training image 52 obtained by sliding the image frame 56 during the generation of the region classification result image 62. Similar pixel values ​​at corresponding pixel positions in the training image 52 mean that, among the group of elements with similar positions in the region classification result image 62, the difference in pixel values ​​between the included elements is within a predetermined range. For example, luminance values, color values, etc., can be used as pixel values. This predetermined range can be set in advance. Furthermore, this predetermined range may be changed as appropriate by user instructions on the UI unit 32.

[0085] The correction unit 40E corrects the classification result 58 defined for each of the multiple elements included in the element group identified by the identification unit 40D to the maximum number of classification results 58 defined for the elements included in the element group. Specifically, for example, let's assume that the element group identified by the identification unit 40D is an element group consisting of 10 elements. Let's assume that of the elements included in the element group consisting of 10 elements, 7 elements are defined with classification result 58 "mixed" and 3 elements are defined with classification result 58 "structure". In this case, the correction unit 40E corrects the classification result 58 defined for each of the elements constituting the element group to the maximum number of classification results 58, which is "mixed".

[0086] Then, the correction unit 40E performs the above correction process for each group of elements identified by the identification unit 40D, thereby generating a corrected region classification result image by correcting the region classification result image 62.

[0087] Next, we will explain the region classifier generation unit 40F.

[0088] The region classifier generation unit 40F uses training data consisting of pairs of training images 52 and region classification result images 62 to generate a region classifier that takes image 50 as input and outputs region classification result images 62.

[0089] Figure 7 is an explanatory diagram illustrating an example of the generation of a region classifier 64.

[0090] The region classifier generation unit 40F uses pairs of training images 52 and region classification result images 62 as second training data, and trains the region classifier 64 using machine learning with multiple second training data. Deep learning or similar methods can be used for this machine learning. For example, semantic segmentation, a deep learning algorithm, can be used for machine learning. Through this training process, the region classifier generation unit 40F generates a region classifier 64 that takes image 50 as input and outputs region classification result 68, which is the region classification result image 62.

[0091] As described above, the region classification result image 62 is an image in which a classification result 58 is defined for each element, that is, for each pixel. Therefore, the region classifier generation unit 40F can generate a region classifier 64 through learning, thereby generating a region classifier 64 that obtains a region classification result 68 from the image 50, in which a classification result 58 is defined for each pixel contained in the image 50.

[0092] Alternatively, the region classifier generation unit 40F may train the region classifier 64 using machine learning with a corrected region classification result image 66 obtained by correcting the region classification result image 62, instead of the region classification result image 62. In this case, the region classifier generation unit 40F uses pairs of the training image 52 and the corrected region classification result image 66 corrected by the identification unit 40D and the correction unit 40E as second training data, and trains the region classifier 64 using machine learning with multiple second training data. Through this training process, the region classifier generation unit 40F generates a region classifier 64 that takes image 50 as input and outputs a region classification result 68 which is the corrected region classification result image 66.

[0093] By training the region classifier 64 using machine learning with the corrected region classification result image 66, the region classifier generation unit 40F can generate a region classifier 64 that outputs a region classification result 68 that is closer to the correct answer or defines the correct answer as the classification result 58 of the elements that constitute the boundary between regions with different states contained in the image 50. In other words, the region classifier generation unit 40F can generate a region classifier 64 with higher accuracy.

[0094] Returning to Figure 2, the explanation continues. The control unit 40 stores the region classifier 64 generated by the region classifier generation unit 40F in the storage unit 30. The control unit 40 may also transmit the region classifier 64 generated by the region classifier generation unit 40F to an external information processing device via the communication unit 34. The information processing device that receives the region classifier 64 can easily obtain the region classification result 68 of the image 50 by inputting the image 50 to the received region classifier 64.

[0095] Furthermore, the patch classifier generation unit 40B and the region classifier generation unit 40F may repeatedly perform the learning process for generating the patch classifier 60 and the learning process for generating the region classifier 64, respectively. For example, the patch image acquisition unit 40A acquires a new patch image 54 from a new training image 52. Then, the patch classifier generation unit 40B can perform the above process to train the patch classifier 60 using the patch image 54 newly acquired by the patch image acquisition unit 40A. Also, the region classifier generation unit 40F can perform the above learning process to generate a region classifier 64B each time a new patch classifier 60 is trained.

[0096] By repeatedly performing the learning process, the patch classifier generation unit 40B and the region classifier generation unit 40F can generate a patch classifier 60 with higher accuracy.

[0097] During repeated execution of the learning process, it is preferable for the patch image acquisition unit 40A to preferentially acquire a patch image 54 that is different from the already acquired patch image 54 as a new patch image 54.

[0098] For example, the patch image acquisition unit 40A preferentially acquires other patch images 54 whose image differences are greater than or equal to a threshold value from the patch images 54 that have already been acquired, i.e., used for training, as new patch images 54. The image differences can be, for example, the variation in the pixel values ​​that make up the image, the difference in the average value of the pixel values, the difference in the maximum or minimum value of the pixel values, the difference in the date and time of capture, etc. Alternatively, for example, the patch image acquisition unit 40A may input a new image 50 to be acquired into the generated patch classifier 60 and preferentially acquire patch images 54 whose neuronal firing patterns in the patch classifier 60 are different from those of the already acquired patch images 54 as new patch images 54.

[0099] The patch image acquisition unit 40A prioritizes acquiring patch images 54 that are different from the already acquired patch images 54 as new patch images 54 to be used for training. By repeating the training process, a region classifier 64 can be generated that can output highly accurate region classification results 68 from images 50 that capture objects 20 in a wider variety of states.

[0100] Next, the viewpoint transformation image generation unit 40G will be described. The classifier generation device 10 may be configured to include the viewpoint transformation image generation unit 40G.

[0101] The viewpoint transformation image generation unit 40G inputs the image 50 to the region classifier 64, and converts the region classification result 68, which is either a region classification result image 62 or a corrected region classification result image 66 obtained as an output from the region classifier 64, into a viewpoint transformation image.

[0102] In detail, the viewpoint transformation image generation unit 40G converts the region classification result 68 into a viewpoint transformation image viewed from an arbitrary viewpoint direction in three-dimensional space. The arbitrary viewpoint direction can be set in advance. Furthermore, the arbitrary viewpoint direction may be changed as appropriate by user instructions on the UI unit 32. The viewpoint transformation image generation unit 40G acquires three-dimensional information of the real space in which the object 20 exists from another information processing device or storage unit 30, etc. Then, using the acquired three-dimensional information and the region classification result 68, the viewpoint transformation image generation unit 40G converts the region classification result 68 into a viewpoint transformation image viewed from an arbitrary viewpoint direction using a known viewpoint transformation process.

[0103] The viewpoint transformation image generation unit 40G outputs the generated viewpoint transformation image to the UI unit 32. Alternatively, the viewpoint transformation image generation unit 40G may transmit the viewpoint transformation image to an external information processing device via the communication unit 34.

[0104] The viewpoint transformation image generation unit 40G outputs the viewpoint transformation image to the UI unit 32, making it possible to provide the user with the region classification result 68 in a display format that allows it to be viewed from any viewpoint direction the user chooses.

[0105] Next, an example of the information processing flow performed by the classifier generation device 10 of this embodiment will be described.

[0106] Figure 8 is a flowchart showing an example of the information processing flow performed by the classifier generation device 10 of this embodiment.

[0107] The patch image acquisition unit 40A acquires a partial region of the training image 52 as a patch image 54 (step S100).

[0108] The patch classifier generation unit 40B uses the patch image 54 acquired in step S100 and the classification result 58 taught for the patch image 54 as first training data, and generates a patch classifier 60 by training the patch classifier 60 using machine learning with multiple first training data (step S102).

[0109] The region classification result image generation unit 40C generates a region classification result image 62 in which a classification result 58 is defined for each element constituting the image 50, using the patch classifier 60 generated in step S102 (step S104).

[0110] The identification unit 40D identifies a group of elements from the region classification result image 62 generated in step S104, in which the pixel values ​​of the corresponding pixel positions in the training image 52 used to generate the region classification result image 62 are similar (step S108).

[0111] The correction unit 40E performs correction processing for each group of elements identified in step S106 to generate a corrected region classification result image 66 by correcting the region classification result image 62 (step S108).

[0112] The region classifier generation unit 40F generates a region classifier 64 (step S110). The region classifier generation unit 40F uses pairs of training images 52 and corrected region classification result images 66 as second training data, and generates the region classifier 64 by training it using machine learning with multiple second training data. Then, this routine ends.

[0113] As described above, the classifier generation device 10 of this embodiment comprises a patch image acquisition unit 40A, a patch classifier generation unit 40B, a region classification result image generation unit 40C, and a region classifier generation unit 40F. The patch image acquisition unit 40A acquires a partial region consisting of multiple pixels in a training image 52, which is an image 50 of a state-variable object 20, as a patch image 54. The patch classifier generation unit 40B takes the patch image 54 as input and generates a patch classifier 60 that outputs a classification result 58 of the state of the object 20 captured in the patch image 54. The region classification result image generation unit 40C uses the patch classifier 60 to generate a region classification result image 62 in which a classification result 58 is defined for each element that constitutes the image 50 and is a region smaller than the patch image 54. The region classifier generation unit 40F uses second training data consisting of pairs of training images 52 and region classification result images 62 to generate a region classifier 64 that takes image 50 as input and outputs region classification result 68, which is the region classification result image 62.

[0114] In conventional methods, a region classifier is generated by manually teaching the correct class to each pixel in image 50, which places a heavy workload on the user performing the teaching task. Furthermore, in conventional methods, if an object 20 containing multiple regions representing different classification results 58 is captured in image 50, it can be difficult for the user to teach the correct class to pixels near the boundaries of these regions. In addition, conventional methods that teach the types of objects captured in image 50 require a large number of images 50 containing various types of objects to be prepared in order to generate a region classifier. Furthermore, conventional methods that teach regions of objects captured in image 50 by enclosing them in rectangular frames require images 50 with clear boundaries between multiple regions. In addition, conventional methods that teach the correct class to some pixels in image 50 require a large number of images containing various types of objects at various positions to be prepared in order to generate a region classifier. Thus, it was difficult to easily generate a region classifier using conventional methods.

[0115] On the other hand, in the classifier generation device 10 of this embodiment, a patch classifier 60 is generated using a patch image 54 which is a partial region consisting of multiple pixels in the training image 52, and a region classifier 64 is generated using the generated patch classifier 60 to output a region classification result 68 which defines a classification result 58 for each element from the image 50.

[0116] Thus, in the classifier generation device 10 of this embodiment, region classifiers 64 are generated using patch images 54, which are partial regions consisting of multiple pixels, rather than pixel regions for each individual pixel. For this reason, the classifier generation device 10 of this embodiment can reduce the workload of the teaching process and improve teaching accuracy. Furthermore, the classifier generation device 10 of this embodiment can generate region classifiers 64 without requiring a large number of training images 52.

[0117] Therefore, the classifier generation device 10 of this embodiment can easily generate a region classifier 64.

[0118] (Second embodiment) This embodiment describes a method for generating the patch classifier 60 in a way different from the above embodiment. In this embodiment, the same reference numerals are used for components that are the same as in the above embodiment, and detailed descriptions are omitted.

[0119] Figure 2 is a block diagram showing an example of the functional configuration of the generation system 1B of this embodiment. The generation system 1B is the same as the generation system 1 of the above embodiment, except that it includes a classifier generation device 10B instead of a classifier generation device 10. The classifier generation device 10B is the same as the classifier generation device 10 of the above embodiment, except that it includes a patch classifier generation unit 41B instead of a patch classifier generation unit 40B.

[0120] The patch classifier generation unit 41B, similar to the patch classifier generation unit 40B in the above embodiment, takes the patch image 54 as input and generates a patch classifier 60 that outputs a classification result 58 of the state of the object 20 captured in the patch image 54.

[0121] In this embodiment, the patch classifier generation unit 41B generates a patch classifier 60 from the patch image 54, outputting a classification result 58 of the state of the object 20 captured in the patch image 54 for each divided region obtained by dividing the patch image 54 into multiple regions.

[0122] Figure 9A is an explanatory diagram illustrating an example of the generation of a patch classifier 61A by the patch classifier generation unit 41B. Patch classifier 61A is an example of patch classifier 60. Figure 9A shows patch image 54F as an example. Patch image 54F is an example of patch image 54.

[0123] For example, the patch classifier generation unit 41B uses elements as divided regions P obtained by dividing the patch image 54 into multiple regions. As described in the above embodiment, an element is a region smaller than the patch image 54 and consisting of one or more groups of pixels. In this embodiment as well, a configuration in which one element corresponds to one pixel will be described as an example.

[0124] The patch classifier generation unit 41B then takes the patch image 54 as input and trains a patch classifier 61A that outputs classification results 58 for each divided region P obtained by dividing the patch image 54 into elements P1, i.e., pixels. Figure 9A shows an example of a patch classifier 61A that outputs classification results 58F1 to 58Fn as classification results 58 for each element P1 contained in the patch image 54F. n represents the number of elements that make up the patch image 54F.

[0125] The patch classifier generation unit 41B can generate a patch classifier 61A that outputs a classification result 58 for each divided region P obtained by dividing the patch image 54 into elements P1, using a method such as Grad-CAM (Gradient-weighted Class Activation Mapping) from the patch image 54.

[0126] In detail, for example, the patch classifier generation unit 40B uses a method such as Grad-CAM to teach each element P1 of the patch image 54 a classification result 58.

[0127] The patch classifier generation unit 41B then uses pairs of patch images 54 and classification results 58 taught to each element P1 of the patch image 54 as first training data, and trains the patch classifier 61B using machine learning with multiple first training data. Through this training process, the patch classifier generation unit 41B generates a patch classifier 61A that takes the patch image 54 as input and outputs classification results 58 for each element P1, which are divided regions P.

[0128] The region classification result image generation unit 40C can generate a more accurate region classification result image 62 by using the patch classifier 61A generated by the patch classifier generation unit 41B in the same manner as in the above embodiment. Then, by using the more accurate region classification result image 62 or the corrected region classification result image 66 obtained by correcting the region classification result image 62, the region classifier generation unit 40F can generate a more accurate region classifier 64.

[0129] The patch classifier generation unit 41B only needs to generate a patch classifier 60 from the patch image 54 that outputs classification results 58 for each divided region P obtained by dividing the patch image 54 into multiple regions. For this reason, the divided region P is not limited to element P1.

[0130] For example, the patch classifier generation unit 41B may generate a patch classifier 60 that outputs a classification result 58 for each divided region P, which is divided into the structural region of structures 22 other than the object 20 captured in the image 50 and the object region of the object 20.

[0131] Figure 9B is an explanatory diagram illustrating an example of the generation of patch classifier 61B by the patch classifier generation unit 41B. Patch classifier 61B is an example of patch classifier 60. Figure 9B shows patch image 54G as an example. Patch image 54G is an example of patch image 54.

[0132] For example, the patch classifier generation unit 41B uses two separate divided regions P: the object region, which is the area of ​​the object 20 captured in the patch image 54, and the structure region, which is the area of ​​the structure 22 captured in the patch image 54, excluding the object 20. The patch classifier generation unit 41B then generates a patch classifier 61B that outputs a classification result 58 for each divided region P obtained by dividing the patch image 54 into the object region P2A and the structure region P2B.

[0133] In this case, the patch classifier generation unit 41B calculates the difference in pixel values ​​between the training image 52, which is the original image from which the patch image 54 was extracted, and the image 50, which was taken in a state where the object 20 is not captured within the field of view of the training image 52. Based on the calculation results, the patch classifier generation unit 41B identifies the region in the training image 52 where the difference in pixels is greater than or equal to a threshold as the object region of the object 20. The patch classifier generation unit 41B also identifies the region in the training image 52 other than the identified object region as the structure region. The patch classifier generation unit 41B then identifies whether the position of each pixel in the patch image 54 in the training image 52 corresponds to the object region P2A or the structure region P2B. Through this identification process, the patch classifier generation unit 41B teaches each pixel in the patch image 54 either the object region P2A or the structure region P2B.

[0134] The patch classifier generation unit 41B then uses pairs of patch images 54 and classification results 58 taught to each divided region P for the patch images 54 as first training data, and trains the patch classifier 61B using machine learning with multiple first training data. Through this training process, the patch classifier generation unit 41B generates a patch classifier 61B that takes patch images 54 as input and outputs classification results 58 for each divided region P.

[0135] The region classification result image generation unit 40C can generate a more accurate region classification result image 62 by using the patch classifier 61B generated by the patch classifier generation unit 41B in the same manner as in the above embodiment. Then, by using the more accurate region classification result image 62 or the corrected region classification result image 66 obtained by correcting the region classification result image 62, the region classifier generation unit 40F can generate a more accurate region classifier 64.

[0136] The patch classifier generation unit 41B may also generate a patch classifier 60 that outputs classification results 58 for each divided region P obtained by dividing the patch image 54 into multiple regions, from the patch image 54 and the division results obtained by dividing the patch image 54 into an object region P2A and a structure region P2B.

[0137] Figure 9C is an explanatory diagram illustrating an example of the generation of a patch classifier 61C by the patch classifier generation unit 41B. Patch classifier 61C is an example of patch classifier 60. Figure 9C shows patch image 54F as an example. Patch image 54F is an example of patch image 54.

[0138] For example, the patch classifier generation unit 41B obtains a patch image 54 and a region classification result 68 obtained by dividing the patch image 54 into an object region P2A and a structure region P2B. Figure 9C shows the region classification result 68F as an example of the region classification result 68 of the patch image 54F. The patch classifier generation unit 41B can calculate the region classification result 68 of the patch image 54F by performing the same processing as described above using the difference in pixel values ​​for each pixel between the training image 52 and the image 50 taken in a state where the object 20 is not captured within the shooting angle of the training image 52.

[0139] Furthermore, the patch classifier generation unit 41B uses a method such as Grad-CAM to teach the classification results 58 to each element P1 of the patch image 54 from the patch image 54 and the region classification results 68.

[0140] The patch classifier generation unit 41B then uses pairs of patch images 54 and region classification results 68, and classification results 58 taught to each element P1 of the patch image 54, as first training data, and trains the patch classifier 61C using machine learning with multiple first training data. Through this training process, the patch classifier generation unit 41B generates a patch classifier 61C that takes patch images 54 and region classification results 68 as input and outputs classification results 58 for each element P1, which is a divided region P. Therefore, the patch classifier generation unit 41B can generate a patch classifier 61C in which elements P1 constituting the object region P2A selectively contribute to the classification results 58.

[0141] The region classification result image generation unit 40C generates a region classification result image 62 in the same manner as in the above embodiment using the patch classifier 61C generated by the patch classifier generation unit 41B, thereby generating a region classification result image 62 with higher accuracy. Then, by using the region classification result image 62 with higher accuracy or the corrected region classification result image 66 obtained by correcting the region classification result image 62, the region classifier generation unit 40F can generate a region classifier 64 with higher accuracy.

[0142] The information processing flow performed by the control unit 41 in this embodiment is the same as in the above embodiment, except that the patch classifier generation unit 41B performs the process of generating the patch classifier 60 instead of the patch classifier generation unit 40B.

[0143] As described above, in the classifier generation device 10B of this embodiment, the patch classifier generation unit 41B generates patch classifiers 60 (61A, 61B, 61C) from the patch image 54, outputting classification results 58 of the state of the object 20 captured in the patch image 54 for each divided region P obtained by dividing the patch image 54 into multiple regions.

[0144] Therefore, in addition to the effects of the above embodiment, the classifier generation device 10B of this embodiment can generate an even more accurate region classifier 64.

[0145] (Variation 1) The region classifier 64 generated by the classifier generation device 10 or classifier generation device 10B of the above embodiment can be suitably applied to an image classification device that outputs region classification results from any image. The image classification device can be any information processing device that derives region classification results from an image.

[0146] For example, the image classification device can use the region classifier 64 generated by the classifier generation device 10 or the classifier generation device 10B of the above embodiment, input a desired image to the region classifier 64, and derive the region classification result of the image as the output from the region classifier 64.

[0147] Next, the hardware configuration of the classifier generation device 10 and the classifier generation device 10B of the above embodiment will be described.

[0148] Figure 10 is a hardware configuration diagram of an example of the classifier generation device 10 and classifier generation device 10B according to the above embodiment.

[0149] The classifier generation device 10 and the classifier generation device 10B of the above embodiment include a control device such as a CPU 91, a storage device such as a ROM (Read Only Memory) 92 and a RAM (Random Access Memory) 93, a communication I / F 94 that connects to a network for communication, and a bus 95 that connects each part.

[0150] The programs executed by the classifier generation device 10 and the classifier generation device 10B in the above embodiment are provided pre-loaded into a ROM 92 or the like.

[0151] The programs executed by the classifier generation device 10 and the classifier generation device 10B in the above embodiment may be configured to be provided as a computer program product by recording them in an installable or executable file format onto a computer-readable recording medium such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), or a DVD (Digital Versatile Disk).

[0152] Furthermore, the programs executed by the classifier generation device 10 and the classifier generation device 10B of the above embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Alternatively, the programs executed by the classifier generation device 10 and the classifier generation device 10B of the above embodiment may be provided or distributed via a network such as the Internet.

[0153] The programs executed by the classifier generation device 10 and the classifier generation device 10B in the above embodiment can cause a computer to function as a component of the classifier generation device 10 in this embodiment. This computer can read the program from a computer-readable storage medium onto the main memory and execute it using the CPU 91. The image classification device may also have the same hardware configuration as the classifier generation device 10 and the classifier generation device 10B.

[0154] Although embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. The novel embodiments described above can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0155] 10, 10B classifier generator 14 Operating mechanism 20 Objects 22 Structures 40A Patch Image Acquisition Unit 40B, 41B Patch Classifier Generation Unit 40C Region classification result image generation unit 40D Specific part 40E Correction Unit 40F Region classifier generator 40G Viewpoint Transformation Image Generation Unit

Claims

1. A patch image acquisition unit that acquires a patch image from a training image which is an image of an object whose state is variable, and A patch classifier generation unit takes the aforementioned patch image as input and generates a patch classifier that outputs a classification result of the state of the object captured in the patch image, A region classification result image generation unit generates a region classification result image that defines the classification result for each element that constitutes the image and is a region smaller than the patch image, using the patch classifier; A region classifier generation unit generates a region classifier that takes the aforementioned images as input and outputs the region classification result, which is the region classification result image, using training data consisting of pairs of the aforementioned training images and the aforementioned region classification result images. A classifier generation device equipped with the following features.

2. The region classification result image generation unit is: The position of the image frame of the patch image is slid within the training image by a slide width smaller than the size of the patch image, the images within the image frame at each slid position are input to the patch classifier, and the region classification result image is generated using the classification result for each of the images within the frame output from the patch classifier. The classifier generating device according to claim 1.

3. A selection unit identifies a group of elements consisting of multiple elements whose position in the region classification result image and the pixel value of the corresponding pixel position in the training image are similar, A correction unit generates a corrected region classification result image by correcting the classification result defined for each of the multiple elements included in the element group to the maximum number of classification results defined for the elements included in the element group, for each of the identified element groups. Equipped with, The region classifier generation unit is, Using the training data consisting of pairs of the aforementioned training image and the corrected region classification result image obtained by correcting the region classification result image, a region classifier is generated that takes the image as input and outputs the region classification result which is the corrected region classification result image. The classifier generating device according to claim 1.

4. The aforementioned patch classifier generation unit is: From the aforementioned patch image, a patch classifier is generated that outputs the classification result of the state of the object captured in the patch image for each divided region obtained by dividing the patch image into multiple regions. The classifier generating device according to claim 1.

5. The aforementioned patch classifier generation unit is: From the patch image, a patch classifier is generated that outputs the classification result for each of the divided regions obtained by dividing the patch image into elements. The classifier generating device according to claim 4.

6. The aforementioned patch classifier generation unit is: A patch classifier is generated which outputs the classification result for each divided region obtained by dividing the patch image into the structural region of the structure captured in the image and the object region of the object captured in the image. The classifier generating device according to claim 4.

7. The aforementioned patch classifier generation unit is: A patch classifier is generated that outputs the classification result for each of the divided regions obtained by dividing the patch image into multiple regions from the patch image and the result of dividing the patch image into the structural region of the structure captured in the patch image and the object region of the object. The classifier generating device according to claim 4.

8. The aforementioned patch classifier generation unit is: The patch classifier is generated using the patch image in which only one type of the object in the aforementioned state is captured from among the acquired patch images. The classifier generating device according to claim 1.

9. The aforementioned patch classifier generation unit is: The movement information of the operating mechanism for performing operations on the aforementioned object is acquired. Based on the movement information of the operating mechanism at the time of capturing the training image, the classification result is taught to the patch image extracted from the training image. A patch classifier is generated that takes the aforementioned patch image as input and outputs the taught classification result. The classifier generating device according to claim 1.

10. The aforementioned patch classifier generation unit is: Of the acquired patch images, any patch image that shows an operating mechanism for performing an operation on the object is excluded from the patch images used to generate the patch classifier. The classifier generating device according to claim 1.

11. The aforementioned patch image acquisition unit, Prioritizing the acquisition of a patch image different from the previously acquired patch image as the new patch image, The classifier generating device according to claim 1.

12. The region classification result image is A viewpoint transformation image generation unit that converts the image into a viewpoint transformation image viewed from any viewpoint direction in three-dimensional space. A classifier generating device according to claim 1, comprising:

13. The steps include: obtaining a patch image from a training image which is an image of an object whose state is variable, and The steps include: generating a patch classifier that takes the aforementioned patch image as input and outputs a classification result of the state of the object captured in the patch image; Using the patch classifier, a region classification result image is generated in which the classification result is defined for each element that constitutes the image and is a region smaller than the patch image. A step of generating a region classifier that takes the aforementioned images as input and outputs the region classification result, which is the region classification result image, using training data consisting of pairs of the aforementioned training images and the aforementioned region classification result images. A method for generating a classifier that includes this.

14. The steps include: obtaining a patch image from a training image which is an image of an object whose state is variable, and The steps include: generating a patch classifier that takes the aforementioned patch image as input and outputs a classification result of the state of the object captured in the patch image; Using the patch classifier, a region classification result image is generated in which the classification result is defined for each element that constitutes the image and is a region smaller than the patch image. A step of generating a region classifier that takes the aforementioned images as input and outputs the region classification result, which is the region classification result image, using training data consisting of pairs of the aforementioned training images and the aforementioned region classification result images. A classifier generation program to run on a computer.

15. An image classification device that classifies images using the region classifier generated by the classifier generation device described in claim 1.

Images