Mechanical learning support device, mechanical learning support method, and mechanical learning support program

Abstract

To generate an image for accurately and efficiently conducting additional learning of an identification model.SOLUTION: In a mechanical learning device 101 for generating images for additional learning, an identification model has been learned according to a plurality of basic images acquired under a first image acquisition condition. The mechanical learning device comprises: an image input part 201 for receiving an additional image acquired under a second image acquisition condition; an image state feature amount calculation part 202 for calculating a second image state feature amount of the additional image; an image state feature amount control part 204 for generating a third image state feature amount between a first image state feature amount and the second image state feature amount of the basic image; a form feature amount calculation part 203 for calculating a first form feature amount based on a form of a subject of the additional image; an image generation part 205 for generating candidate images, based on the third image state feature amount and the first form feature amount; and a learning part 207 for determining at least one of the candidate images as the image for additional learning.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a technique for supporting additional learning of a discriminative model that classifies images into classes. [Background technology]

[0002] A technique is known that identifies the class of an image using a discrimination model generated by machine learning. For example, when machine learning is performed on a discrimination model using an image acquired at a certain facility, if this discrimination model is used to perform discrimination processing on an image acquired at another facility, the acquisition conditions of the images taken at the certain facility and the other facilities are different, and therefore, the discrimination accuracy may not be as good as the discrimination accuracy for the image acquired at the certain facility.

[0003] In response to this, a technology has been disclosed in which an identification model is generated through machine learning using images taken inside the garbage pits of multiple facilities and training data labeled with the types of waste in the images, and then images taken inside the garbage pit of a second facility different from the multiple facilities and training data labeled with the types of waste in the images are additionally trained to make the identification model compatible with the second facility (see, for example, Patent Document 1).

[0004] Another technique disclosed is one in which correction parameters used to convert an image are generated based on the results of determining how an image shown by image data appears, the image is converted using the correction parameters, the converted image is subjected to machine learning inference, and the inference results are associated with the image data to generate training data, which is then used to perform additional learning of an inference model (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2023-12094 A [Patent Document 2] JP 2020-160804 A Summary of the Invention [Problem to be solved by the invention]

[0006] As described above, in the technology disclosed in Patent Document 1, additional learning of the classification model is performed using images captured at other facilities. In this case, for example, if the number of images captured at other facilities is small, overlearning may occur, and good detection accuracy may not be obtained. On the other hand, if the number of images captured at other facilities is large, the labor required to associate annotations with the captured images may increase, or the classification accuracy for images of facilities that have already been learned may decrease. The technology disclosed in Patent Document 2 also has similar problems.

[0007] The present invention has been made in consideration of the above circumstances, and has an object to provide a technique for generating an image for performing additional learning of a discriminative model accurately and efficiently. [Means for solving the problem]

[0008] In order to achieve the above object, a machine learning assistance device according to one aspect is a machine learning assistance device that generates additional training images for additional training of a discriminative model that identifies a class into which an image is classified, the discriminative model being trained using a plurality of base images acquired under a first image acquisition condition, and the discriminative model is trained using a plurality of base images acquired under a first image acquisition condition, and the device is equipped with: a reception unit that receives an additional image acquired under a second image acquisition condition to be discriminated by the discriminative model; an image state feature calculation unit that calculates a second image state feature based on an image state of the additional image; an image state feature control unit that generates one or more third image state features that are features between the first image state feature and the second image state feature based on the image state of the base image; a morphological feature calculation unit that calculates a first morphological feature based on a morphology of a subject of the additional image; an image generation unit that generates one or more candidate images based on the one or more third image state features and the first morphological feature; and an additional training image determination unit that determines at least one of the candidate images as an additional training image, the image state including at least one state of an image's brightness, contrast, color (hue and / or saturation), blur, noise amount, and resolution. Effect of the Invention

[0009] According to the present invention, it is possible to generate an image for performing additional learning of a discriminative model accurately and efficiently. [Brief description of the drawings]

[0010] [Figure 1A] FIG. 1A is a diagram illustrating a first example of image classification using a discriminative model according to a conventional example. [Figure 1B] FIG. 1B is a diagram illustrating a second example of image classification using a classification model according to a conventional example. [Figure 1C] FIG. 1C is a diagram illustrating a third example of image classification using a classification model according to a conventional example. [Figure 1D] FIG. 1D is a diagram conceptually illustrating discrimination using a discriminative model according to one embodiment. [Diagram 2] FIG. 2 is a hardware configuration diagram of a machine learning system according to one embodiment. [Diagram 3] FIG. 3 is a functional configuration diagram of a machine learning device according to an embodiment. [Figure 4] FIG. 4 is a diagram illustrating pre-learning for an image state feature amount calculation unit, a morphological feature amount calculation unit, and an image generation unit according to an embodiment. [Diagram 5] FIG. 5 is a diagram illustrating the additional learning image determination process according to an embodiment. [Figure 6A] FIG. 6A illustrates images when the contrast of the images is affected by changing the deformation coefficient according to one embodiment. [Figure 6B] FIG. 6B illustrates images when the luminance values ​​of the images are affected by changing the deformation coefficient according to an embodiment. [Figure 6C] FIG. 6C illustrates images in which the degree of blurring of an image is affected by changing the deformation coefficient according to an embodiment. [Figure 7] FIG. 7 is a diagram showing the relationship between an image obtained by changing the deformation coefficient according to an embodiment and a discrimination surface. [Figure 8] FIG. 8 is a diagram showing a result display screen according to one embodiment. [Figure 9] FIG. 9 is a diagram showing an additional learning image display screen according to an embodiment. [Figure 10] FIG. 10 is a diagram showing a correct answer setting screen according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The following embodiments will be described with reference to the drawings. Note that the following embodiments do not limit the scope of the invention, and not all of the elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention.

[0012] In describing this embodiment, image classification using a conventional classification model will first be described, and then an overview of the classification model according to this embodiment will be provided.

[0013] Fig. 1A is a diagram for explaining a first example of image classification using a classification model according to a conventional example. In Fig. 1A, the horizontal direction is divided by the facilities where the images were acquired (facilities A and B: an example of the image acquisition conditions under which the images were acquired), and the vertical direction is divided by the classes (classes a and b) to which the images belong, and corresponding images are arranged in the regions. Note that similar displays are also used in Figs. 1B to 1D described later.

[0014] In FIG. 1A, an image 301 belonging to class a captured in facility A is represented by a white rectangle, and an image 302 belonging to class b captured in facility A is represented by a white circle.

[0015] 1A shows an example of a case where machine learning is performed on a discrimination model using only images captured in facility A. In this case, discrimination plane 303, which is a boundary for discriminating whether an image belongs to class a or class b in the discrimination model, can be conceptually represented as shown in FIG.

[0016] In this way, if machine learning is performed using only images taken at facility A, images taken at facility B are not taken into account, and there is a risk that the image class for images taken at facility B cannot be properly identified.

[0017] Next, a case where additional machine learning (additional learning) is performed including a large number of images captured at facility B will be described.

[0018] FIG. 1B is a diagram for explaining a second example of image classification using a discrimination model according to a conventional example. FIG. 1B shows an example in which additional machine learning (additional learning) is performed including a large number of images captured in facility B. In FIG. 1B, an image 304 captured in facility B belonging to class a is represented by a black rectangle, and an image 305 captured in facility B belonging to class b is represented by a black circle. In this case, in the discrimination model, a discrimination plane 306 that is a boundary for discriminating whether an image is class a or class b can be conceptually represented as shown in FIG. 1B.

[0019] As shown in FIG. 1B, preparing a large number of images captured at facility B is time-consuming, and labeling each image with a class for additional learning is also time-consuming.

[0020] Furthermore, if additional learning is performed using a large number of images taken at facility B, according to classification surface 306, the classification accuracy for images taken at facility B will improve, but this will affect the classification surface, and there is a risk that the classification accuracy for images taken at facility A will decrease.

[0021] Next, a case will be described where additional learning is performed including a small number of images captured at facility B after the machine learning shown in FIG. 1A is performed.

[0022] Fig. 1C is a diagram for explaining a third example of image classification using a discrimination model according to a conventional example. Fig. 1C shows an example of a case where additional learning is performed including a small number of images captured in facility B. In Fig. 1C, an image 307 captured in facility B belonging to class a is represented by a black rectangle, and an image 308 captured in facility B belonging to class b is represented by a black circle. In this case, in the discrimination model, a discrimination plane 309 that is a boundary for discriminating whether an image is class a or class b can be conceptually represented as shown in Fig. 1C.

[0023] In this way, if additional learning is performed using only a small number of images captured at facility B, the images may be acquired in a significantly different state from the images captured at facility A, resulting in learning specialized to the images used for learning (overlearning), which may result in a discrimination surface 309 that is significantly different from the discrimination surface 303 before learning. For this reason, images captured at facility B may not be able to be generally classified, and there is a risk that the classification accuracy may not be improved.

[0024] Next, an overview of the identification model according to this embodiment will be described.

[0025] FIG. 1D is a diagram conceptually illustrating discrimination using a discriminative model according to one embodiment.

[0026] Fig. 1D shows an example of a case where an image close to the discrimination surface is generated by bringing a small number of images captured at facility B closer to the image acquisition state (e.g., color, brightness, etc.) of the image captured at facility A, and additional learning is performed including the generated images. In Fig. 1D, an image 307 captured at facility B belonging to class a is represented by a black rectangle, and an image 310 close to discrimination surface 303 generated based on this image is represented by a gray rectangle. In this case, discrimination surface 311, which is the boundary for discriminating whether an image is class a or class b in the discrimination model, can be conceptually represented as shown in Fig. 1D.

[0027] In this way, by performing additional learning using an image close to the discrimination surface 303, it is possible to obtain a discrimination surface 311 that is a gradually enlarged version of the discrimination surface 303. This makes it possible to reduce the occurrence of overlearning as shown in Fig. 1C, and to generally discriminate images captured in facility B, thereby improving the discrimination accuracy.

[0028] Next, a machine learning system according to an embodiment will be described.

[0029] FIG. 2 is a hardware configuration diagram of a machine learning system according to one embodiment.

[0030] The machine learning system 100 includes a machine learning device 101 as an example of a machine learning support device, an input device 120, and a display device 121.

[0031] The input device 120 is, for example, a mouse, a keyboard, etc., and accepts information input by a user. The display device 121 is, for example, a display, and displays and outputs a user interface (screen) including various types of information.

[0032] The machine learning device 101 is configured by a computer such as a PC (Personal Computer) or a general-purpose server, and includes an interface unit 110, a processor 111, a memory 112, a storage device 113, and a bus 114. The interface unit 110, the processor 111, the memory 112, and the storage device 113 are connected via the bus 114 so as to be able to communicate with each other.

[0033] The interface unit 110 includes an interface such as a wired LAN card or a wireless LAN card, and communicates with other devices via a network. The interface unit 110 can receive various images via the network. The interface unit 110 also receives information from the input device 120 and passes it to the processor 111. The interface unit 110 also causes the display device 121 to display various screens in accordance with instructions from the processor 111.

[0034] The processor 111 executes various processes according to programs stored in the memory 112 and / or the storage device 113 .

[0035] The memory 112 is, for example, a RAM (RANDOM ACCESS MEMORY), and stores the programs executed by the processor 111 and other necessary information.

[0036] The storage device 113 is, for example, a hard disk or a flash memory, and stores a program (machine learning support program) executed by the processor 111 and data used by the processor 111. In this embodiment, the storage device 113 stores, for example, teacher data (a set of an image and a class name of the image: basic learning data) and evaluation data (a set of an image and a class name of the image: evaluation images) for a plurality of images acquired at facility A, teacher data (a set of an image and a class name of the image: additional learning data) for images acquired at facility B, and generated additional learning images.

[0037] Next, the functional configuration of the machine learning device 101 according to an embodiment will be described.

[0038] FIG. 3 is a functional configuration diagram of a machine learning device according to an embodiment.

[0039] The machine learning device 101 includes an image input unit 201 as an example of a receiving unit, an image state feature amount calculation unit 202, a morphological feature amount calculation unit 203, an image state feature amount control unit 204, an image generation unit 205, a classification unit 206, a learning unit 207, and an output unit 208. Here, each functional unit (201 to 208) is mainly configured by the processor 111 executing a program (machine learning support program).

[0040] The image input unit 201 inputs one or more images. In this embodiment, the image input unit 201 inputs an image (additional image) captured at a facility (facility B) to be classified by the classification unit 206 from, for example, the storage device 113. It is assumed that facility B is a facility different from the image facility (facility A) at which an image (basic learning image) used in the initial learning of the classification unit 206 was captured. Note that the image acquisition conditions (first image acquisition conditions) when capturing an image at facility A and the image acquisition conditions (second image acquisition conditions) when capturing an image at facility B are generally different.

[0041] The image state feature amount calculation unit 202 calculates a feature amount (image state feature amount) based on the image state of the input image. Here, the image state means at least one of the brightness, contrast, color tone (hue and / or saturation), blur, noise amount, and resolution of the image. The image state does not include a state related to the shape itself in the image. In this embodiment, the image state feature amount calculation unit 220 has a model (image state feature amount calculation model) for calculating the image state feature amount for the input image, and calculates the image state feature amount (for example, the second image state feature amount for the additional image) of the input image (for example, the additional image) using this model. The machine learning process executed in advance to construct this image state feature amount calculation model will be described later.

[0042] The morphological feature amount calculation unit 203 calculates features (morphological feature amounts) based on the shape of the input image. Here, "shape" refers to the shape of edges, curved parts, etc. in the image. In this embodiment, the morphological feature amount calculation unit 203 has a model (morphological feature amount calculation model) for calculating morphological feature amounts for the input image, and calculates morphological feature amounts (first morphological feature amounts) of the input image (e.g., additional image) using this model. The machine learning process executed in advance to construct this morphological feature amount calculation model will be described later.

[0043] The image state feature quantity control unit 204 outputs an image state feature quantity (third image state feature quantity) obtained by modifying the image state feature quantity of the input image so as to approximate the image state feature quantity based on the image state of one or more basic learning images. The detailed processing of the image state feature quantity control unit 204 will be described later.

[0044] The image generating unit 205 generates an image based on the input image state feature amount and morphological feature amount. In this embodiment, the image generating unit 205 has a model (image generation model) that generates an image corresponding to the input image state feature amount and morphological feature amount, and generates an image corresponding to the input image state feature amount and morphological feature amount using this model. The machine learning process executed in advance to construct this image generation model will be described later.

[0045] The identification unit 206 performs processing to identify the class of an input image. In this embodiment, the identification unit 206 has an identification model that identifies the class of an input image, and identifies the class of the input image using the identification model. In an initial state, the identification model is trained in advance based on teacher data (basic learning images) of images captured in a certain facility (facility A), for example.

[0046] The learning unit 207 performs machine learning of the discriminative model by using learning images for training the discriminative model of the classification unit 206. Here, the learning images are basic learning images and / or additional learning images. The learning unit 207 also performs pre-learning for the image state feature amount calculation model of the image state feature amount calculation unit 202, the morphological feature amount calculation model of the morphological feature amount calculation unit 203, and the image generation model of the image generation unit 205. Details of this pre-learning will be described later. The learning unit 207 is an example of a perturbation image generation unit, an image generation learning unit, a discriminative model learning unit, and an additional learning image determination unit.

[0047] The output unit 208 outputs the learning results of the classification unit 206 learned by the learning unit 207 (for example, information including parameters of the classification model and information on classification accuracy), images generated by the image generation unit 205, images for additional learning, etc. In this embodiment, the output unit 208 outputs (displays) a result display screen 801 (see FIG. 8), an image display screen 901 for additional learning (see FIG. 9), and a correct answer setting screen 1001 (see FIG. 10). The output unit 208 is an example of a learning result output unit, a candidate image output unit, a transition information output unit, an image output unit for additional learning, a class information receiving unit, and a storage control unit.

[0048] Next, pre-learning for the image state feature amount calculation unit 202, the morphological feature amount calculation unit 203, and the image generation unit 205 will be described.

[0049] FIG. 4 is a diagram illustrating pre-learning for an image state feature amount calculation unit, a morphological feature amount calculation unit, and an image generation unit according to an embodiment.

[0050] First, the learning unit 207 applies a perturbation (slight change) to the image state of one image 401 (input image A) of facility A to create a perturbed image 403 (input image A'). Here, input image A' is an image in which the form remains unchanged from input image A, and only the image state has changed.

[0051] Next, the learning unit 207 causes the image state feature amount calculation unit 202 to calculate the image state feature amount 404 (image state feature amount A) of the input image A, and also calculates the image state feature amount 406 (image state feature amount A') of the input image A'. The calculated image state feature amount 406 of the input image A' may be stored in the storage device 113 in association with the input image. Furthermore, the learning unit 207 causes the morphological feature amount calculation unit 203 to calculate the morphological feature amount 405 (morphological feature amount A) of the input image A, and also calculates the morphological feature amount 407 (morphological feature amount A') of the input image A'.

[0052] Next, the learning unit 207 inputs the image state feature A and the morphological feature A through the image generation unit 205 to generate an output image 408 corresponding to the input image A (output image A: image corresponding to the basic image), and inputs the image state feature A' and the morphological feature A' to generate an output image 409 corresponding to the input image A' (output image A': image corresponding to the perturbed image).

[0053] For example, the learning unit 207 creates a plurality of perturbed images for one image and executes the above-mentioned processing, and further executes a similar processing for a plurality of images captured in facility A.

[0054] Then, using the images and features calculated by these processes, the learning unit 207 trains the image state feature quantity calculation unit 202, the morphological feature quantity calculation unit 203, and the image generation unit 205 so that the input image A and the output image A become closer, the input image A' and the output image A' become closer, and the morphological feature quantity A and the morphological feature quantity A' become closer in value. As a specific example, the learning unit 207 trains the image state feature quantity calculation unit 202, the morphological feature quantity calculation unit 203, and the image generation unit 205 so that the value of Loss in the following formula (1) becomes smaller.

[0055] Loss=Loss0A+Loss0B+Loss1 ···(1) Loss0A = MSE(input image A, output image A) Loss0B = MSE(input image A', output image A') Loss1 = MSE(morphological feature A, morphological feature A') Here, MSE() indicates a function for calculating the mean square error between the elements in the ().

[0056] The pre-trained image state feature calculation unit 202, morphological feature calculation unit 203, and image generation unit 205 shown in Figure 4 can generate an image that is close to the original image using the image state features and morphological features, and by changing only the image state features, it is possible to appropriately generate an image in which only the image state has been changed without changing the morphology of the subject in the image.

[0057] Next, the additional learning image determination process will be described.

[0058] FIG. 5 is a diagram illustrating the additional learning image determination process according to an embodiment.

[0059] The image state feature amount control unit 204 acquires, from the memory 112, an image state feature amount 501 (image state feature amount L: first image state feature amount) based on an image (basic image) captured in the facility A. Here, the image state feature amount L may be, for example, an average value of the image state feature amounts of a plurality of images captured in the facility A.

[0060] In addition, the image state feature control unit 204 acquires from the image state feature calculation unit 202 the image state feature 502 (image state feature M: second image state feature) of an image (target image) captured at facility B that is the subject of additional learning.

[0061] Furthermore, the image state feature quantity control unit 204 receives the transformation coefficient 503 from the learning unit 207. Here, the learning unit 207 gradually changes the transformation coefficient from 0 to 1.0, for example.

[0062] The image state feature quantity control unit 204 executes a transformation process (505) for calculating a transformed image state feature quantity 506 (third image state feature quantity) which is a value between the image state feature quantity M and the image state feature quantity L, based on the image state feature quantity L and the image state feature quantity M. In this embodiment, the image state feature quantity control unit 204 calculates the transformed image state feature quantity 506 based on the image state feature quantity L, the image state feature quantity M, and a transformation coefficient. For example, the image state feature quantity control unit 204 calculates the transformed image state feature quantity 506 by the following formula (2). Here, a transformation coefficient of 0 means the image state feature quantity of the target image, and a transformation coefficient of 1.0 means the image state feature quantity of the base image.

[0063] Post-deformation image state feature amount = image state feature amount M × (1 - deformation coefficient) + Image state feature L × deformation coefficient (2)

[0064] The image state feature amount control unit 204 outputs the post-deformation image state feature amount 506 to the image generation unit 205 .

[0065] As a result, the image generation unit 205 generates an image (candidate image) that is a candidate for an additional learning image based on the post-transformation image state features and the morphological features (first morphological features) of the image captured at facility B output from the morphological feature calculation unit 203, and the classification unit 206 classifies the candidate image and notifies the learning unit 207 of the result.

[0066] The learning unit 207 judges whether or not the classification result by the classification unit 206 has changed from the candidate image generated immediately before for each candidate image generated by changing the deformation coefficient. If the classification result has changed as a result, the learning unit 207 determines the candidate image generated by the deformation coefficient before (immediately before) the classification result changed as an image for additional learning, and stores it in the storage device 113. In this way, the image generated by the deformation coefficient immediately before the classification result changed means an image relatively close to the classification surface, and by using this image for additional learning, it is possible to appropriately expand the classification surface while suppressing overlearning. Note that if the classification result does not change even if the deformation coefficient is changed, the learning unit 207 does not select these candidate images as images for additional learning.

[0067] Next, a specific example of an image generated by the image generating unit 205 when the transformation coefficient is changed will be described.

[0068] Fig. 6A is a diagram showing images in a case where the contrast of an image is affected by a change in a deformation coefficient according to an embodiment. Fig. 6B is a diagram showing images in a case where the luminance value of an image is affected by a change in a deformation coefficient according to an embodiment. Fig. 6C is a diagram showing images in a case where the blurriness of an image is affected by a change in a deformation coefficient according to an embodiment. Note that an image with a deformation coefficient of 0.0 corresponds to a target image, and an image with a deformation coefficient of 1.0 corresponds to a base image.

[0069] When the deformation coefficient is changed stepwise from 0.0 to 1.0, if the image state feature is a feature that is strongly influenced by the contrast of the image, the contrast gradually changes to approach the base image as shown in Fig. 6A. If the image state feature is a feature that is strongly influenced by the brightness value of the image, the brightness value gradually changes to approach the base image as shown in Fig. 6B. If the image state feature is a feature that is strongly influenced by the blurriness of the image, the degree of blur gradually changes to approach the base image as shown in Fig. 6C.

[0070] Next, the relationship between the image obtained by changing the deformation coefficient and the discrimination surface will be described.

[0071] FIG. 7 is a diagram showing the relationship between an image obtained by changing the deformation coefficient according to an embodiment and a discrimination surface.

[0072] For example, when the deformation coefficient for the target image 304 is increased stepwise, the image changes so as to approach the basic image 701. Here, if the discrimination plane 303 is between the target image 304 and the basic image 701, the discrimination result for the candidate image changes between any of the deformation coefficients by changing the deformation coefficient. In the example of FIG. 7, the discrimination result for the candidate image changes between a deformation coefficient of 0.4 and a deformation coefficient of 0.6. This means that the discrimination plane 303 exists in the part where the discrimination result changes, as shown in FIG. 7. In this embodiment, the learning unit 207 determines the candidate image corresponding to the deformation coefficient (0.4 in this example) immediately before the part where the discrimination result changes as the additional learning image. Since this image has an image state closer to the basic image 701 than the target image 304, the influence on the discrimination plane 303 is suppressed compared to additional learning with the target image 304, that is, overlearning can be suppressed.

[0073] Next, the result display screen 801 will be described.

[0074] FIG. 8 is a diagram showing a result display screen according to one embodiment.

[0075] The result display screen 801 is a screen displayed by the output unit 208 when additional learning is performed, for example, and includes a classification accuracy transition display area 802 and a training image number transition display area 803.

[0076] The classification accuracy transition display area 802 displays a classification accuracy transition graph with the number of additional learnings on the horizontal axis and the classification accuracy on the vertical axis. For example, this graph is generated by the output unit 208 based on the evaluation results, which are generated by causing the classification unit 206 to perform classification processing on the evaluation images after additional learning is performed by the learning unit 207 and storing the evaluation results in a storage device or the like. In this embodiment, the classification accuracy transition graph includes graphs of classification accuracy for images for each facility (for each image acquisition condition), for example, a classification accuracy graph for images captured at facility A and a classification accuracy graph for images captured at facility B. According to these graphs, it is possible to properly grasp the change in classification accuracy for images of facility A due to additional learning, and to properly determine whether the additional learning has had a negative effect on the classification accuracy for images of facility A.

[0077] A learning image number transition graph with the number of additional learnings on the horizontal axis and the number of learning images on the vertical axis is displayed in learning image number transition display area 803. This graph can be created based on the total number of images, for example, the number of basic learning images and the number of additional learning images, stored at the time of each additional learning.

[0078] Next, the additional learning image display screen 901 will be described.

[0079] FIG. 9 is a diagram showing an additional learning image display screen according to an embodiment.

[0080] The additional learning image display screen 901 is a screen displayed by the output unit 208 when an additional learning image is decided, for example, and includes a candidate image display area 902 and an additional learning image indication icon 903.

[0081] A plurality of deformation coefficients and candidate images generated corresponding to the deformation coefficients are displayed in the candidate image display area 902. The additional learning image instruction icon 903 is an icon indicating an image determined as an additional learning image from among the candidate images displayed in the candidate image display area 902. Note that in this embodiment, the learning unit 207 determines the additional learning image based on internal processing, but this is not limited thereto. For example, the learning unit 207 may determine a candidate image selected by the user via the input device 120 from among the candidate images displayed in the candidate image display area 902 as the additional learning image.

[0082] Next, the correct answer setting screen 1001 will be described.

[0083] FIG. 10 is a diagram showing a correct answer setting screen according to an embodiment.

[0084] The correct answer setting screen 1001 is a screen displayed by the output unit 208, for example, after an additional learning image has been determined, and includes an additional learning image display area 1002, a recognition result display area 1003, a correct answer input area 1004, a display image change button 1005, and a registration button 1006.

[0085] The additional learning image display area 1002 is an area where an additional learning image is displayed. The classification result display area 1003 is an area where a classification result of the additional learning image is displayed. The correct answer value input area 1004 is an area where a correct answer value (class information) indicating a class into which the additional learning image is classified is input. The display image change button 1005 is a button for switching the additional learning image displayed in the additional learning image display area 1002. The registration button 1006 is a button for accepting an instruction to associate a correct answer value with the additional learning image displayed in the additional learning image display area 1002 and register it. When the registration button 1006 is pressed, the output unit 208 associates a correct answer value label with the additional learning image displayed in the additional learning image display area 1002 and stores it in the storage device 113.

[0086] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit of the present invention.

[0087] For example, in the above embodiment, the candidate image is gradually brought closer to the base image from the target image by changing the deformation coefficient, and the candidate image immediately before the image classification result changes is used as the additional learning image. However, the present invention is not limited to this, and for example, an image that is in an image state between the candidate image and the target image may be used as the additional learning image, or an image corresponding to a deformation coefficient that is within a predetermined range relative to the range of deformation coefficients in which the classification result has changed may be used as the additional learning image.

[0088] In the above embodiment, an example of generating an additional learning image has been shown, but an additional evaluation image may be generated. For example, the image state feature amount calculation unit 202 may calculate an image state feature amount (fourth image state feature amount) based on the image state of the evaluation image in the storage device (one example of an evaluation image storage unit), the morphological feature amount calculation unit 203 may calculate a morphological feature amount (second morphological feature amount) based on the morphology of the subject of the evaluation image, the image state feature amount control unit 204 may adjust the image state feature amount (fourth image state feature amount) to generate a new image state feature amount (fifth image state feature amount), and the image generation unit 205 may generate a new evaluation image based on the fifth image state feature amount and the second morphological feature amount. Note that since the morphology itself of the new evaluation image is the same as that of the original evaluation image, the label of the new evaluation image may be the same as that of the original evaluation image.

[0089] In the above embodiment, the facility where the images are taken is different as an example of the first image acquisition condition and the second image acquisition condition, but the present invention is not limited to this, and may be any other case where the image acquisition conditions are different. For example, when an image of an object to be examined for a pathological examination of bacteria or cells is to be identified by the identification unit, the state of the object to be examined differs depending on the institution or person in charge who generates the object to be examined, and the state of the image obtained by imaging the object to be examined also differs, so this case can also be considered as a case where the image acquisition conditions are different. In this case, for example, an image of the object to be examined generated by person A may be considered as an image under the first image acquisition condition, and an image of the object to be examined generated by person B may be considered as an image under the second image acquisition condition. Also, when the cameras to be used for imaging are different, the image acquisition conditions may be different.

[0090] In the above-described embodiments, a part or all of the processing performed by the processor may be performed by a hardware circuit. The programs in the above-described embodiments may be installed from a program source. The program source may be a program distribution server or a recording medium (e.g., a portable recording medium). [Explanation of symbols]

[0091] 100...machine learning system, 101...machine learning device, 111...processor, 112...memory, 113...storage device, 114...bus, 120...input device, 121...display device, 201...image input unit, 202...image state feature amount calculation unit, 203...mobile information calculation unit, 204...image state feature amount control unit, 205...image generation unit, 206...recognition unit, 207...learning unit, 208...output unit

Claims

1. A machine learning support device that generates additional learning images for additional learning of a discrimination model that identifies a class into which an image is classified, comprising: the discriminant model is trained using a plurality of base images acquired under a first image acquisition condition; a receiving unit that receives an additional image acquired under a second image acquisition condition to be classified by the classification model; an image state feature amount calculation unit that calculates a second image state feature amount based on an image state of the additional image; an image state feature amount control unit that generates one or more third image state feature amounts that are feature amounts between a first image state feature amount based on an image state of the base image and the second image state feature amount; a morphological feature amount calculation unit that calculates a first morphological feature amount based on a morphology of a subject of the additional image; an image generating unit that generates one or more candidate images based on the one or more third image state feature amounts and the first morphological feature amount; an additional learning image determination unit that determines at least one of the candidate images as an additional learning image; Equipped with The image state includes at least one state of the brightness, contrast, color (hue and / or saturation), blur, noise amount, and resolution of the image. Machine learning assistant.

2. A classification unit that uses the classification model to classify a class into which an image is classified, The additional learning image determination unit the image state feature amount control unit sequentially changes the third image state feature amount so that the third image state feature amount becomes closer to the first image state feature amount in order from the value closest to the second image state feature amount; causing the image generation unit to sequentially generate the candidate images based on the third image state feature amount and the morphological feature amount; causing the identification unit to identify a class into which the candidate image is classified; The additional learning image is determined based on the classification result by the classification unit. The machine learning assistance device according to claim 1 .

3. The additional learning image determination unit A candidate image generated immediately before the candidate image for which the classification result by the classification unit has changed is determined as the additional learning image. The machine learning assistance device according to claim 2 .

4. a perturbed image generating unit that generates a perturbed image by perturbing an image state of each of the plurality of basic images acquired under the first image acquisition condition; (A) the basic image is similar to a basic-image corresponding image output by the image generating unit based on image state features output by the image state feature calculation unit for the basic image and morphological feature information output by the morphological feature calculation unit for the basic image, (B) the perturbed image is similar to the perturbed image corresponding image output by the image generating unit based on the image state feature amount output by the image state feature amount calculation unit for the perturbed image and the morphological feature amount information output by the morphological feature amount calculation unit for the perturbed image; and (C) an image generation learning unit that performs machine learning on the image state feature calculation unit, the morphological feature calculation unit, and the image generation unit so that morphological feature information output by the morphological feature calculation unit for the basic image and morphological feature information output by the morphological feature calculation unit for the perturbed image are similar to each other. The machine learning assistance device according to claim 1 .

5. The image processing device further includes a discriminative model training unit that trains the discriminative model using the additional learning image. The machine learning assistance device according to claim 1 .

6. The apparatus further includes a learning result output unit that outputs a learning result by the discrimination model learning unit. The machine learning assistance device according to claim 5 .

7. The image processing apparatus further includes an additional learning image output unit that outputs the additional learning image in a distinguishable manner. The machine learning assistance device according to claim 1 .

8. The image processing apparatus further includes a candidate image output unit that outputs one or more of the candidate images. The machine learning assistance device according to claim 1 .

9. A reception unit that receives a selection of an image to be used as the additional learning image from the one or more candidate images from a user, The additional learning image determination unit determines the selected candidate image as the additional learning image. The machine learning assistance device according to claim 1 .

10. a class information receiving unit that displays the additional learning image and receives, from a user, class information indicating a class into which the additional learning image is to be classified; a storage control unit that stores the additional learning image in association with the class information; Further comprising The machine learning assistance device according to claim 1 .

11. The present invention further includes a transition information output unit that displays transition information regarding the transition of classification accuracy due to additional learning of the classification process by the classification unit and / or the number of learning images. The machine learning assistance device according to claim 1 .

12. The transition information output unit displays transition information regarding a transition of classification accuracy for images for each image acquisition condition of the classification process performed by the classification unit. The machine learning assistance device according to claim 11.

13. Further, an evaluation image storage unit stores a plurality of evaluation images for evaluating the classification model, the image state feature amount calculation unit calculates a fourth image state feature amount based on an image state of the evaluation image; the morphological feature amount calculation unit calculates a second morphological feature amount based on a morphology of a subject of the evaluation image; the image state feature control unit adjusts the fourth image state feature to generate a fifth image state feature; The image generating unit generates a new evaluation image based on the fifth image state feature amount and the second morphological feature amount. The machine learning assistance device according to claim 1 .

14. A machine learning support method using a machine learning support device that generates additional learning images for additional learning of a discrimination model that identifies a class into which an image is classified, comprising: the discriminant model is trained using a plurality of base images acquired under a first image acquisition condition; receiving an additional image captured under a second image capture condition to be classified by the classification model; calculating a second image state feature based on an image state of the additional image; generating one or more third image state features that are features between a first image state feature based on an image state of the base image and the second image state feature; calculating a first morphological feature based on a morphology of a subject of the additional image; generating one or more candidate images based on the one or more third image state features and the first morphological feature; determining at least one of the candidate images as an additional learning image; The image state includes at least one state of the brightness, contrast, color (hue and / or saturation), blur, noise amount, and resolution of the image. Machine learning assisted methods.

15. A machine learning support program for causing a computer to execute the program to generate additional learning images for additional learning of a discrimination model that identifies a class into which an image is classified, the program comprising: the discriminant model is trained using a plurality of base images acquired under a first image acquisition condition; The machine learning assistance program causes the computer to: a receiving unit that receives an additional image acquired under a second image acquisition condition to be classified by the classification model; an image state feature amount calculation unit that calculates a second image state feature amount based on an image state of the additional image; an image state feature amount control unit that generates one or more third image state feature amounts that are feature amounts between a first image state feature amount based on an image state of the base image and the second image state feature amount; a morphological feature amount calculation unit that calculates a first morphological feature amount based on a morphology of a subject of the additional image; an image generating unit that generates one or more candidate images based on the one or more third image state feature amounts and the first morphological feature amount; an additional learning image determination unit that determines at least one of the candidate images as an additional learning image; and make it work. The image state includes at least one state of the brightness, contrast, color (hue and / or saturation), blur, noise amount, and resolution of the image. Machine learning assistance program.

Images