Machine learning assistance device, machine learning assistance method, and machine learning assistance program

Abstract

An image for performing additional learning of a recognition model with high precision and high efficiency is generated. In a machine learning device (101) that generates an additional learning image, a recognition model is learned using a plurality of basic images acquired under a first image acquisition condition, and the machine learning device is provided with: an image input unit (201) that receives an additional image acquired under a second image acquisition condition; an image state feature amount calculation unit (202) that calculates a second image state feature amount of the additional image; an image state feature amount control unit (204) that generates a third image state feature amount between the first image state feature amount and the second image state feature amount of the base image; a morphological feature amount calculation unit (203) that calculates a first morphological feature amount based on the form of the subject of the additional image; an image generation unit (205) that generates a candidate image on the basis of the third image state feature amount and the first morphological feature amount; and a learning unit (207) that determines at least one of the candidate images as an additional learning image.

Description

TECHNICAL FIELD

[0001] The present application relates to a technology for assisting additional learning of an identification model for identifying an image as a class. BACKGROUND

[0002] A technology of identifying a class of an image using an identification model generated by machine learning is known. For example, in a case where an identification model is subjected to machine learning using images taken in a certain facility, if the identification model is used to perform an identification process on an image taken in another facility, the conditions for taking the images in the certain facility and the other facility are different, and as a result, a good identification accuracy cannot be obtained in some cases compared to the identification accuracy for the image taken in the certain facility.

[0003] In this regard, a technology is disclosed in which an identification model is generated by machine learning using images taken in garbage pits of a plurality of facilities and teacher data in which labels are attached to the types of waste in the images, and then the identification model is subjected to additional learning using images taken in a garbage pit of a second facility different from the plurality of facilities and trainer data in which labels are attached to the types of waste in the images, so that the identification model is made to correspond to the second facility (for example, refer to Patent Literature 1).

[0004] In addition, as another technology, a technology is disclosed in which correction parameters used in conversion of an image are generated based on a result obtained by determining the appearance of an image represented by image data, the image is converted using the correction parameters, inference based on machine learning is performed on the converted image, teacher data is generated by associating the inference result with the image data, and additional learning of an inference model is performed using the teacher data (for example, refer to Patent Literature 2).

[0005] PRIOR ART DOCUMENTS PATENT LITERATURE Patent Literature 1: Japanese Patent Application Publication No. 2023-12094 Patent Literature 2: Japanese Patent Application Publication No. 2020-160804 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION As described above, in the technology disclosed in Patent Literature 1, additional learning of an identification model is performed using images taken in another facility. At this time, for example, in a case where the images taken in the other facility are small in number, overfitting occurs, and it can be difficult to obtain a good detection accuracy. On the other hand, in a case where the images taken in the other facility are large in number, it can lead to an increase in the man-hours for associating annotations with the taken images, or a decrease in the identification accuracy for the images of the facility that has already been learned. In addition, the technology disclosed in Patent Literature 2 also has the same problem.

[0007] The present application has been made in view of the above circumstances, and has an object to provide a technique of generating an image for performing additional learning of a recognition model with high accuracy and high efficiency.

[0008] Means for solving the problem To achieve the above object, a machine learning assistance device that generates an additional learning image for performing additional learning of a recognition model that recognizes a category to which an image is classified, the recognition model being learned using a plurality of base images acquired under a first image acquisition condition, includes: a reception unit that receives an additional image acquired under a second image acquisition condition as an object of recognition in the recognition model; an image state feature amount calculation unit that calculates a second image state feature amount based on an image state of the additional image; an image state feature amount control unit that generates one or more third image state feature amounts that are feature amounts between a first image state feature amount based on an image state of the base image and the second image state feature amount; a form feature amount calculation unit that calculates a first form feature amount based on a form of a subject of the additional image; an image generation unit that generates one or more candidate images based on the one or more third image state feature amounts and the first form feature amount; and an additional learning image decision unit that decides at least one of the candidate images as an additional learning image, the image state including a state of at least one of brightness, contrast, hue (hue and / or chroma), blurring degree, noise amount, and resolution of an image.

[0009] Effects of the Invention According to the present application, it is possible to generate an image for performing additional learning of a recognition model with high accuracy and high efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1A FIG. 1 is a diagram illustrating a first example of image recognition using a recognition model of a conventional example.

[0011] Figure 1B FIG. 2 is a diagram illustrating a second example of image recognition using a recognition model of a conventional example.

[0012] Figure 1C FIG. 3 is a diagram illustrating a third example of image recognition using a recognition model of a conventional example.

[0013] Figure 1D FIG. 4 is a diagram conceptually illustrating recognition using a recognition model of an embodiment.

[0014] Figure 2 FIG. 5 is a hardware configuration diagram of a machine learning system of an embodiment.

[0015] Figure 3 is a functional configuration diagram of a machine learning device of an embodiment.

[0016] Figure 4 is a diagram illustrating pre-learning of an image state feature amount calculation section, a form feature amount calculation section, and an image generation section of an embodiment.

[0017] Figure 5 is a diagram illustrating an additional learning image decision process of an embodiment.

[0018] Figure 6A is a diagram of each image in a case where the contrast of the image is affected by a change in the deformation coefficient of an embodiment.

[0019] Figure 6B is a diagram of each image in a case where the luminance value of the image is affected by a change in the deformation coefficient of an embodiment.

[0020] Figure 6C is a diagram of each image in a case where the blurring degree of the image is affected by a change in the deformation coefficient of an embodiment.

[0021] Figure 7 is a diagram illustrating the relationship between the image obtained by changing the deformation coefficient and the recognition surface of an embodiment.

[0022] Figure 8 is a diagram of a result display screen of an embodiment.

[0023] Figure 9 is a diagram of an additional learning image display screen of an embodiment.

[0024] Figure 10 is a diagram of a correct answer value setting screen of an embodiment. DETAILED DESCRIPTION

[0025] Embodiments are described with reference to the accompanying drawings. Furthermore, the embodiments described below are not intended to limit the invention encompassed by the claims, and all components not described herein are not excluded from the scope of the invention. Furthermore, the embodiments described below are not necessarily restrictive and are given by way of example only. Not necessarily all objects of the invention are achieved by means of the embodiments described below, and some objects can be achieved independently.

[0026] In describing the present embodiment, first, image recognition using a conventional recognition model is described, and then an outline of the recognition model of the present embodiment is described.

[0027] Figure 1A is a diagram illustrating a first example of image recognition using a recognition model of a conventional example. In the example, a recognition model of a conventional example is used, and the recognition model is trained using a plurality of images of a certain form. Figure 1AIn the case of FIG. 6, an area divided in the lateral direction by facilities that have taken images (facilities A, B: one example of image-taking conditions under which images are taken) and divided in the vertical direction by categories to which the images belong (categories a, b) is shown, and corresponding images are arranged in the area. Further, in the case of FIG. 7, which will be described later, the same display is performed. Figures 1B-1D

[0028] In the case of FIG. 6, an area divided in the lateral direction by facilities that have taken images (facilities A, B: one example of image-taking conditions under which images are taken) and divided in the vertical direction by categories to which the images belong (categories a, b) is shown, and corresponding images are arranged in the area. Further, in the case of FIG. 7, which will be described later, the same display is performed. Figure 1A

[0029] Figure 1A An example of a case in which the recognition model is subjected to machine learning using only images taken in facility A is shown. In this case, a recognition surface 303 in the recognition model that serves as a boundary between whether an image is a category a or a category b can be conceptually represented as shown in FIG. 8. Figure 1A

[0030] In this way, if machine learning is performed using only images taken in facility A, since images taken in facility B are not taken into account, it can be difficult to appropriately recognize the category of an image for an image taken in facility B.

[0031] Next, a case in which machine learning (additional learning) is performed including a plurality of images taken in facility B is described.

[0032] Figure 1B FIG. 9 is a diagram illustrating a second example of image recognition using the recognition model of the foregoing example. Figure 1B An example of a case in which machine learning (additional learning) is performed including a plurality of images taken in facility B is shown. In the case of FIG. 9, an area divided in the lateral direction by facilities that have taken images (facilities A, B: one example of image-taking conditions under which images are taken) and divided in the vertical direction by categories to which the images belong (categories a, b) is shown, and corresponding images are arranged in the area. Further, in the case of FIG. 10, which will be described later, the same display is performed. Figure 1B Figure 1B

[0033] As shown in FIG. 10, it takes effort to prepare a plurality of images taken in facility B, and it also takes effort to attach a label indicating a category to each image in order to perform additional learning. Figure 1B

[0034] Further, if additional learning is performed using a plurality of images taken in facility B, the recognition accuracy for images taken in facility B is improved according to the recognition surface 306, but the recognition surface is affected, and thus it can be difficult to improve the recognition accuracy for images taken in facility A.

[0035] ​​​​​​Next, a case where additional learning is performed including a small number of images photographed in facility B after the machine learning illustrated in Figure 1A

[0036] Figure 1C is a diagram illustrating a third example of image recognition using the recognition model of the prior example. Figure 1C An example of a case where additional learning is performed including a small number of images photographed in facility B is illustrated. In Figure 1C , an image 307 belonging to class a photographed in facility B is indicated by a black rectangle, and an image 308 belonging to class b photographed in facility B is indicated by a black circle. In this case, a recognition surface 309 in the recognition model that is a boundary of whether an image is class a or class b can be conceptually indicated as illustrated in Figure 1C

[0037] In this way, if additional learning is performed using only a small number of images photographed in facility B, the image acquisition state of the image can sometimes greatly differ from that of the image photographed in facility A, learning (overlearning) of the image used exclusively for learning can occur, and it can become a recognition surface 309 that greatly changes from the recognition surface 303 before learning. Therefore, it can not be possible to universally recognize images photographed in facility B, and it can not be possible to improve recognition accuracy.

[0038] Next, a summary of the recognition model of the present embodiment is described.

[0039] Figure 1D is a diagram conceptually illustrating recognition using the recognition model of the present embodiment.

[0040] Figure 1D An example of a case where additional learning is performed including an image generated to approach the recognition surface 303 by making a small number of images photographed in facility B approach the image acquisition state (e.g., color tone, brightness, etc.) of the image photographed in facility A is illustrated. In Figure 1D , an image 307 belonging to class a photographed in facility B is indicated by a black rectangle, and an image 310 generated to approach the recognition surface 303 based on the image is indicated by a gray rectangle. In this case, a recognition surface 311 in the recognition model that is a boundary of whether an image is class a or class b can be conceptually indicated as illustrated in Figure 1D

[0041] In this way, when additional learning is performed using an image that approaches the recognition surface 303, it is possible to obtain a recognition surface 311 that gently amplifies the recognition surface 303. Therefore, it is possible to reduce the occurrence of overlearning illustrated in Figure 1C , it is possible to universally recognize images photographed in facility B, and it is possible to improve recognition accuracy. ​​​

[0042] Next, a machine learning system of an embodiment will be described.

[0043] Figure 2 Fig. 1 is a hardware configuration diagram of a machine learning system of an embodiment.

[0044] The machine learning system 100 includes a machine learning device 101 as an example of a machine learning assistance device, an input device 120, and a display device 121.

[0045] The input device 120 is, for example, a mouse, a keyboard, or the like, and receives input of information from a user. The display device 121 is, for example, a display, and displays and outputs a user interface (screen) including various information.

[0046] The machine learning device 101 is configured by, for example, a PC (Personal Computer), a general-purpose server, or the like, and includes an interface section 110, a processor 111, a memory 112, a storage device 113, and a bus 114. The interface section 110, the processor 111, the memory 112, and the storage device 113 are communicably connected via the bus 114.

[0047] The interface section 110 includes, for example, a wired LAN card, a wireless LAN card, or the like, and communicates with other devices via a network. The interface section 110 can receive various images via the network. In addition, the interface section 110 receives information from the input device 120 and delivers it to the processor 111. In addition, the interface section 110 causes the display device 121 to display various screens in accordance with an instruction of the processor 111.

[0048] The processor 111 performs various processing in accordance with a program stored in the memory 112 and / or the storage device 113.

[0049] The memory 112 is, for example, a RAM (RANDOM ACCESS MEMORY), and stores a program executed by the processor 111, necessary information.

[0050] The storage device 113 is, for example, a hard disk, a flash memory, or the like, and stores a program (machine learning assistance program) executed by the processor 111, data used by the processor 111. In the present embodiment, the storage device 113 stores, for example, teacher data (a group of images and class names of images: basic learning data) and evaluation data (a group of images and class names of images: evaluation images) regarding a plurality of images acquired in facility A, an image acquired in facility B, teacher data (a group of images and class names of images: additional learning data) regarding a generated additional learning image, and the like.

[0051] Next, a functional configuration of the machine learning device 101 of an embodiment will be described.

[0052] Figure 3 FIG. 1 is a functional configuration diagram of a machine learning device according to an embodiment.

[0053] The machine learning device 101 includes an image input section 201 as an example of a reception section, an image state feature amount calculation section 202, a form feature amount calculation section 203, an image state feature amount control section 204, an image generation section 205, an identification section 206, a learning section 207, and an output section 208. Here, each functional section (201 to 208) is mainly configured by executing a program (machine learning assistance program) by the processor 111.

[0054] The image input section 201 inputs one or more images. In the present embodiment, the image input section 201 inputs, for example, an image (additional image) photographed in a facility (facility B) as an identification target to be identified by the identification section 206 from the storage device 113. The facility B is a facility different from an image facility (facility A) in which an image (basic learning image) used in the initial learning by the identification section 206 is photographed. In addition, an image acquisition condition (first image acquisition condition) at the time of photographing the image in the facility A and an image acquisition condition (second image acquisition condition) at the time of photographing the image in the facility B are usually different.

[0055] The image state feature amount calculation section 202 calculates a feature amount (image state feature amount) based on an image state of the input image. Here, the image state refers to a state of at least one or more of luminance, contrast, hue (hue and / or chroma), blurring degree, noise amount, and resolution of the image. In addition, the image state does not include a state related to a shape itself in the image. In the present embodiment, the image state feature amount calculation section 220 has a model (image state feature amount calculation model) that calculates an image state feature amount for the input image, and calculates an image state feature amount (for example, a second image state feature amount for the additional image) of the input image (for example, the additional image) using the model. The processing of machine learning performed in advance in order to construct this image state feature amount calculation model is described later.

[0056] The form feature amount calculation section 203 calculates a feature amount (form feature amount) based on a form of the input image. Here, the form refers to a shape of an edge, a curved portion, or the like in the image. In the present embodiment, the form feature amount calculation section 203 has a model (form feature amount calculation model) that calculates a form feature amount for the input image, and calculates a form feature amount (first form feature amount) of the input image (for example, the additional image) using the model. The processing of machine learning performed in advance in order to construct this form feature amount calculation model is described later.

[0057] The image state feature quantity control section 204 outputs an image state feature quantity in which the image state feature quantity of the input image is deformed so as to be close to the image state feature quantity based on the image state of one or more basic learning images (third image state feature quantity). Details of the processing of the image state feature quantity control section 204 will be described later.

[0058] The image generation section 205 generates an image based on the input image state feature quantity and the shape feature quantity. In the present embodiment, the image generation section 205 has a model (image generation model) that generates an image corresponding to the input image state feature quantity and the shape feature quantity, and generates an image corresponding to the input image state feature quantity and the shape feature quantity using the model. Details of the processing of machine learning performed in advance in order to construct the image generation model will be described later.

[0059] The recognition section 206 performs processing of recognizing the category of the input image. In the present embodiment, the recognition section 206 has a recognition model that recognizes the category of the input image, and recognizes the category of the input image using the recognition model. The recognition model is pre-learned based on, for example, teacher data (basic learning image) of images captured in a certain facility (facility A) in an initial state.

[0060] The learning section 207 performs machine learning of the recognition model of the recognition section 206 using a learning image for learning the recognition model. Here, the learning image refers to a basic learning image and / or an additional learning image. Further, the learning section 207 performs pre-learning of the image state feature quantity calculation model of the image state feature quantity calculation section 202, the shape feature quantity calculation model of the shape feature quantity calculation section 203, and the image generation model of the image generation section 205. Details of the pre-learning will be described later. The learning section 207 is an example of a perturbed image generation section, an image generation learning section, a recognition model learning section, and an additional learning image decision section.

[0061] The output section 208 outputs a learning result of the recognition section 206 learned by the learning section 207 (for example, information including parameters of the recognition model, information of recognition accuracy), an image generated by the image generation section 205, an additional learning image, and the like. In the present embodiment, the output section 208 outputs (displays) a result display screen 801 (refer to Figure 8 ), an additional learning image display screen 901 (refer to Figure 9 ), and a correct answer value setting screen 1001 (refer to Figure 10 ). The output section 208 is an example of a learning result output section, a candidate image output section, a shift information output section, an additional learning image output section, a category information reception section, and a saving control section.

[0062] Next, the pre-learning of the image state feature amount calculation section 202, the form feature amount calculation section 203, and the image generation section 205 will be described.

[0063] Figure 4 is a diagram illustrating the pre-learning of the image state feature amount calculation section, the form feature amount calculation section, and the image generation section of one embodiment.

[0064] First, the learning section 207 creates a perturbed image 403 (input image A') by applying a perturbation (a slight change) to the image state with respect to one image 401 (input image A) of the facility A. Here, the input image A' is an image in which the form is not changed with respect to the input image A and only the image state is changed.

[0065] Next, the learning section 207 causes the image state feature amount calculation section 202 to calculate the image state feature amount 404 (image state feature amount A) of the input image A and the image state feature amount 406 (image state feature amount A') of the input image A'. In addition, the calculated image state feature amount 406 of the input image A' can be stored in the storage device 113 in association with the input image. In addition, the learning section 207 causes the form feature amount calculation section 203 to calculate the form feature amount 405 (form feature amount A) of the input image A and the form feature amount 407 (form feature amount A') of the input image A'.

[0066] Next, the learning section 207 causes the image generation section 205 to input the image state feature amount A and the form feature amount A and generate an output image 408 (output image A: basic image corresponding image) corresponding to the input image A, and input the image state feature amount A' and the form feature amount A' and generate an output image 409 (output image A': perturbed image corresponding image) corresponding to the input image A'.

[0067] The learning section 207 creates a plurality of perturbed images and performs the above processing with respect to one image, for example, and further performs the same processing with respect to a plurality of images captured in the facility A.

[0068] Further, the learning section 207 learns the image state feature amount calculation section 202, the form feature amount calculation section 203, and the image generation section 205 using the images and the feature amounts calculated in these processes so that the input image A and the output image A are close to each other, the input image A' and the output image A' are close to each other, and the form feature amount A and the form feature amount A' become close values. As a specific example, the learning section 207 learns the image state feature amount calculation section 202, the form feature amount calculation section 203, and the image generation section 205 so that the value of Loss of the following formula (1) is smaller.

[0069] Loss = LossOA + LossOB + Lossl... (1) LossOA = MSE (input image A, output image A) LossOB = MSE (input image A', output image A') Lossl = MSE (morphological feature amount A, morphological feature amount A') Here, MSE() represents a function that calculates the mean square error between elements within ().

[0070] According to Figure 4 the image state feature amount calculation section 202, the morphological feature amount calculation section 203, and the image generation section 205 that have been pre-learned as illustrated in FIG. 2, an image close to the original image can be generated by the image state feature amount and the morphological feature amount, and an image in which only the image state changes without changing the morphology of the subject in the image can be appropriately generated by changing only the image state feature amount.

[0071] Next, the additional learning image decision processing will be described.

[0072] Figure 5 is a diagram that illustrates the additional learning image decision processing of an embodiment.

[0073] The image state feature amount control section 204 acquires the image state feature amount 501 (image state feature amount L: 1st image state feature amount) based on the image (basic image) photographed in the facility A from the storage 112. Here, as the image state feature amount L, for example, the average of the image state feature amounts of a plurality of images photographed in the facility A can be set.

[0074] In addition, the image state feature amount control section 204 acquires the image state feature amount 502 (image state feature amount M: 2nd image state feature amount) of the image (target image) photographed in the facility B that is the target of the additional learning from the image state feature amount calculation section 202.

[0075] In addition, the image state feature amount control section 204 receives the morphing coefficient 503 from the learning section 207. Here, the learning section 207, for example, gradually changes the morphing coefficient from 0 to 1.0.

[0076] The image state feature amount control section 204 performs a morphing process (505) that calculates the value between the image state feature amount M and the image state feature amount L, that is, the morphed image state feature amount 506 (the 3rd image state feature amount), based on the image state feature amount L and the image state feature amount M. In the present embodiment, the image state feature amount control section 204 calculates the morphed image state feature amount 506 based on the image state feature amount L, the image state feature amount M, and the morphing coefficient. For example, the image state feature amount control section 204 calculates the morphed image state feature amount 506 by the following equation (2). Here, the case where the morphing coefficient is 0 refers to the image state feature amount of the target image, and the case where the morphing coefficient is 1.0 refers to the image state feature amount of the base image.

[0077] Morphed image state feature amount = image state feature amount M x (1 - morphing coefficient) + image state feature amount L x morphing coefficient (2) The image state feature amount control section 204 outputs the morphed image state feature amount 506 to the image generation section 205.

[0078] As a result, the image generation section 205 generates an image (a candidate image) that becomes a candidate of an additional learning image based on the morphed image state feature amount and the shape feature amount (the 1st shape feature amount) of the image photographed in the facility B that is output from the shape feature amount calculation section 203, and the recognition section 206 performs recognition of the candidate image and notifies the result to the learning section 207.

[0079] The learning section 207 determines whether the recognition result of the recognition section 206 is changed from the immediately preceding generated candidate image for each of the candidate images generated by changing the morphing coefficient. As a result, in the case where the recognition result is changed, the learning section 207 determines the candidate image generated according to the morphing coefficient before (immediately before) the recognition result is changed as an additional learning image and stores it in the storage device 113. In this way, the image generated according to the morphing coefficient immediately before the recognition result is changed refers to an image that is closer to the recognition surface, and by using this image for additional learning, it is possible to suppress overlearning and to appropriately enlarge the recognition surface. In addition, in the case where the recognition result is not changed even if the morphing coefficient is changed, the learning section 207 does not determine these candidate images as additional learning images.

[0080] Next, a specific example of an example of the image generated by the image generation section 205 in the case where the morphing coefficient is changed will be described.

[0081] Figure 6A is a graph that shows each image in the case where the contrast of the image is affected by the change of the morphing coefficient according to one embodiment. Figure 6Bis a graph showing each image in a case where the luminance value of the image is affected by the change of the morphing coefficient according to one embodiment. Figure 6C is a graph showing each image in a case where the blur degree of the image is affected by the change of the morphing coefficient according to one embodiment. In addition, the image with the morphing coefficient of 0.0 corresponds to the target image, and the image with the morphing coefficient of 1.0 corresponds to the base image.

[0082] If the morphing coefficient is changed from 0.0 to 1.0 in stages, in a case where the image state feature quantity is a feature quantity in which the contrast of the image is strongly affected, as shown in Figure 6A , the contrast changes in a manner of gradually approaching the base image. In addition, in a case where the image state feature quantity is a feature quantity in which the luminance value of the image is strongly affected, as shown in Figure 6B , the luminance value changes in a manner of gradually approaching the base image. In addition, in a case where the image state feature quantity is a feature quantity in which the blur degree of the image is strongly affected, as shown in Figure 6C , the blur degree changes in a manner of gradually approaching the base image.

[0083] Next, the relationship between the image obtained by the change of the morphing coefficient and the recognition surface will be described.

[0084] Figure 7 is a graph showing the relationship between the image obtained by the change of the morphing coefficient and the recognition surface according to one embodiment.

[0085] For example, if the morphing coefficient is increased in stages with respect to the target image 304, the image changes in a manner of approaching the base image 701. Here, in a case where the recognition surface 303 is located between the target image 304 and the base image 701, by changing the morphing coefficient, the recognition result of the candidate image changes between any of the morphing coefficients. In Figure 7 , in the example, the recognition result of the candidate image changes between the morphing coefficient 0.4 and the morphing coefficient 0.6. This means that there is the recognition surface 303 in the portion where the recognition result changes as shown in Figure 7 In the present embodiment, the learning unit 207 determines the candidate image corresponding to the morphing coefficient immediately before the portion where the recognition result changes (0.4 in this example) as the image for additional learning. This image is an image in which the image state is closer to the base image 701 than the target image 304, and thus, compared to the additional learning using the target image 304, it is possible to suppress the influence on the recognition surface 303, that is, it is possible to suppress overlearning.

[0086] Next, the result display screen 801 will be described.

[0087] Figure 8 is a graph showing the result display screen according to one embodiment.

[0088] The result display screen 801, for example, is a screen displayed by the output section 208 in a case where additional learning is performed, and includes a recognition accuracy transition display region 802 and a learning image number transition display region 803.

[0089] In the recognition accuracy transition display region 802, a recognition accuracy transition chart in which the horizontal axis is set to the number of times of additional learning and the vertical axis is set to the recognition accuracy is displayed. This chart is, for example, created as follows: after the learning section 207 performs additional learning, the recognition section 206 performs recognition processing on the evaluation images, and stores the evaluation results thereof in a storage device or the like, and the output section 208 creates this chart based on the evaluation results. In the present embodiment, as the recognition accuracy transition chart, there are included a chart of the recognition accuracy of images for each facility (each image acquisition condition), such as a recognition accuracy chart for images captured in facility A and a recognition accuracy chart for images captured in facility B. According to these charts, the change in the recognition accuracy of images for facility A based on additional learning can be appropriately grasped, and it can be appropriately determined whether or not additional learning has a bad influence on the recognition accuracy of images for facility A.

[0090] In the learning image number transition display region 803, a learning image number transition chart in which the horizontal axis is set to the number of times of additional learning and the vertical axis is set to the number of learning images is displayed. This chart can be created, for example, by storing the number of images of the total of the number of basic learning images and the number of additional learning images at each additional learning, and creating the chart based on the number of images.

[0091] Next, the additional learning image display screen 901 will be described.

[0092] Figure 9 is a view that shows an additional learning image display screen according to an embodiment.

[0093] The additional learning image display screen 901, for example, is a screen displayed by the output section 208 in a case where an additional learning image is decided, and includes a candidate image display region 902 and an additional learning image indication icon 903.

[0094] A plurality of deformation coefficients and candidate images generated in correspondence with the deformation coefficients are displayed in the candidate image display region 902. The additional learning image indication icon 903 is an icon that indicates an image decided as an additional learning image among the candidate images displayed in the candidate image display region 902. In the present embodiment, the learning section 207 decides an additional learning image based on internal processing, but is not limited thereto, and for example, the learning section 207 can also decide as an additional learning image a candidate image selected by a user from the candidate images displayed in the candidate image display region 902 through the input device 120.

[0095] Next, the correct answer value setting screen 1001 will be explained.

[0096] Figure 10 This is a diagram showing the correct solution value setting screen of one implementation method.

[0097] The correct value setting screen 1001 is, for example, a screen displayed by the output unit 208 after the additional learning image has been determined. It includes an additional learning image display area 1002, a recognition result display area 1003, a correct value input area 1004, a display image change button 1005, and a registration button 1006.

[0098] The supplementary learning image display area 1002 is the area for displaying supplementary learning images. The recognition result display area 1003 is the area for displaying the recognition results of the supplementary learning images. The correct answer value input area 1004 is the area for accepting the input of correct answer values ​​(category information), which indicate the category to which the supplementary learning image is classified. The display image change button 1005 is a button for switching the supplementary learning images displayed in the supplementary learning image display area 1002. The registration button 1006 is a button for accepting an instruction to register by establishing a correspondence between the correct answer value and the supplementary learning image displayed in the supplementary learning image display area 1002. If the registration button 1006 is pressed, the output unit 208 stores the label of the correct answer value and the supplementary learning image displayed in the supplementary learning image display area 1002 in the storage device 113.

[0099] Furthermore, the present invention is not limited to the embodiments described above, and can be implemented with appropriate modifications without departing from the spirit of the present invention.

[0100] For example, in the above embodiment, by changing the deformation coefficient, the candidate image gradually approaches the base image from the object image, and the candidate image immediately preceding the change in the image recognition result is set as the additional learning image. However, the present invention is not limited to this. For example, the image of the image state between the candidate image and the object image can also be set as the additional learning image. In addition, the image corresponding to the deformation coefficient that is within a specified range relative to the deformation coefficient that has changed with respect to the recognition result can also be set as the additional learning image.

[0101] In addition, in the above-described embodiment, an example of generating an image for additional learning is shown, but an image for evaluation to be added can also be generated. For example, the image state feature amount calculation section 202 is caused to calculate an image state feature amount (4th image state feature amount) of an image state based on an image for evaluation (an example of an image for evaluation storage section) of a storage device, the shape feature amount calculation section 203 is caused to calculate a shape feature amount (2nd shape feature amount) of a shape of a subject based on the image for evaluation, the image state feature amount control section 204 is caused to adjust the image state feature amount (4th image state feature amount) to generate a new image state feature amount (5th image state feature amount), and the image generation section 205 is caused to generate a new image for evaluation based on the 5th image state feature amount and the 2nd shape feature amount. Further, the shape itself of the new image for evaluation is the same as that of the original image for evaluation, and thus the label of the new image for evaluation can be the same as that of the original image for evaluation.

[0102] In addition, in the above-described embodiment, as an example of the case where the image acquisition conditions differ between the 1st image acquisition condition and the 2nd image acquisition condition, a case where the facilities for capturing the images differ is shown, but the present application is not limited to this, and can be a case where the image acquisition conditions differ as long as they differ. For example, in a case where an image of an examination object for pathological examination of bacteria or cells is the recognition target of the recognition section, the state of the examination object differs depending on the mechanism for generating the examination object, the person in charge, and the like, and thus the state of the image obtained by capturing the examination object also differs, and thus this case can also be a case where the image acquisition conditions differ. In this case, for example, an image obtained by capturing an examination object generated by a person in charge A can be set as the image under the 1st image acquisition condition, and an image obtained by capturing an examination object generated by a person in charge B can be set as the image under the 2nd image acquisition condition. In addition, a case where the cameras for capturing differ can also be set as a case where the image acquisition conditions differ.

[0103] In addition, in the above-described embodiment, a part or all of the processing performed by the processor can also be performed by a hardware circuit. In addition, the program in the above-described embodiment can be installed from a program source. The program source can also be a program distribution server or a recording medium (for example, a removable recording medium).

[0104] Explanation of Reference Numerals 100 … machine learning system, 101 … machine learning device, 111 … processor, 112 … memory, 113 … storage device, 114 … bus, 120 … input device, 121 … display device, 201 … image input section, 202 … image state feature amount calculation section, 203 … portable information calculation section, 204 … image state feature amount control section, 205 … image generation section, 206 … recognition section, 207 … learning section, 208 … output section.

Claims

1. A machine learning aid device for generating additional learning images for additional learning of a recognition model, wherein the recognition model identifies the category to which the image is classified, wherein, The recognition model is learned using multiple basic images obtained under the conditions of the first image acquisition. The machine learning assistance device includes: The receiving department receives additional images obtained under the condition of obtaining the second image, which are the objects to be identified in the recognition model. The image state feature calculation unit calculates a second image state feature based on the image state of the appended image; The image state feature control unit generates one or more third image state features, which are features between the first image state features based on the image state of the base image and the second image state features. The morphological feature calculation unit calculates a first morphological feature based on the morphology of the subject in the additional image; The image generation unit generates one or more candidate images based on one or more of the third image state feature quantities and the first morphological feature quantities; and The additional learning image determination unit determines at least one of the candidate images as the additional learning image. The image state includes at least one of the following: brightness, contrast, hue, blur level, noise level, and resolution. The hue refers to hue and / or chroma.

2. The machine learning assist device according to claim 1, wherein, It also includes a recognition unit that uses the recognition model to identify the category to which the image is classified. The additional learning uses an image-based decision unit. The image state feature control unit causes the third image state feature to be changed sequentially from a value close to that of the second image state feature to a value close to that of the first image state feature. The image generation unit generates candidate images sequentially based on the third image state feature quantity and the morphological feature quantity. The recognition unit identifies the category to which the candidate image is classified. Based on the recognition result of the recognition unit, the additional learning image is determined.

3. The machine learning assist device according to claim 2, wherein, The additional learning image determination unit determines the candidate image generated immediately before the candidate image whose recognition result in the recognition unit has changed as the additional learning image.

4. The machine learning assist device according to claim 1, wherein, It also has: The perturbation image generation unit generates a perturbation image, which perturbs the image state, for each of the plurality of basic images acquired under the first image acquisition condition; and The image generation learning unit performs machine learning on the image state feature calculation unit, the morphological feature calculation unit, and the image generation unit, so that: (A) The basic image is similar to the image corresponding to the basic image. The image corresponding to the basic image is an image output by the image generation unit based on the image state feature quantity output by the image state feature quantity calculation unit for the basic image and the morphological feature quantity information output by the morphological feature quantity calculation unit for the basic image. (B) The perturbation image is similar to the image corresponding to the perturbation image, and the image corresponding to the perturbation image is an image output by the image generation unit based on the image state feature quantity output by the image state feature quantity calculation unit for the perturbation image and the morphological feature quantity information output by the morphological feature quantity calculation unit for the perturbation image; and (C) The morphological feature quantity information output by the morphological feature quantity calculation unit for the basic image is similar to the morphological feature quantity information output by the morphological feature quantity calculation unit for the perturbed image.

5. The machine learning assist device according to claim 1, wherein, It also includes a recognition model learning unit, which uses the additional learning images to learn the recognition model.

6. The machine learning assist device according to claim 5, wherein, It also has a learning result output unit, which outputs the learning results of the recognition model learning unit.

7. The machine learning assist device according to claim 1, wherein, It also includes an additional learning image output unit, which outputs the additional learning image in a recognizable manner.

8. The machine learning assist device according to claim 1, wherein, It also includes a candidate image output unit that outputs one or more of the candidate images.

9. The machine learning assist device according to claim 1, wherein, It also includes a receiving unit that selects an image from the more than one candidate images received by the user as the additional learning image. The additional learning image determination unit will accept the selected candidate image and determine it as the additional learning image.

10. The machine learning assist device according to claim 1, wherein, It also has: The category information receiving unit displays the additional learning image, and category information indicating the category for classifying the additional learning image from the user's acceptance; and The storage control unit stores the category information by establishing a correspondence between it and the additional learning images.

11. The machine learning assist device according to claim 1, wherein, It also has a shift information output unit that displays shift information and / or the number of learning images related to the shift in recognition accuracy caused by the additional learning of the recognition processing performed by the recognition unit.

12. The machine learning assist device according to claim 11, wherein, The shift information output unit displays shift information related to the shift in recognition accuracy of an image, which is an image obtained under the conditions of the image acquisition process of the recognition unit.

13. The machine learning assist device according to claim 1, wherein, It also includes an evaluation image storage unit that stores multiple evaluation images used to evaluate the recognition model. The image state feature calculation unit calculates the fourth image state feature based on the image state of the evaluation image. The morphological feature calculation unit calculates a second morphological feature based on the morphology of the subject in the evaluation image. The image state feature quantity control unit adjusts the fourth image state feature quantity to generate the fifth image state feature quantity. The image generation unit generates a new evaluation image based on the fifth image state feature and the second morphological feature.

14. A machine learning-assisted method, which is a machine learning-assisted method using a machine learning-assisted device, wherein the machine learning-assisted device generates an image for additional learning of a recognition model, the recognition model identifying the category to which the image is classified, wherein, The recognition model is learned using multiple basic images obtained under the conditions of the first image acquisition. In the machine learning-assisted method described above Accept the additional image obtained under the condition of obtaining the second image, which is the object to be identified in the recognition model. Calculate the second image state feature based on the image state of the appended image. Generate one or more third image state features, which are features between the first image state features and the second image state features based on the image state of the base image. Calculate the first morphological feature quantity of the subject's shape based on the additional image. Based on one or more of the third image state features and the first morphological features, one or more candidate images are generated. At least one of the candidate images is selected as an image for further learning. The image state includes at least one of the following: brightness, contrast, hue, blur level, noise level, and resolution. The hue refers to hue and / or chroma.

15. A machine learning aid program for execution by a computer to generate additional learning images for additional learning of a recognition model, said recognition model being used to identify the category to which the image is classified, wherein, The recognition model is learned using multiple basic images obtained under the conditions of the first image acquisition. The machine learning-assisted program enables the computer device to function as a structure in the following way: The receiving department receives additional images obtained under the condition of obtaining the second image, which are the objects to be identified in the recognition model. The image state feature calculation unit calculates a second image state feature based on the image state of the appended image; The image state feature control unit generates one or more third image state features, which are features between the first image state features based on the image state of the base image and the second image state features. The morphological feature calculation unit calculates a first morphological feature based on the morphology of the subject in the additional image; The image generation unit generates one or more candidate images based on one or more of the third image state feature quantities and the first morphological feature quantities; and The additional learning image determination unit determines at least one of the candidate images as the additional learning image. The image state includes at least one of the following: brightness, contrast, hue, blur level, noise level, and resolution. The hue refers to hue and / or chroma.

Images