Image processor and method for processing information

Abstract

To provide an image processor and a method for processing an image which can improve a specific evaluation value without increasing processing load of a learning image.SOLUTION: The method specifies a group according to an evaluation value which satisfies a stipulated condition of a plurality of evaluation values associated with an input image, of a plurality of groups obtained by classifying a learning image group, and expands data with a larger intensity for the specified group.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to learning techniques.

Background Art

[0002] As a method for solving tasks such as data detection, classification, and authentication, a detector or classifier is created using a convolutional network or the like, and the parameters of the convolutional network are updated by a learning process using learning data, thereby improving performance such as detection, classification, and authentication. is known. In general network learning methods, a huge amount of diverse learning data is required to achieve sufficient performance, so the generation cost of learning data is high. Therefore, methods for efficiently generating learning data and methods for pseudo-increasing learning data are known. In addition, methods for pseudo-increasing learning data are often called by expressions such as "data augmentation", "data augmentation", and "data padding". Hereinafter, a method for pseudo-increasing learning data will be referred to as "data augmentation".

[0003] For example, Patent Document 1 discloses a method of improving the accuracy of a prediction model by adding image data having a classification error larger than a predetermined threshold to learning data. Non-Patent Document 1 also discloses a method of automatically discovering an effective data augmentation strategy for a target learning dataset by reinforcement learning.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] Consider the case of evaluating multiple evaluation metrics using a machine learning model. For example, when evaluating the amount of image degradation (PSNR) in a noise reduction task, it is the case of evaluating PSNR for each of a plurality of regions in the image. Other cases include evaluating the accuracy rate for each class in an object detection task.

[0007] Thus, when a machine learning model evaluates multiple evaluation metrics, a certain level or higher of evaluation score may be required for any of the multiple evaluation metrics. If the evaluation score is low for a specific evaluation metric, it is necessary to improve the corresponding evaluation score.

[0008] In Patent Document 1, since the configuration adds image data with a large classification error to the learning data, the more image data with a large classification error there is, the more the learning data increases, and the time required for processing the learning data increases. Also, since the image data is added to the learning data without performing processing including image processing, there is a possibility that the effect on the image data with a large classification error is not sufficient.

[0009] In Non-Patent Document 1, since reinforcement learning is used to comprehensively explore data augmentation methods, probabilities, and intensities, the amount of calculation becomes enormous. Also, although an improvement in overall accuracy can be expected, the accuracy for a specific image / region is not necessarily improved. For example, it is also possible that the accuracy of the good-at images / regions is further improved, while the accuracy of the bad-at images / regions is not improved much. The present invention provides a technique capable of improving a specific evaluation value without increasing the processing load of the learning images.

Means for Solving the Problem

[0010] In one aspect of the present invention, among a plurality of groups obtained by classifying a group of learning images, specific means for specifying a group corresponding to an evaluation value that satisfies a specified condition among a plurality of evaluation values related to an input image, and for the group specified by the specific means, an expansion means for performing data expansion with a stronger intensity are provided.

Effect of the Invention

[0011] According to the present invention, a specific evaluation value can be improved without increasing the processing load of learning images.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Mode for Carrying Out the Invention

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0014] [First Embodiment] In this embodiment, a learning device that performs data augmentation on a learning image group used for learning a machine learning model (hereinafter simply referred to as a model) that executes a noise reduction task will be described. Here, the noise reduction task is a task of estimating a noise-free image (pre-degradation image) before degradation by noise from a noisy image (degraded image) degraded by noise.

[0015] A functional configuration example of the learning device 100 according to this embodiment is shown in the block diagram of FIG. 1. The operation of the learning device 100 having the functional configuration example shown in FIG. 1 will be described according to the flowchart of FIG. 2.

[0016] In step S101, the classification unit 140 acquires a learning image group as a learning data set. The method of acquiring the learning image group is not limited to a specific acquisition method, and it may be acquired from a memory device in the learning device 100, or may be acquired from a device outside the learning device 100 via a wired or wireless network. Then, the classification unit 140 classifies (groups by feature) the acquired learning image group into a plurality of groups according to features. Various methods can be applied to classify (group by feature) the learning image group according to features, and it is not limited to a specific method.

[0017] For example, the classification unit 140 calculates a feature vector from each learning image in the learning image group (learning dataset). As shown in FIG. 4(a), when the feature vector is a three-dimensional vector, the feature vector of each learning image is represented by a point 201 in the feature space to which the feature vector belongs (in this case, a three-dimensional space defined by the x-axis, y-axis, and z-axis). Then, the classification unit 140 applies a well-known clustering technique such as the k-means method to the feature vectors of a plurality of learning images to generate a plurality of clusters. Thereby, groups corresponding to each cluster can be generated, such as a group of learning images corresponding to the feature vectors of the first cluster and a group of learning images corresponding to the feature vectors of the second cluster.

[0018] Also, for example, the classification unit 140 extracts a two-dimensional feature vector having two components (frequency and contrast) from each learning image in the learning image group (learning dataset). As shown in FIG. 4(b), in a two-dimensional plane with the vertical axis being frequency and the horizontal axis being contrast, the two-dimensional feature vector of each learning image will be classified into any one of four events in this two-dimensional plane. In this case, the learning images corresponding to the two-dimensional feature vectors classified in the first quadrant constitute a high-frequency high-contrast image group, and the learning images corresponding to the two-dimensional feature vectors classified in the second quadrant constitute a high-frequency low-contrast image group. Also, the learning images corresponding to the two-dimensional feature vectors classified in the third quadrant constitute a low-frequency low-contrast image group, and the learning images corresponding to the two-dimensional feature vectors classified in the fourth quadrant constitute a low-frequency high-contrast image group. That is, in this method, the classification unit 140 can classify each learning image into any one of a group corresponding to the first event, a group corresponding to the second event, a group corresponding to the third event, and a group corresponding to the fourth event.

[0019] In step S102, the learning unit 150 performs model learning processing using a part of the learning image group. The learning unit 150 may perform model learning using a plurality of learning images randomly selected from the learning image group, or may perform model learning using a plurality of predetermined learning images from the learning image group. In the second and subsequent steps S102, the learning unit 150 selects a learning image corresponding to the area to be learned and performs model learning.

[0020] In step S103, the learning unit 150 acquires an evaluation image. The evaluation image may be a preset image or an image set by the user. Then, the learning unit 150 inputs the evaluation image into the model that has undergone the learning process in step S102 and performs the operation of the model, thereby obtaining (calculating) an evaluation value (evaluation score) for each area in the evaluation image. In this embodiment, PSNR (Peak Signal-to-Noise Ratio) is used as the evaluation value, but other types of evaluation values such as MSE (Mean Squared Error) and SSIM (Structural SIMilarity) (hereinafter, the evaluation value is also referred to as the "evaluation score") may be used.

[0021] Note that the process of step S103 may be executed at a predetermined timing. As the "predetermined timing", periodic indicators such as time and epoch may be used, and the indicator and timing to be used may be arbitrarily specified by the user, for example.

[0022] Also, the "area in the evaluation image" may be the entire area in the evaluation image or a partial area in the evaluation image. The "area in the evaluation image" may be, for example, each area when the evaluation image is divided into a plurality of rectangular areas, or each area divided by a segmentation technique such as semantic segmentation.

[0023] In step S104, the discrimination unit 110 determines whether or not there are N or more PSNRs less than the threshold value among the PSNRs obtained in step S103. N is an integer of 1 or more, and any value can be set in advance.

[0024] Also, as the threshold value, when there is a target model, the PSNR for each region of the model may be set, or the user may set an arbitrary value. An example of the evaluation score (PSNR), the corresponding threshold value, and the determination result of the magnitude relationship between the evaluation score and the threshold value ( "×" indicates that the evaluation score < the threshold value, and "○" indicates that the evaluation score ≧ the threshold value) in the region of each of the evaluation images A, B, and C is shown in FIG. 6(a).

[0025] As a result of such determination, if there are N or more PSNRs less than the threshold value among the PSNRs obtained in step S103, the process proceeds to step S105. On the other hand, if there are not N or more PSNRs less than the threshold value among the PSNRs obtained in step S103, the process according to the flowchart of FIG. 2 ends.

[0026] Note that when there are N or more PSNRs less than the threshold value among the PSNRs obtained in step S103, the discrimination unit 110 may inquire of the user whether to end the process according to the flowchart of FIG. 2 or to perform the processes after step S105. In this case, the learning device 100 performs a process according to an instruction from the user in response to this inquiry.

[0027] In step S105, the selection unit 120 uses the region for which the PSNR less than the threshold value is obtained as the target region, and classifies the target region into any one of the above-mentioned plurality of groups for each feature in the same manner as in step S101 above. Then, the selection unit 120 selects (identifies) the group to which the target region is classified (the group corresponding to the feature most similar to the feature of the target region) as the target group.

[0028] For example, in step S101 described above, when the learning image group is classified by feature by applying clustering technology to the feature vectors of learning images to generate a plurality of clusters, the selection unit 120 obtains the feature vector of the target area. Then, the selection unit 120 calculates the similarity between the feature vector of each learning image and the feature vector of the target area, and identifies, as the target group, the group corresponding to the cluster to which the "feature vector of the learning image" with the minimum similarity belongs.

[0029] As the similarity, for example, indicators such as cosine similarity or Euclidean distance between feature vectors can be used. The cosine similarity between the feature vector x = {x1, x2, …, xn} and the feature vector y = {y1, y2, …, yn} is defined by the following formula.

[0030] [Number]

[0031] When using cosine similarity as the similarity, the "minimum similarity" means the "cosine similarity closest to 1". Note that the selection unit 120 may calculate, for each cluster, the average vector that is the average of the feature vectors belonging to the cluster, calculate the similarity between each average vector and the feature vector of the target area, and identify, as the target group, the group corresponding to the cluster to which the average vector with the minimum similarity belongs. Fig. 5(a) shows a state in which a point 202 corresponding to the feature vector of the target area belongs to a cluster 203 in the feature space. In this case, the selection unit 120 selects the group corresponding to the cluster 203 as the target group.

[0032] For example, in the above step S101, when the learning image group is classified by feature based on the two-dimensional feature vector of the learning image, the selection unit 120 extracts a two-dimensional feature vector having two components (frequency and contrast) from the target area. Then, among the above four events, the selection unit 120 identifies the event to which the two-dimensional feature vector of the target area belongs, and identifies the group corresponding to the identified event. FIG. 5(b) shows a state in which a point 303 corresponding to the two-dimensional feature vector of the target area belongs to the upper left event in the two-dimensional plane. In this case, the selection unit 120 selects the group corresponding to the upper left event as the target group.

[0033] In step S106, the generation unit 130 sets to perform data augmentation on the target group with a stronger intensity, and performs data augmentation on the target group based on the setting. For example, as shown in FIG. 7, in the initial stage, in terms of intensity setting, the intensity of the data augmentation method of "shearing" is set to 20, but as a result of setting to perform data augmentation with a stronger intensity, the intensity of the data augmentation method of "shearing" is changed to a value of 30. Here, "shearing" is taken as an example, but the data augmentation method for which the data augmentation intensity is increased is not limited to "shearing", and for example, rotation, enlargement / reduction, noise addition, etc. may also be used.

[0034] Then, in the next step S102, since learning is performed based on the learning image group with the increased data augmentation intensity, it becomes possible to improve the PSNR of the area where the PSNR was less than the threshold.

[0035] [Second Embodiment] In each of the following embodiments including this embodiment, the differences from the first embodiment will be described, and unless otherwise specifically mentioned below, it shall be the same as the first embodiment. In this embodiment, a learning device that performs data augmentation on a learning image group used for learning a machine learning model (hereinafter simply referred to as a model) that executes an object detection task will be described. Here, the object detection task is a task of identifying in which region of the input image the target object identified from the input image is located and to which class it is classified. The operation of the learning device 100 according to this embodiment will be described according to the flowchart of FIG. 3.

[0036] In step S201, the classification unit 140 acquires a learning image group as a learning data set and classifies the learning image group into a plurality of groups (classes) according to correct labels. Object information such as the name, position, and size of the object in each learning image is pre-assigned as a correct label. The classification unit 140 classifies the learning image group for each class of similar object information.

[0037] In step S202, the learning unit 150 performs the model learning process using a part of the learning image group in the same manner as in step S102 above. In the second and subsequent steps S202, the learning unit 150 selects learning images according to the class and performs model learning.

[0038] In step S203, the learning unit 150 acquires an evaluation image that is an image belonging to each of the above plurality of classes for each of the above plurality of classes. Then, for each of the above plurality of classes, the learning unit 150 inputs the evaluation image belonging to the class into the model for which the learning process was performed in step S202 and performs the operation of the model, thereby calculating the likelihood that the evaluation image belongs to the class (the correct classification rate of class classification) as an evaluation value. Note that the process of step S203 may be executed at a predetermined timing in the same manner as in step S103 above.

[0039] In step S204, the discrimination unit 110 determines whether or not there are N or more correct answer rates less than the threshold among the correct answer rates obtained in step S203. As a result of such determination, if there are N or more correct answer rates less than the threshold among the correct answer rates obtained in step S203, the process proceeds to step S205. On the other hand, if there are not N or more correct answer rates less than the threshold among the correct answer rates obtained in step S203, the process according to the flowchart of FIG. 3 ends.

[0040] An example of the evaluation score (correct answer rate), the corresponding threshold value, and the determination result of the magnitude relationship between the evaluation score and the threshold value (「×」 indicates that the evaluation score < the threshold value, and 「○」 indicates that the evaluation score ≧ the threshold value) in each of the classes of class A, class B, and class C is shown in FIG. 6(b).

[0041] Note that when there are N or more correct answer rates less than the threshold among the correct answer rates obtained in step S203, the discrimination unit 110 may inquire of the user whether to end the process according to the flowchart of FIG. 3 or to perform the processes after step S205. In this case, the learning device 100 performs a process according to an instruction from the user in response to this inquiry.

[0042] In step S205, the selection unit 120 selects (identifies) a class for which the correct answer rate less than the threshold was obtained (the class of the evaluation image for which the correct answer rate is less than the threshold) from among the plurality of classes described above as the target class (target group).

[0043] In step S206, the generation unit 130 makes a setting to perform data augmentation with a stronger intensity for the target group, and performs data augmentation for the target group based on this setting.

[0044] Thus, according to the present embodiment, by using learning data with an increased data augmentation intensity for learning, it becomes possible to improve the correct answer rate of a class for which the correct answer rate was less than a predetermined threshold.

[0045] [Third Embodiment] Even if the processes of steps S102 to S106 in FIG. 2 are repeatedly performed, the number of regions where the PSNR is less than the threshold may not be less than N. Also, even if the processes of steps S202 to S206 in FIG. 3 are repeatedly performed, all the correct answer rates may not be equal to or higher than the threshold. These are both due in part to the fact that the learning images necessary to achieve each task with sufficient accuracy are not sufficient.

[0046] Therefore, for example, when the number of regions where the PSNR is less than the threshold does not fall below a specified number even if the processes of steps S102 to S106 are repeatedly performed, the learning device 100 may request the user to add a learning image having the characteristics of the region where PSNR < threshold.

[0047] Also, for example, when all the correct answer rates do not become equal to or higher than the threshold even if the processes of steps S202 to S206 in FIG. 3 are repeatedly performed, the learning device 100 may request the user to add a learning image corresponding to a class with a correct answer rate less than the threshold.

[0048] And when the learning device 100 obtains a request for adding a learning image from the user, it performs a process for adding the learning image. Note that the learning device 100 may perform a process for adding a learning image without a request from the user.

[0049] Also, in the above-described embodiment, the case where PSNR or the correct answer rate is used as the evaluation value has been described. However, what is used as the evaluation value may vary depending on the task or the like. Therefore, in the above-described embodiment, the target group is specified by determining the magnitude relationship between the evaluation value and the threshold. However, the method for specifying the target group may also vary depending on the task or the like. That is, among the plurality of groups obtained by classifying the learning image group, various methods can be considered for specifying the group corresponding to the evaluation value that satisfies the specified conditions among the plurality of evaluation values related to the input image.

[0050] [Fourth Embodiment] Each functional unit of the learning device 100 shown in FIG. 1 may be implemented in hardware or software. In the latter case, a computer device capable of executing such software is applicable to the learning device 100. A hardware configuration example of a computer device applicable to the learning device 100 will be described with reference to the block diagram of FIG. 8.

[0051] The CPU 801 executes various processes using the computer programs and data stored in the RAM 802. Thereby, the CPU 801 controls the operation of the entire computer device and executes or controls various processes described as the processes performed by the learning device 100.

[0052] The RAM 802 has an area for storing computer programs and data loaded from the ROM 803 and the storage unit 806, and an area for storing computer programs and data received from the outside via the I / F 807. Further, the RAM 802 has a work area used when the CPU 801 executes various processes. In this way, the RAM 802 can appropriately provide various areas.

[0053] The ROM 803 stores setting data of the computer device, computer programs and data related to the startup of the computer device, computer programs and data related to the basic operation of the computer device, and the like.

[0054] The operation unit 804 is a user interface such as a keyboard, a mouse, or a touch panel screen, and various instructions and information can be input to the computer device by the user's operation.

[0055] The display unit 805 is a device having a screen such as a liquid crystal screen or a touch panel screen, and can display the processing result by the CPU 801 as an image, characters, or the like. Note that the display unit 805 may be a projection device such as a projector that projects an image or characters.

[0056] The storage unit 806 is a large-capacity non-volatile memory such as a hard disk drive device. The storage unit 806 stores an operating system, computer programs and data for causing the CPU 801 to execute or control various processes described as processes performed by the learning device 100, and the like.

[0057] The I / F 807 is a communication interface for performing data communication with an external device. For example, the computer device can receive various information such as a learning image group, model parameters (such as weight values), and learning parameters from an external device via the I / F 807.

[0058] The CPU 801, the RAM 802, the ROM 803, the operation unit 804, the display unit 805, the storage unit 806, and the I / F 807 are all connected to the system bus 808. Note that the configuration shown in FIG. 8 is merely an example of the hardware configuration of a computer device applicable to the learning device 100, and can be appropriately modified / changed.

[0059] The numerical values, processing timings, processing orders, processing entities, methods of acquiring / sending destinations / sending sources / storage locations of data (information), etc. used in the above-described embodiments are given as examples for the purpose of specific description, and are not intended to be limited to such examples.

[0060] Also, some or all of the above-described embodiments may be used in appropriate combination. Also, some or all of the above-described embodiments may be selectively used.

[0061] (Other Embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and causing one or more processors in the computer of the system or device to read and execute the program. It can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

[0062] The invention described in this specification includes the following image processing apparatus, image processing method, and computer program. (Item 1) Among a plurality of groups obtained by classifying a learning image group, specific means for specifying a group corresponding to an evaluation value that satisfies a specified condition among a plurality of evaluation values related to an input image; Expansion means for performing data expansion with a stronger intensity on the group specified by the specific means; An image processing apparatus characterized by comprising the above. (Item 2) The specific means according to item 1, wherein the specific means specifies, among the plurality of groups, a group corresponding to the characteristics of a region in which an evaluation value less than a threshold value is calculated among the evaluation values calculated for each region in the input image. (Item 3) Furthermore, Learning means for performing learning of a model using a part of the learning image group; The specific means according to item 2, wherein the specific means specifies, among the plurality of groups, a group corresponding to the characteristics of a region in which an evaluation value less than a threshold value is calculated among the evaluation values calculated by the model for each region in the input image. (Item 4) Furthermore, When the number of regions in which an evaluation value less than a threshold value is calculated does not fall below a specified number, the image processing apparatus according to item 2 or 3, further comprising request means for requesting the user to add a learning image having the characteristics of the region in which an evaluation value less than a threshold value is calculated. (Item 5) The specific means according to item 1, wherein the specific means specifies, among the plurality of groups, a group of input images whose classification accuracy rate is less than a threshold value. (Item 6) Furthermore, Learning means for performing learning of a model using a part of the learning image group; The specific means according to item 5, wherein the specific means specifies, among the plurality of groups, a group of input images whose classification accuracy rate calculated by the model is less than a threshold value. (Item 7) Furthermore, When the correct answer rates of all groups do not reach the threshold value, the image processing apparatus according to item 5 or 6, characterized in that it comprises a requesting means for requesting the user to add learning images corresponding to the groups with correct answer rates less than the threshold value. (Item 8) An image processing method performed by an image processing apparatus, Among a plurality of groups obtained by classifying a learning image group, a specifying step of specifying a group corresponding to an evaluation value that satisfies a specified condition among a plurality of evaluation values related to an input image by a specifying means of the image processing apparatus; An expanding step of performing data expansion with a stronger intensity on the group specified in the specifying step by an expanding means of the image processing apparatus An image processing method characterized by comprising. (Item 9) A computer program for causing a computer to function as each means of the image processing apparatus according to any one of items 1 to 7.

[0063] The invention is not limited to the above embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, claims are attached to disclose the scope of the invention.

Explanation of Signs

[0064] 100: Learning apparatus 110: Discrimination unit 120: Selection unit 130: Generation unit 140: Classification unit 150: Learning unit

Claims

1. Among a plurality of groups obtained by classifying a learning image group, a specifying means for specifying a group corresponding to an evaluation value that satisfies a specified condition among a plurality of evaluation values related to an input image; An expanding means for performing data expansion with a stronger intensity on the group specified by the specifying means An image processing apparatus characterized by comprising the same.

2. The specifying means according to claim 1, wherein among the plurality of groups, a group corresponding to a feature of a region in which an evaluation value less than a threshold value is calculated among the evaluation values calculated for each region in the input image is specified. The described image processing apparatus.

3. Furthermore, A learning means for performing learning of a model using a part of the learning image group is provided, The specifying means according to claim 2, wherein among the plurality of groups, a group corresponding to a feature of a region in which an evaluation value less than a threshold value is calculated among the evaluation values calculated by the model for each region in the input image is specified. The described image processing apparatus.

4. Furthermore, When the number of regions in which evaluation values less than the threshold value are calculated does not fall below a specified number, a requesting means for requesting the user to add a learning image having the feature of the region in which the evaluation value less than the threshold value is calculated is provided. The image processing apparatus according to claim 2, characterized in that it is provided.

5. The specifying means according to claim 1, wherein among the plurality of groups, a group of input images whose correct classification rate is less than a threshold value is specified. The described image processing apparatus.

6. Furthermore, A learning means for performing learning of a model using a part of the learning image group is provided, The specifying means according to claim 5, wherein among the plurality of groups, a group of input images whose correct classification rate calculated by the model is less than a threshold value is specified. The described image processing apparatus.

7. Furthermore, When the correct classification rate of all groups does not become equal to or higher than the threshold value, a requesting means for requesting the user to add a learning image corresponding to the group whose correct classification rate is less than the threshold value is provided. The image processing apparatus according to claim 5, characterized in that it is provided.

8. An image processing method performed by an image processing apparatus, A specifying step in which the specifying means of the image processing apparatus specifies a group corresponding to an evaluation value that satisfies a specified condition among a plurality of evaluation values related to an input image among a plurality of groups obtained by classifying a learning image group; An expanding step in which the expanding means of the image processing apparatus performs data expansion with a stronger intensity on the group specified in the specifying step An image processing method characterized by comprising

9. A computer program for causing a computer to function as each means of the image processing apparatus according to any one of Claims 1 to 7.

Images