Information processing device, information processing method, and program

Abstract

To provide an information processing apparatus for training an image identification model in consideration of a degree of similarity between an original image and a converted image.SOLUTION: An information processing apparatus (1) includes: a learning data generating unit (12) which generates converted images from original images using image conversion parameters and assigns, to the generated converted images, pseudo classes corresponding to classes of the original images and the image conversion parameters, respectively; a loss-for-learning determining unit (13) which generates an objective function in which correction terms are reflected, in consideration of a degree of similarity between a class and a pseudo classes, the correction terms being determined for the respective image conversion parameters; and a learning unit (14) which learns an image identification model using the converted images and the objective function.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Techniques for applying image recognition processing to a target image are known. Further, a learning data generation device capable of mechanically generating learning data for causing a model to perform machine learning for identifying an image is known. For example, Patent Document 1 discloses a pattern identifier based on supervised learning, which includes a data augmentation unit that performs data augmentation on unknown data such as image data in the same manner as the data augmentation performed on training data when generating an identification model to generate pseudo-unknown data, and an identification unit that applies the pseudo-unknown data to the identification model and performs class classification based on the expected values of the results. According to this technique, after expanding the training data for learning, when identifying unknown data, it is said that the rule (decision rule) regarding which class to assign the unknown data to can be improved, thereby improving the identification performance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, generating new images suitable for training product identification models is difficult. For example, training images for product identification can be generated by applying processes such as color conversion, text conversion, and horizontal (left-right) flipping to existing product images. However, it has been found that models trained using such images sometimes fail to perform image recognition that would seem easy to a user. The inventors investigated and concluded that this is because, during the training phase, the images that underwent image conversion processing were generated as data that could not be considered to belong to the same class of products, and these were used for training.

[0005] Furthermore, when a pseudo-new product class was assigned to an image that had undergone image transformation, it was suspected that overfitting, due to excessive optimization of the training data, was the cause, as the model was trained to identify them as different types of products without considering the degree of similarity between the transformed image obtained through image transformation and the original image.

[0006] One aspect of the present invention has been made in view of the above-mentioned problems, and one example of its objective is to provide a technique for training an image recognition model by taking into account the degree of similarity between the original image and the transformed image. [Means for solving the problem]

[0007] An information processing device according to one aspect of the present invention includes: a learning data generation means that generates a transformed image from an original image using image transformation parameters and assigns a pseudo-class to the generated transformed image according to the class of the original image and the image transformation parameters; a learning loss determination means that generates an objective function that reflects a correction term that takes into account the degree of similarity between the class and the pseudo-class determined for each image transformation parameter; and a learning means that trains an image recognition model using the transformed image and the objective function.

[0008] An information processing method according to one aspect of the present invention includes: at least one processor generating a transformed image from an original image using image transformation parameters; assigning a pseudo-class to the generated transformed image according to the class of the original image and the image transformation parameters; generating an objective function that reflects a correction term that takes into account the degree of similarity between the class and the pseudo-class determined for each image transformation parameter; and training an image recognition model using the transformed image and the objective function.

[0009] An information processing program according to one aspect of the present invention causes a computer to perform the following processes: generate a converted image from an original image using image conversion parameters; assign a pseudo-class to the generated converted image according to the class of the original image and the image conversion parameters; generate an objective function that reflects a correction term that takes into account the degree of similarity between the class and the pseudo-class determined for each image conversion parameter; and train an image recognition model using the converted image and the objective function. [Effects of the Invention]

[0010] According to one aspect of the present invention, an image recognition model can be trained by taking into account the degree of similarity between the original image and the transformed image. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing the configuration of an information processing device 1 according to an exemplary embodiment 1 of the present invention. [Figure 2] This is a flowchart showing the flow of the information processing method S1 according to exemplary embodiment 1. [Figure 3] This is a block diagram showing the configuration of the information processing device 1A according to exemplary embodiment 2 of the present invention. [Figure 4] This is an example showing the process by which the learning data generation unit according to Exemplary Embodiment 2 generates a converted image from the original image. [Figure 5] This is another example showing the process by which the learning data generation unit according to Exemplary Embodiment 2 generates a converted image from the original image. [Figure 6] It is a conceptual diagram showing the difference in the degree of similarity between the original image and the converted image, or the degree of similarity between the converted images. [Figure 7] It is a table showing an example of setting the margin intensity of the converted image. [Figure 8] It is a table showing another example of setting the margin intensity of the converted image. [Figure 9] It is a conceptual diagram showing the concept that the learning loss determination unit adjusts the distance considering the margin intensity. [Figure 10] It is a conceptual diagram showing the concept of another example where the learning loss determination unit adjusts the distance considering the margin intensity. [Figure 11] It is a table showing an example where the learning loss determination unit determines different patterns of margin intensity for the converted image according to each parameter. [Figure 12] It is a block diagram showing the configuration of the information processing apparatus 1B according to the exemplary embodiment 3 of the present invention. [Figure 13] It is a configuration diagram for realizing the information processing apparatus by software. [Figure 14] It is a conceptual diagram showing the concept of distance metric learning in the prior art.

Mode for Carrying Out the Invention

[0012] 〔Exemplary Embodiment 1〕 The first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described later.

[0013] (Configuration of Information Processing Apparatus 1) The configuration of the information processing apparatus 1 according to this exemplary embodiment will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the information processing apparatus 1.

[0014] As shown in FIG. 1, the information processing apparatus 1 includes a learning data generation unit 12, a learning loss determination unit 13, and a learning unit 14. The learning data generation unit 12, the learning loss determination unit 13, and the learning unit 14 are each a form of the learning data generation means, the learning loss determination means, and the learning means described in the claims.

[0015] The learning data generation unit 12 generates a converted image from the original image using image conversion parameters. In this exemplary embodiment, the original image is an existing image, and as an example, it is an image of a product. The original image is assigned an original class, and different original classes are pre-assigned to original images of different products. The image conversion parameters are parameters indicating the method of image conversion, and different parameters are set for each method. The parameters are merely for distinguishing the methods, and their types are not limited. Examples of the image conversion method include, but are not limited to, color conversion, character conversion, horizontal flip, etc. Color conversion is a hue conversion (rotation) that changes the hue of an image. Character conversion is a conversion that changes the character string included in an image to another character string. Horizontal flip is a conversion that changes an image to a horizontally symmetric image. Also, as an option for the image conversion method, there may be a selection of "no conversion".

[0016] The learning data generation unit 12 selects any one of the parameters indicating color conversion, character conversion, horizontal flip, etc., and generates a converted image using that method. The method for selecting the parameters is arbitrary. Further, the learning data generation unit 12 may assign a pseudo-class corresponding to the original class and the conversion parameters to the generated converted image. The pseudo-class is a newly assigned class different from the original class, but since the original image and the converted image have similar parts, it is referred to as a pseudo-class in this exemplary embodiment. When the learning data generation unit 12 selects the conversion method of "no conversion", it does not generate a converted image. Alternatively, when the conversion method of "no conversion" is selected, the learning data generation unit 12 may generate an image that has not been converted as the converted image and assign the same class as the class of the original image.

[0017] The learning loss determination unit 13 generates an objective function that reflects a correction term that takes into account the degree of similarity between the classes and pseudo-classes. More specifically, the learning loss determination unit 13 generates an objective function derived including a correction term set by the user or the information processing device 1. The learning loss determination unit 13 may also calculate the objective function. The objective function is a value that evaluates the recognition result of an image recognition model that identifies images of all classes, including the original class and pseudo-classes, and its type is not limited. As an example, the objective function may be a loss function that shows the magnitude of the error between the ideal output for the input data and the actual output.

[0018] The correction term that considers the degree of similarity between the original image and the transformed image is a correction term added to the objective function depending on how similar, or conversely, how different, the two images look. This correction term can be set appropriately according to the method for evaluating image similarity. Hereafter, the degree of similarity will also be referred to as similarity. As an example, image similarity may be evaluated as the distance in the multidimensional feature vector space output by a convolutional neural network under training, with the images as input. In this case, the greater the similarity between the two images, the shorter the distance between the two images is evaluated. Conversely, the smaller the similarity between the two images, the longer the distance between the two images is evaluated. As for the objective function, for example, an objective function may be set so that the distance between feature vectors is small for images of the same class, and large for images of different classes. Also, as a correction term, a correction term may be set that allows the distance between feature vectors of an original class and a pseudo-class that look similar to each other to be smaller than the distance between feature vectors of other classes and pseudo-classes generated from other classes. By selecting an appropriate objective (loss) function, the network model is trained so that images of the same class are placed at relatively close distances, and images of different classes are placed at relatively far distances. Therefore, in this exemplary embodiment, the correction term added to the loss function is set so that it can be transformed into the distance relationship intended by the trained model.

[0019] In this exemplary embodiment, the degree of similarity between the original image and the transformed image can be changed or specified. The image recognition model can then be trained to include the specified degree of similarity. In other words, according to the information processing device 1 of this exemplary embodiment 1, for two images with high similarity, the image recognition model can be trained so that there is a high relationship of similarity between the original class and the pseudo-class generated from the original class. Conversely, for two images with low similarity, the image recognition model can be trained so that there is a low relationship of similarity between the original class and the pseudo-class. This allows the generated training data to be used more effectively for training. Furthermore, by selecting transformation parameters that take into account the characteristics of the product image, the ability to identify unknown products can be enhanced.

[0020] The learning unit 14 trains an image recognition model using the transformed images generated by the learning data generation unit 12 and the objective function generated by the learning loss determination unit 13. As described above, the objective function takes into account a correction term that indicates the degree of similarity. The learning unit 14 then trains the image recognition model so that the objective function, taking the correction term into account, approaches a target position such as the maximum or minimum value. As an example, the image recognition model is a neural network model using a convolutional neural network or the like. The neural network model is learned by updating the weights, biases, etc., inside the model so that the calculation result of the objective function calculated using its output moves towards a desirable position. As an example, the objective function in a neural network model is a loss function aimed at minimization. Specifically, this can be expressed as a loss function expressed as softmax cross-entropy error or sum of squared errors. The learning unit 14 updates the weights, biases, etc., of the image recognition model according to the calculation result of the objective function (loss function) derived by the learning loss determination unit 13.

[0021] Although Figure 1 shows the information processing device 1 with the learning data generation unit 12, the learning loss determination unit 13, and the learning unit 14 all located in one place, the system is not limited to this configuration. In other words, at least some of the components constituting the information processing device 1 may be distributed and arranged so that they can communicate with each other. Alternatively, at least some of the components constituting the information processing device 1 may be located on a cloud so that they can communicate with each other.

[0022] As described above, the information processing device 1 according to this exemplary embodiment employs a configuration comprising: a learning data generation unit 12 that generates a transformed image from an original image using image transformation parameters and assigns a pseudo-class to the generated transformed image according to the class of the original image and the image transformation parameters; a learning loss determination unit 13 that generates an objective function that reflects a correction term that takes into account the degree of similarity between the class and the pseudo-class determined for each image transformation parameter; and a learning unit 14 that trains an image recognition model using the transformed image and the objective function. Therefore, according to the information processing device 1 according to this exemplary embodiment, the effect is obtained that an image recognition model can be trained while taking into account the degree of similarity between the original image and the transformed image. Furthermore, by changing the magnitude of the correction term during training, the training data can be used more effectively.

[0023] (Information processing flow) The flow of the information processing method S1 according to this exemplary embodiment will be explained with reference to Figure 2. Figure 2 is a flowchart showing the flow of the information processing method S1.

[0024] As shown in Figure 2, the information processing method S1 includes steps S11 to S13. In step S11, at least one processor (for example, the learning data generation unit 12) generates a transformed image from the original image using image transformation parameters, and assigns a pseudo-class to the generated transformed image according to the class of the original image and the image transformation parameters. The image transformation parameters are as described in the configuration of the information processing device 1.

[0025] Furthermore, in step S12, at least one processor (for example, the learning loss determination unit 13) generates an objective function that reflects a correction term that takes into account the degree of similarity between the class and the pseudo-class determined for each image transformation parameter. The objective function and the correction term that takes into account the degree of similarity are as described in the configuration of the information processing device 1.

[0026] Furthermore, in step S13, at least one processor (for example, the learning unit 14) trains an image recognition model using the transformed image and the objective function. The image recognition model is as described in the configuration of the information processing device 1.

[0027] As described above, the information processing method S1 according to this exemplary embodiment employs a configuration in which at least one processor generates a transformed image from the original image using image transformation parameters, assigns a pseudo-class to the generated transformed image according to the class of the original image and the image transformation parameters, generates an objective function that reflects a correction term that takes into account the degree of similarity between the class and the pseudo-class determined for each image transformation parameter, and trains an image recognition model using the transformed image and the objective function. Therefore, according to the information processing method S1 according to this exemplary embodiment, the effect is obtained that an image recognition model can be trained while taking into account the degree of similarity between the original image (original class) and the transformed image (pseudo-class). Furthermore, by changing the correction term and training, the training data can be used more effectively.

[0028] [Exemplary Embodiment 2] A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Components having the same function as those described in Exemplary Embodiment 1 will be denoted by the same reference numerals, and their descriptions will be omitted as appropriate. Figure 3 is a block diagram showing the configuration of the information processing device 1A according to Exemplary Embodiment 2.

[0029] The information processing device 1A according to Exemplary Embodiment 2 comprises an acquisition unit 10, an image conversion parameter storage unit 11, a learning data generation unit 12, a learning loss determination unit 13, a learning unit 14, an original image storage unit 15, and a converted image storage unit 16. The information processing device 1A is connected to a product identification model M in a communication manner. The product identification model M is a form of an image identification model. The functions of the learning data generation unit 12, the learning loss determination unit 13, and the learning unit 14 are the same as those of the respective units described in Exemplary Embodiment 1, so their description is omitted here. The acquisition unit 10, the image conversion parameter storage unit 11, the original image storage unit 15, and the converted image storage unit 16 will be described below.

[0030] The acquisition unit 10 acquires the source image. The class of the source image is referred to as the source class. In this exemplary embodiment, the source image is, for example, an image of a single product. Different classes are pre-assigned to images of different products. For example, the acquisition unit 10 acquires an image input by the user to the information processing device 1. Alternatively, the acquisition unit 10 may access an external storage device of the information processing device 1 and acquire a stored image.

[0031] The image conversion parameter storage unit 11 stores image conversion parameters for converting the original image and generating a converted image. The image conversion parameters are as described in Exemplary Embodiment 1. Examples of image conversion methods include color conversion, character conversion, and horizontal inversion. Such conversion methods are used to simulate the differences between new product packaging and existing product packaging. In other words, new product packaging that actually enters the market often looks similar to existing product packaging. Therefore, in order to identify such new products, images generated by applying the above-mentioned conversions to existing product images are useful as training data.

[0032] In this exemplary embodiment, the learning data generation unit 12 selects one of the image conversion parameters stored in the image conversion parameter storage unit 11 and generates a converted image using that image conversion parameter.

[0033] The original image storage unit 15 stores the original image acquired by the acquisition unit 10. The converted image storage unit 16 stores the converted image generated by the training data generation unit 12. The converted image, including the label attached to it, becomes training data for training the product identification model M.

[0034] The information processing device 1A according to this exemplary embodiment is a device for training a product identification model M that identifies whether a product in an image is the same as or different from a product of a class that has already been registered. In retail stores that sell various types of products, a service has been proposed that analyzes the store's shelf layout based on images. In that case, a product identification model is needed to analyze what kinds of products are available from the shelf layout images. Retail stores receive a large number of new products, products with new packaging, etc. The product identification model needs to identify whether the product in the image is an existing product already handled by the store, a new product that has never been handled before, or a product that has been handled before but has new packaging, etc. The information processing device 1A is a device for training such a product identification model M. As an example, the product identification model M is a neural network model using a convolutional neural network or the like.

[0035] Figure 4 shows an example of the process by which the learning data generation unit 12 of the information processing device 1 according to the exemplary embodiment 2 generates a transformed image from an original image. The original image X is an image arbitrarily selected by the learning data generation unit 12 from a plurality of original images stored in the original image storage unit 15. As an example, the learning data generation unit 12 generates transformed images Xa and Xb by performing a color transformation on the original image X. The method of color transformation is arbitrary, but color transformation may be performed by hue rotation using a hue wheel that changes the hue in a ring shape. In the illustrated example, the color transformation is a transformation obtained by hue rotation, and the transformed images Xa and Xb are images with different hue rotation angles.

[0036] Figure 5 shows another example illustrating the process by which the learning data generation unit 12 generates a transformed image from an original image. In the illustrated example, the learning data generation unit 12 generates a transformed image Ya by transforming a string in the original image Y. As an example, the learning data generation unit 12 generates a transformed image Ya by transforming the character "sake" in the original image Y into "ikura". The transformed character may be any string of characters such as hiragana, katakana, kanji, or Roman letters, or it may be a commonly used string of characters that does not contradict the product in the image. Such string data may be stored in any memory unit.

[0037] By horizontally flipping an image of an original class that does not have horizontal reversal (mirror symmetry), such as one containing text, the original class can be transformed into a different pseudo-class. On the other hand, if the original class is an image with horizontal reversal (mirror symmetry), such as the face of a dog or cat, then even after horizontal flipping, the original class cannot be transformed into a different pseudo-class.

[0038] The image conversion parameter storage unit 11 stores image conversion parameters such as parameters indicating color conversion, parameters indicating character conversion, and parameters indicating left-right image inversion. The image conversion parameter storage unit 11 may also store pseudo-classes to be assigned to the converted image converted using the image conversion parameters. The learning data generation unit 12 arbitrarily retrieves image conversion parameters stored in the image conversion parameter storage unit 11 and converts the original image based on them to generate a converted image. The learning data generation unit 12 may also assign pseudo-classes to the generated converted image. The generated converted image and the assigned pseudo-classes are stored by the learning data generation unit 12 in the converted image storage unit 16.

[0039] Figure 6 is a conceptual diagram showing the degree of similarity between the original image and the transformed image, or the difference in the degree of similarity between transformed images themselves. As an example of an indicator of the degree of similarity, the distance in the multidimensional feature vector space output by a convolutional neural network that takes an image as input can be used. In this case, a larger distance in the feature vector space indicates a smaller degree of similarity, and a smaller distance indicates a larger degree of similarity. Conventional methods train a convolutional neural network model according to an objective function intended to ensure that the distances between all classes are equal. However, by adding a correction term, the objective function becomes one that allows for a relatively close distance in the feature space compared to other classes when the appearance of the original class and the pseudo-class are similar.

[0040] In the following, different source images are represented by class numbers. That is, m different source images are represented as Class 1, Class 2, ... Class m. Then, the transformed images obtained by image transformation from the source images are represented by changing the superscript according to the image transformation method. That is, a transformed image obtained by color transformation of a Class 1 source image is represented as Class 1'. Also, a transformed image obtained by horizontally flipping a Class m source image is represented as Class m''. Therefore, the classes of transformed images obtained using n different image transformation methods from a Class 1 source image are represented as Class 1', Class 1'', ... Class 1 (n) This is how it is expressed.

[0041] The image of Class 1' obtained by color conversion of the image of Class 1 has a lower similarity to the image of Class 1'' obtained by horizontal flipping. Expressing this relationship in terms of distance, as shown in Figure 6, the image of Class 1' is farther from the image of Class 1 than the image of Class 1''. The same relationship is observed between the image of Class 2 and the images of Class 2' and 2'' obtained by transforming it. In conventional techniques, such similarity relationships were not considered during the learning stage, but in this exemplary embodiment, the product identification model M is trained while considering such distance relationships. This learning method that focuses on distance in the feature vector space is called distance-metric learning. In other words, the product identification model M is a model trained by distance-metric learning.

[0042] In this exemplary embodiment, the degree of similarity, i.e., the degree of distance, is set by the strength of the margin added to the likelihood of each class for the input image. An example of likelihood is the dot product value, which will be described later. Hereafter, the strength of the margin added to the likelihood of each class, more specifically, the strength of the margin that adjusts the likelihood distribution of the estimated class of the image output by the image recognition model, will also be simply referred to as the margin strength. The margin strength is an example of a correction term. The objective function used for learning is a loss function in which the features of the image input to the convolutional neural network model are given a margin between the distances of the input class, the pseudo-class generated from the input class, and the representative features of other classes. In other words, the learning loss determination unit 13 sets the strength of the margin according to the similarity between each class and generates a loss function as the objective function that reflects this as a correction term. This makes it possible to train a product recognition model using a neural network model with a loss function that considers the margin strength as a correction term.

[0043] Figure 7 is a table showing an example of setting margin strengths for training images. Figure 7 shows the margin strengths applied to pseudo-classes generated from class 1, and to original classes other than class 1 and pseudo-classes generated from classes other than class 1, for input of a class 1 original image. In the illustrated example, image transformation parameter 1 (hereinafter referred to as parameter 1) is a parameter that rotates the hue. Image transformation parameter 2 (hereinafter referred to as parameter 2) is a parameter that combines hue rotation and horizontal flipping. Image transformation parameter n (hereinafter referred to as parameter n) is a parameter that flips the image horizontally. Furthermore, the visual similarity between any original class C and other classes is such that the similarity between class C and pseudo-class C'' < similarity between class C and original classes other than class C and similarity between class C and pseudo-classes generated from classes other than class C ≤ similarity between class C and pseudo-class C' < similarity between class C and pseudo-class C (n) It is assumed that the relationship is something like similarity to [another entity].

[0044] In this case, the margin strength to be applied to the pseudo-class of class 1 is set as parameter 2 > parameter 1 > parameter n. In other words, the size of the margin strength is adjusted to correspond to the visual similarity. Specifically, pseudo-class 1 is generated from class 1 of the original image using the image transformation parameter n. (n) Since the pseudo-class has a high visual similarity to the original class, the margin strength is reduced. This reduces the degree to which the model tries to increase the distance during training. On the other hand, class 1'', which is generated from class 1 of the original image using image transformation parameter 2, has a low visual similarity to the original class, so the margin strength is increased. In this way, the margin strength plays a role in adjusting the feature distance between the original class and the pseudo-class so that it corresponds to the visual relationship during training. Note that the similarity between the original class of class 1 and the original classes other than class 1 and their pseudo-classes is the same, so the margin strength given is the same for all of them.

[0045] Figure 8 shows the margin strengths applied to the pseudo-classes generated from Class 2, and to the original classes other than Class 2 and the pseudo-classes generated from those classes, for a given input of a Class 2 original image. As shown in Figure 7, the margin strength applied to the Class 2 pseudo-class is adjusted to correspond to the visual similarity. Furthermore, since the similarity between the Class 2 original class, the original classes other than Class 2, and their pseudo-classes is the same, the applied margin strengths are the same.

[0046] The examples in Figures 7 and 8 show the margin strengths from the perspective of the original classes 1 and 2, respectively. The margin strengths when training with transformed images generated by image transformation parameters can be determined in a similar manner. For example, from the perspective of a pseudo-class generated by parameter n, the original class is horizontally flipped, parameter 2 is hue-rotated, and parameter 1 is hue-rotated and horizontally flipped. In this case, the margin strengths follow the relationship parameter 1 > parameter 2 > original class, and the correction term when training with transformed images generated from transformation parameter n can also be determined.

[0047] In the examples shown in Figures 7 and 8, the learning loss determination unit 13 determines a common correction term for each image transformation parameter used to transform the original image for multiple classes of original images. In other words, during learning on the original image, a common correction term is determined for each image transformation parameter for the transformed image of the original image, regardless of the class. However, for classes other than the original image class and their pseudo-classes, a common correction term is determined regardless of the image transformation parameters. The margin strength may be determined by the user, or it may be determined mechanically by the information processing device 1. The information processing device 1 may search for and set an effective margin strength using a grid search or the like. Alternatively, an effective margin strength may be set using reinforcement learning.

[0048] Figure 9 is a conceptual diagram illustrating how the learning loss determination unit 13 determines the correction term while considering the margin strength. The vertical axis of the graph in Figure 9 represents the dot product of the feature vector output by the model when the original class is input, and the representative feature vector of each class, on an n-dimensional hypersphere of radius 1. Each dot product represents the likelihood that the original image belongs to the corresponding class. The closer the dot product value is to 1, the greater the likelihood that the input image is that label, and the closer the value is to -1, the smaller the similarity to the original image. The horizontal axis of the graph represents the labels 0 to 9 of the original image and the labels 0' to 9' of the transformed image obtained by horizontally flipping labels 0 to 9. Figure 9 shows an example of adjusting the product recognition model M to allow a small distance from label 3' when the product with label 3 is input. Specifically, as shown in Figure 9, the margin of label 3' is weakened. Weakening the margin means setting a small margin strength for label 3'. This weakens the learning of the product recognition model M for the image with label 3'.

[0049] Alternatively, the margin strength may be set as shown in Figure 10. Figure 10 shows an example where the margin of label 3' is increased, i.e., the margin strength of label 3' is set high, in order to enable discriminatory learning for the product labeled 3. With such a setting, learning for the image labeled 3' can be strengthened. This can improve the discriminatory ability of the product identification model M to label 3'.

[0050] The learning unit 14 trains the product recognition model M to reduce the loss function, which takes margin strength as a correction term. Specifically, the learning loss determination unit 13 determines the margin strength to be added to the loss function. The learning unit 14 has the product recognition model M recognize the transformed image and modifies the model's weights and biases so that the correction value reflecting the margin strength in the derived loss function becomes smaller. The learning unit 14 repeats this process and terminates learning when the value of the loss function reaches a predetermined range.

[0051] As explained above, the reason for considering margin strength in the distance as a loss function is as follows. As shown in Figure 14, in conventional techniques that assign a pseudo-class different from the original class to a transformed image of the original image, the likelihood of the image with label 3 is trained to approach 1, while the likelihood of all images with labels other than label 3 is trained to approach -1. The margin that lowers the likelihood for the input of label 3 has an effect on all other labels. In other words, by lowering the likelihood of the correct answer with the margin, it is necessary to raise the likelihood during training. To raise the likelihood, the distance between feature vectors of inputs of the same class is reduced, and the distance between feature vectors of labels other than itself (label 3) is increased. On the other hand, considering that label 3' looks similar, the likelihood can be made smaller than the actual estimated value, so that the distance between label 3' and its feature vector is not artificially increased during training. In particular, by weakening the margin strength, it is possible to prevent training that artificially increases the distance when there are classes that look similar between the original class and the pseudo-class. This makes it possible to avoid overfitting, which causes a decrease in accuracy during testing.

[0052] The learning unit 14 may train the product recognition model M by changing the frequency with which it uses converted images generated using predetermined image conversion parameters. For example, during the training of the product recognition model M, the learning unit 14 may adjust the number of converted images with specific labels that are considered to have a high learning effect, or decrease the number of converted images with specific labels that are considered to have a low learning effect. This allows the model to be trained on a larger number of products in a specific class or a class that has a high learning effect.

[0053] During the learning phase, for example, the learning data may be divided into learning data, validation data, and evaluation data. The learning unit 14 may then train the product recognition model M using the learning data, the user may validate it using the validation data, and the user may evaluate it using the evaluation data. This is a split-fold cross-validation method. The criteria for setting the margin strength as described above can be based on the user's judgment of the degree of similarity. However, the relative magnitudes of the margin strengths may be determined by other methods.

[0054] Furthermore, the acquisition unit 10, image conversion parameter storage unit 11, learning data generation unit 12, learning loss determination unit 13, learning unit 14, original image storage unit 15, and converted image storage unit 16 that constitute the information processing device 1A, and the product identification model M, may be distributed and arranged so that at least a portion of them can communicate with each other. In addition, at least a portion of each of the above-mentioned units may be located on the cloud so that they can communicate with each other.

[0055] In the exemplary embodiment described above, as shown in Figures 7 and 8, the learning loss determination unit 13 determined a common margin intensity for each image transformation parameter used to transform multiple original images, which are classes 1, 2, ..., m. However, the learning loss determination unit 13 may also determine correction terms that depend on the image transformation parameters and classes used to transform the original images for multiple classes of original images. Figure 11 is a table showing an example in which the learning loss determination unit 13 determines different patterns of margin intensity for transformed images using each parameter. The distribution of margin intensity of the pseudo-class of original class 1 for original class 1 shown in Figure 11(a) is different from the distribution of margin intensity of the pseudo-class of original class 2 for original class 2 shown in Figure 11(b). Parameters 1, 2, ..., n are as described in Figure 7. As shown in Figure 11, the margin intensity (correction term) for images generated with parameters 1, 2, ..., n depends on the image transformation parameters and classes. In this way, the product identification model M may be trained by changing the distribution pattern of margin intensity. This is expected to prevent overfitting of the product recognition model M, allowing for the generation and use of more effective training images from limited training data. In other words, the product recognition model M can be trained to acquire many feature representations that enable it to identify new products.

[0056] The method for setting the margin strength (correction term) for image transformation parameters may be set by the user, or the learning loss determination unit 13 may set it using a correction term generation model. The correction term generation model may perform transformations for new labels, such as randomly swapping margin strengths or multiplying existing margin strengths by a coefficient. The correction term generation model may be trained to generate correction terms suitable for learning by referring to the learning results performed considering the generated correction terms. This allows for a wide range of correction terms to be set and the product recognition model to be trained.

[0057] (Effects of Information Processing Device 1A) In the information processing device 1A according to this exemplary embodiment, in addition to the configuration of the information processing device 1 according to exemplary embodiment 1, an acquisition unit 10, an image conversion parameter storage unit 11, an original image storage unit 15, and a converted image storage unit 16 are included, and the information processing device 1A is configured to train a product identification model M. Therefore, according to the information processing device 1A according to this exemplary embodiment, in addition to the effects achieved by the information processing device 1 according to exemplary embodiment 2, the effect of being able to train a product identification model M by considering the similarity between the original class and the pseudo-class of the product image is obtained.

[0058] [Exemplary Embodiment 3] A third exemplary embodiment of the present invention will be described in detail with reference to the drawings. Components having the same function as those described in exemplary embodiments 1 and 2 will be denoted by the same reference numerals, and their descriptions will not be repeated.

[0059] Figure 12 is a block diagram showing the configuration of an information processing device 1B according to exemplary embodiment 3 of the present invention. The information processing device 1B comprises an acquisition unit 10, an image conversion parameter storage unit 11, a learning data generation unit 12, a learning loss determination unit 13, a learning unit 14, an original image storage unit 15, a converted image storage unit 16, and a learning parameter storage unit 17. The information processing device 1B is connected to a product identification model M in a manner that enables information communication.

[0060] The functions of the acquisition unit 10, image conversion parameter storage unit 11, learning data generation unit 12, learning loss determination unit 13, learning unit 14, original image storage unit 15, and converted image storage unit 16 are the same as those of each unit described in Exemplary Embodiment 2.

[0061] The learning parameter storage unit 17 stores correction terms determined for each image transformation parameter. These correction terms determined for each image transformation parameter are also called learning parameters. For example, the learning parameters may be a set of margin intensity patterns as shown in Figures 7, 8, or 11, or a set of dot product values ​​considering margin intensity as shown in Figure 9 or 10. Learning parameters that have been confirmed to be effective for training a certain product recognition model are thought to be effective for other product recognition models as well. They can also be used as templates for generating different learning parameters. By saving learning parameters of different patterns, the learning loss determination unit 13 can appropriately select the saved learning parameters and use them to train a new product recognition model, thereby efficiently training the product recognition model.

[0062] (Effects of Information Processing Device 1B) As described above, the information processing device 1B according to this exemplary embodiment is configured to include a learning parameter storage unit 17 in addition to the configuration of the information processing device 1A according to exemplary embodiment 2. Therefore, the information processing device 1B according to this exemplary embodiment provides the effect of being able to train the product identification model more efficiently, in addition to the effects of the information processing device 1A according to exemplary embodiment 2.

[0063] [Examples of implementation using software] Some or all of the functions of the information processing devices 1, 1A, and 1B may be implemented by hardware such as integrated circuits (IC chips) or by software.

[0064] In the latter case, the information processing units 1, 1A, and 1B are implemented by a computer that executes instructions for a program, which is software that implements each of their functions. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 12. Computer C comprises at least one processor C1 and at least one memory C2. The memory C2 stores a program P that causes computer C to operate as information processing units 1, 1A, and 1B. In computer C, the processor C1 reads the program P from the memory C2 and executes it, thereby implementing each of the functions of the information processing units 1, 1A, and 1B.

[0065] Processor C1 can include, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), microcontroller, or a combination thereof. Memory C2 can include, for example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof.

[0066] Computer C may also be equipped with RAM (Random Access Memory) for loading program P at runtime and for temporarily storing various data. Furthermore, computer C may be equipped with communication interfaces for sending and receiving data with other devices. Additionally, computer C may be equipped with input / output interfaces for connecting input / output devices such as keyboards, mice, displays, and printers.

[0067] Furthermore, program P can be recorded on a non-temporary, tangible recording medium M that is readable by computer C. Such a recording medium M could be, for example, tape, disk, card, semiconductor memory, or programmable logic circuitry. Computer C can acquire program P via such a recording medium M. Program P can also be transmitted via a transmission medium. Such a transmission medium could be, for example, a communication network or broadcast waves. Computer C can also acquire program P via such a transmission medium.

[0068] [Additional Note 1] The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the embodiments described above are also included in the technical scope of the present invention.

[0069] [Additional Note 2] Some or all of the embodiments described above may also be described as follows. However, the present invention is not limited to the embodiments described below.

[0070] (Note 1) An information processing device comprising: a learning data generation means for generating a transformed image from an original image using image transformation parameters, and assigning a pseudo-class to the generated transformed image according to the class of the original image and the image transformation parameters; a learning loss determination means for generating an objective function that reflects a correction term that takes into account the degree of similarity between the class and the pseudo-class determined for each image transformation parameter; and a learning means for training an image recognition model using the transformed image and the objective function.

[0071] (Note 2) The image recognition model is a model learned by distance metric learning, and the objective function is a loss function in which the strength of the margin that adjusts the likelihood distribution of the estimated class of the image output by the image recognition model is reflected as the correction term, as described in Appendix 1.

[0072] (Note 3) The information processing apparatus according to Appendix 1 or 2, wherein the learning loss determination means determines a common correction term for each of the image conversion parameters used to convert the original image for a plurality of classes of converted images of the original image.

[0073] (Note 4) The information processing apparatus according to Appendix 1 or 2, wherein the learning loss determination means determines the image transformation parameters for transforming the original image and the class-dependent correction term for the transformed images of the original image of a plurality of classes.

[0074] (Note 5) The learning loss determination means determines the correction term for the image transformation parameters using a correction term generation model, as described in any one of the appendices 1 to 4.

[0075] (Note 6) The information processing apparatus according to any one of the appendices 1 to 5, further comprising a learning parameter storage means for storing the correction term determined for each of the image conversion parameters.

[0076] (Note 7) The information processing apparatus according to any one of the appendices 1 to 6, wherein the learning means trains the image recognition model by changing the frequency at which the converted image generated using predetermined image conversion parameters is used.

[0077] (Note 8) An information processing method comprising: at least one processor generating a transformed image from an original image using image transformation parameters; assigning a pseudo-class to the generated transformed image according to the class of the original image and the image transformation parameters; generating an objective function that reflects a correction term that takes into account the degree of similarity between the class and the pseudo-class determined for each image transformation parameter; and training an image recognition model using the transformed image and the objective function.

[0078] (Note 9) An information processing program that causes a computer to perform the following processes: generate a transformed image from an original image using image transformation parameters; assign a pseudo-class to the generated transformed image according to the class of the original image and the image transformation parameters; generate an objective function that reflects a correction term that takes into account the degree of similarity between the class and the pseudo-class determined for each image transformation parameter; and train an image recognition model using the transformed image and the objective function.

[0079] (Note 10) A non-temporary recording medium on which the information processing program described in Appendix 9 is recorded.

[0080] (Note 11) An information processing device comprising at least one processor, the processor performing: a first generation process which generates a transformed image from an original image using image transformation parameters and assigns a pseudo-class to the generated transformed image according to the class of the original image and the image transformation parameters; a second generation process which generates an objective function which reflects a correction term that takes into account the degree of similarity between the class and the pseudo-class determined for each image transformation parameter; and a learning process which trains an image recognition model using the transformed image and the objective function. Furthermore, this information processing device may also be equipped with memory, and this memory may store a program that causes the processor to execute the first generation process, the second generation process, and the learning process. This program may also be recorded on a computer-readable, non-temporary, tangible recording medium. [Explanation of Symbols]

[0081] 1, 1A, 1B... Information Processing Devices 10…Acquisition part 11…Image conversion parameter storage unit 12...Training data generation unit 13…Learning Loss Determination Unit 14…Learning Department 15... Original image storage unit 16...Conversion Image Storage Unit 17... Learning parameter storage unit

Claims

1. A learning data generation means that generates a transformed image from an original image using image transformation parameters, and assigns a pseudo-class to the generated transformed image according to the class of the original image and the image transformation parameters, A learning loss determination means that generates an objective function that reflects a correction term that takes into account the degree of similarity between the class and the pseudo-class determined for each of the image transformation parameters, A learning means for training an image recognition model using the transformed image and the objective function. An information processing device equipped with the following features.

2. The image recognition model is a model trained by distance metric learning, and the objective function is a loss function that reflects the strength of the margin that adjusts the likelihood distribution of the estimated class of the image output by the image recognition model as the correction term. The information processing apparatus according to claim 1.

3. The learning loss determination means determines a common correction term for each of the image transformation parameters used to transform the original image for multiple classes of transformed original images. The information processing apparatus according to claim 1.

4. The learning loss determination means determines, for the transformed images of the original images of multiple classes, the image transformation parameters used to transform the original images and the correction terms that depend on the class. The information processing apparatus according to claim 1.

5. The learning loss determination means determines the correction term for the image transformation parameters using a correction term generation model. The information processing apparatus according to any one of claims 1 to 4.

6. The system further comprises a learning parameter storage means for storing the correction terms determined for each of the image conversion parameters. The information processing apparatus according to any one of claims 1 to 4.

7. The learning means trains the image recognition model by changing the frequency with which it uses the converted image generated using predetermined image conversion parameters. The information processing apparatus according to any one of claims 1 to 4.

8. At least one processor, A transformed image is generated from the original image using image transformation parameters, and a pseudo-class corresponding to the class of the original image and the image transformation parameters is assigned to the generated transformed image. To generate an objective function that reflects a correction term that takes into account the degree of similarity between the class and the pseudo-class determined for each of the image transformation parameters, The image recognition model is trained using the transformed image and the objective function. Information processing methods including

9. On the computer, A process that generates a converted image from the original image using image conversion parameters, and assigns a pseudo-class to the generated converted image according to the class of the original image and the image conversion parameters, A process to generate an objective function that reflects a correction term that takes into account the degree of similarity between the class and the pseudo-class determined for each of the image transformation parameters, A process for training an image recognition model using the transformed image and the objective function, An information processing program that executes [something].

10. A non-temporary recording medium that stores the information processing program described in claim 9.

Images