Information processing device, information processing method, and information processing program

Abstract

This information processing device includes a training unit that trains an attribute prediction model for predicting N attributes (N is an integer of 2 or more) of a product from an image including the product. The training unit includes: a first training unit that trains an i-th training model for predicting an i-th attribute (i is an integer of 1 to N-1) of the product by using i-th training data for the i-th attribute; and a second training unit that trains the attribute prediction model by using the i-th training model trained by the first training unit, a part of the i-th training data, and N-th training data for an N-th attribute of the product.

Description

Information processing device, information processing method, and information processing program

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program, and more particularly to a technique for predicting attributes of a product from an image including the product.

[0002] In recent years, electronic commerce (EC), which involves selling products over the Internet, has become increasingly common, and many EC sites have been established on the Web for such EC transactions. EC sites are often established using the languages ​​of countries around the world, allowing users (consumers) living in many countries to purchase products. By accessing EC sites from mobile devices such as personal computers (PCs) and smartphones, users can select and purchase desired products without visiting a physical store and regardless of the time.

[0003] In order to encourage users to make purchases, e-commerce sites may display recommended products that have similar attributes (information specific to products) to products previously purchased by the user on the screen the user is browsing. Furthermore, when users purchase a desired product, they may search for the product based on its attributes. For these reasons, identifying product attributes is a common challenge for site operators and product providers in e-commerce.

[0004] In recent years, a technology has been developed that uses a learning model for machine learning to predict attributes of a product from an image containing the product. For example, Patent Literature 1 discloses a technology that predicts multiple attributes of a product by applying an image containing the product to a learning model configured using a neural network.

[0005] Japanese Patent Application Laid-Open No. 2020-71859

[0006] According to the technology disclosed in Patent Literature 1, by applying an image containing a product to a learning model, it is possible to automatically predict multiple attributes of the product. Meanwhile, the types and number of products handled in e-commerce are increasing, and the number of attributes is also increasing accordingly. Therefore, it is desirable to build a learning model that can predict not only existing attributes of a product but also new attributes from an image containing the product.

[0007] Incremental learning is a well-known method for training a learning model to learn a new task by continuously using additional training data. Incremental learning allows the model to be trained using additional training data for new attributes, thereby building a learning model that can predict not only previously learned attributes but also newly learned attributes.

[0008] However, when training a learning model using successively additional training data, a problem called catastrophic forgetting occurs, in which previously trained data is "forgotten." For example, the previously trained data may have a large error from the correct answer data, or the error may become large, or the data may be completely forgotten.

[0009] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a technology for learning new attributes of a product from an image containing the product while suppressing catastrophic forgetting.

[0010] An information processing device according to one aspect of the present invention includes a learning unit that trains an attribute prediction model for predicting N attributes (N is an integer equal to or greater than 2) of a product from an image including the product, and the learning unit includes a first learning unit that trains an i-th learning model for predicting an i-th attribute (i is an integer from 1 to N-1) of the product using i-th learning data for the i-th attribute, and a second learning unit that trains the attribute prediction model using the i-th learning model trained by the first learning unit, a portion of the i-th learning data, and N-th learning data for the N-th attribute of the product.

[0011] An information processing method according to one aspect of the present invention is an information processing method for training an attribute prediction model for predicting N attributes (N is an integer equal to or greater than 2) of a product from an image including the product, the information processing method including: training an i-th learning model for predicting an i-th attribute (i is an integer from 1 to N-1) of the product using i-th learning data for the i-th attribute; and training the attribute prediction model using the i-th learning model trained by the first learning unit, a portion of the i-th learning data, and N-th learning data for the N-th attribute of the product.

[0012] An information processing program according to one aspect of the present invention is an information processing program that causes a computer to train an attribute prediction model for predicting N attributes (N is an integer equal to or greater than 2) of a product from an image including the product, and causes a computer to train an i-th learning model for predicting an i-th attribute (i is an integer from 1 to N-1) of the product using i-th learning data for the i-th attribute, and train the attribute prediction model using the i-th learning model trained by the first learning unit, a portion of the i-th learning data, and N-th learning data for the N-th attribute of the product.

[0013] According to the present invention, it is possible to learn new attributes of a product from an image containing the product while suppressing catastrophic forgetting. The above-mentioned objects, aspects, and advantages of the present invention, as well as other objects, aspects, and advantages of the present invention, will be understood by those skilled in the art from the following detailed description of the invention, with reference to the accompanying drawings and the claims.

[0014] FIG. 1 shows an example configuration of a natural language processing system according to an embodiment. FIG. 2 shows an example hardware configuration of an information processing device according to an embodiment. FIG. 3A is a diagram for explaining product attributes. FIG. 3B shows the relationship between product images and attributes. FIG. 4 shows a conceptual diagram of attribute learning data. FIG. 5A shows a learning procedure for a learning model for a first attribute in a first time step. FIG. 5B shows a learning procedure for a learning model for a first attribute to a second attribute in a second time step. FIG. 5C shows a learning procedure for a learning model for a first attribute to a third attribute in a third time step. FIG. 5D is a diagram for explaining a learning procedure for a learning model for a first attribute to an Nth attribute in an Nth time step. FIG. 6A shows a flowchart of a learning process for an attribute prediction model executed by an information processing device according to an embodiment. FIG. 6B shows a flowchart of a modified example of the learning process for an attribute prediction model executed by an information processing device according to an embodiment. FIG. 7 shows a flowchart of an attribute prediction process executed by an information processing device according to an embodiment.

[0015] Hereinafter, with reference to the accompanying drawings, an embodiment for carrying out the present invention will be described in detail. Among the components disclosed below, those having the same function will be given the same reference numerals, and their description will be omitted. Note that the embodiment disclosed below is an example of a means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions. The present invention is not limited to the following embodiment. Furthermore, not all of the combinations of features described in the present embodiment are necessarily essential to the solution of the present invention.

[0016] [Functional Configuration of Information Processing Device] The information processing device 100 according to this embodiment acquires an image (hereinafter also referred to as a product image) that includes a product (i.e., displays a product) and predicts multiple attributes of the product. Product attributes are information specific to the product, such as the visual characteristics of the product. Product attributes can be indicators for when a user purchases the product. Note that, although this embodiment describes an example of predicting product attributes, this embodiment can also be applied to the case where attributes of any item, including a product, are predicted from an image that includes the item (i.e., displays the item).

[0017] 1 shows an example of the functional configuration of an information processing device 100 according to this embodiment. The information processing device 100 includes a learning data acquisition unit 101, a learning data management unit 102, a learning unit 103, an image acquisition unit 104, an attribute prediction unit 105, an output unit 106, a learning model storage unit 110, and an attribute data storage unit 120. The learning unit 103 includes a first learning unit 1031 and a second learning unit 1032. The learning model storage unit 110 is configured to store a first attribute specialized model 111-1 to an N-th attribute specialized model 111-N and an attribute prediction model 112. The attribute data storage unit 120 is configured to store first attribute learning data 121-1 to N-th attribute learning data 121-N. In the present disclosure, "N" is an integer equal to or greater than 2.

[0018] The learning data acquisition unit 101 acquires learning data (teacher data) for training a learning model. In this embodiment, the learning data includes product images and correct answer data for product attributes. Here, an example of product attributes in this embodiment will be described with reference to FIG. 3A. FIG. 3A is a diagram for explaining product attributes.

[0019] In this embodiment, the products are assumed to be products that can be handled on an EC (Electronic Commerce) site. Since the types and number of products that can be handled on an EC site are enormous, product attributes are set for classified products (product groups). Products can be classified hierarchically. In this embodiment, attributes are assumed for a case where the product category that represents the highest classification among the product classifications is "clothing."

[0020] In FIG. 3A , products in product category 30 "clothing" are classified into subcategories 31, which indicate subcategories of category 30, and sub-subcategories 32, which indicate further subcategories of subcategories 31. Subcategory 31 indicates the objects on which the "clothing" is worn, and in the example of FIG. 3A , includes "men" and "women." Subcategory 31 may also include "children," "seniors," and "unisex," which does not consider gender. Sub-subcategory 32 indicates the type or shape of the "clothing." If subcategory 31 is "men," sub-subcategory 32 includes "T-shirts" and "jeans." If subcategory 31 is "men," sub-subcategory 32 may also include "jackets," "coats," etc.

[0021] In the example of FIG. 3A , attributes are set for products in sub-subcategory 32. If sub-subcategory 32 is "T-shirts," the attributes include "pattern," "sleeve," "neck," and "color." If sub-subcategory 32 is "jeans," the attributes include "pattern," "fit," and "length." Note that the types of attributes for each product shown in FIG. 3A are merely examples and are not limited to those shown. Furthermore, the number of types of attributes may increase in the future. Furthermore, although attributes are set for products in sub-subcategory 32 in FIG. 3A , attributes may also be set for subcategories 31 and category 30.

[0022] FIG. 3B shows the relationship between product images and attributes. FIG. 3B shows data 35 including attributes 37 of product 38 included in product image 36. Product 38 is a product classified in FIG. 3A under category 30 = "clothing," subcategory 31 = "men's," and sub-subcategory 32 = "T-shirts." Referring to FIG. 3A, attributes 37 include "pattern," "sleeves," "neck," and "color." Furthermore, each attribute in attributes 37 has a correct feature (feature value) (hereinafter also referred to as a correct feature). The correct features for each attribute of product 38 in product image 36 are as shown in FIG. 3B, where "pattern" is "border," "sleeves" is "three-quarter sleeves," "neck" is "round," and "color" is "white and black."

[0023] As described above, the training data includes product images and correct data for product attributes. The training data acquisition unit 101 generates and acquires training data from, for example, data 35 shown in FIG. 3B . If the attribute of product 38 shown in FIG. 3B is "pattern," possible features of the "pattern" are, for example, "border," "solid," "check," "dot," and "print." In the case of a classification problem, correct data for training is all 1 or 0. In other words, there are two choices: 100% (correct feature) or 0% (incorrect feature). Since the "pattern" of product 38 included in product image 36 is "border," the first feature in the following order is the correct feature: "border," "solid," "check," "dot," and "print." Therefore, the correct data is given in the form of a probability distribution, for example, {1, 0, 0, 0, 0}.

[0024] The learning data management unit 102 stores the learning data acquired by the learning data acquisition unit 101 as attribute learning data for each attribute in the attribute data storage unit 120. Referring to Fig. 3B , the learning data management unit 102 stores combinations of product images 36 and correct answer data for attributes 37 (i.e., "pattern," "sleeve," "neck," and "color") as attribute learning data for each attribute in the attribute data storage unit 120. Fig. 4 shows a conceptual diagram of the attribute learning data stored in the attribute data storage unit 120.

[0025] As shown in FIG. 1 , the attribute data storage unit 120 is configured to store first attribute learning data 121-1 to N-th attribute learning data 121-N. For the sake of explanation, it is assumed that the first attribute is "pattern" and the second attribute is "sleeve." For example, based on the data 35 shown in FIG. 3B , the learning data management unit 102 classifies data combining a product image 36 with correct answer data in which "border" 41 is "1" into first attribute learning data 121-1. Furthermore, the learning data management unit 102 classifies a combination of the product image 36 with correct answer data in which "three-quarter sleeve" 42 is "1" into second attribute learning data 121-1. In this way, the learning data management unit 102 extracts product images and one or more attributes from the learning data acquired by the learning data acquisition unit 101, and classifies the combinations of the product images and correct answer data into one of the first attribute learning data 121-1 to N-th attribute learning data 121-N for each attribute. As a result, each of the first attribute learning data 121-1 to the Nth attribute learning data 121-N stores a set of combination data of a product image and correct answer data for the first attribute to the Nth attribute of the product included in the image.

[0026] In this embodiment, the learning data management unit 102 is configured to store the learning data acquired by the learning data acquisition unit 101 in the attribute data storage unit 120 for each attribute, but the procedure for storing data in the attribute data storage unit 120 is not limited to this. For example, when the learning data acquisition unit 101 acquires a product image and a correct feature for one attribute, the learning data management unit 102 may store, in accordance with the attribute, combination data of the product image and correct data corresponding to the correct feature in any of the first attribute learning data 121-1 to the Nth attribute learning data 121-N. Alternatively, in this case, the learning data acquisition unit 101 may directly store the combination data in any of the first attribute learning data 121-1 to the Nth attribute learning data 121-N.

[0027] Furthermore, the learning data management unit 102 manages data to be supplied to the learning unit 103 from the first attribute learning data 121-1 to the Nth attribute learning data 121-N stored in the attribute data storage unit 120. Taking the first attribute learning data 121-1 as an example, the learning data management unit 102 may supply the full set of first attribute learning data 121-1 to the learning unit 103. Alternatively, the learning data management unit 102 may select (sample) a portion of the first attribute learning data 121-1 and supply it to the learning unit 103. The portion of data may be selected randomly or may be selected according to a predetermined rule. In the present disclosure, the term "full set of data" means a larger amount of data (e.g., including a larger set of learning data) than "a portion of data." The learning data management unit 102 may supply the attribute learning data to the learning unit 103 in response to a request from the learning unit 103.

[0028] Returning to the description of FIG. 1 , the learning unit 103 trains the first attribute learning data 121-1 to the Nth attribute learning data 121-N and the attribute prediction model 112. These learning models may be configured using a convolutional neural network (CNN). The learning unit 103 includes a first learning unit 1301 and a second learning unit 1302, and is responsible for overall control of the learning process, including control of the first learning unit 1301 and the second learning unit 1302. The first learning unit 1301 is configured to train the first attribute-dedicated model 111-1 to the Nth attribute-dedicated model 111-N. The second learning unit 1302 is configured to train the attribute prediction model 112 using at least one of the trained first attribute-dedicated model 111-1 to the Nth attribute-dedicated model 111-N. The learning procedure of the learning unit 103 for the first attribute-specific model 111-1 to the N-th attribute-specific model 111-N and the attribute prediction model 112 will be described later.

[0029] The image acquisition unit 104 acquires a product image including a target product (i.e., a product whose attributes are to be predicted). The image acquisition unit 104 may acquire the product image by an input operation by a user (operator) via the input unit 25 (FIG. 2), or may acquire the product image from a storage unit (ROM 22 or RAM 23 in FIG. 2) by a user operation. The image acquisition unit 104 may also acquire a product image received from an external device via the communication I / F 27 (FIG. 2).

[0030] The attribute prediction unit 105 applies the product image acquired by the image acquisition unit 104 to the trained attribute prediction model 112, and predicts multiple attributes of the target product included in the product image.

[0031] The output unit 106 outputs information on the attributes predicted by the attribute prediction unit 105 (attribute prediction result). The output unit 106 may output the attribute prediction result, for example, in association with the product image acquired by the image acquisition unit 104. The output unit 106 may display the attribute prediction result on the display unit 26 (FIG. 2). Furthermore, when the product image is acquired from an external device such as a user device, the output unit 106 may transmit the attribute prediction result to the external device via the communication I / F 27 (FIG. 2) so that it is displayed on the display unit of the external device.

[0032] [Hardware Configuration of Information Processing Device] Fig. 2 is a block diagram showing an example of the hardware configuration of the information processing device 100 according to this embodiment. The information processing device 100 can be implemented on a single or multiple computers, mobile devices, or any other processing platform. While Fig. 2 shows an example in which the information processing device 100 is implemented on a single computer, the information processing device 100 according to this embodiment may be implemented in a computer system including multiple computers. The multiple computers may be connected to each other via a wired or wireless network so that they can communicate with each other.

[0033] As shown in FIG. 2, the information processing device 100 may include a CPU (Central Processing Unit) 21, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, a HDD (Hard Disk Drive) 24, an input unit 25, a display unit 26, a communication I / F 27, and a system bus 28. The information processing device 100 may also include an external memory. The CPU 21 comprehensively controls the operations of the information processing device 100, and controls each of the components (22 to 27) via the system bus 28, which is a data transmission path. The CPU 21 is composed of one or more processors. At least one of the one or more processors may be replaced by one or more processors such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a graphics processing unit (GPU), or the like.

[0034] The ROM 22 is a non-volatile memory that stores control programs and the like necessary for the CPU 21 to execute processing. The programs may be stored in a non-volatile memory such as the HDD 24 or an SSD (Solid State Drive) or in an external memory such as a removable storage medium (not shown). The RAM 23 is a volatile memory that functions as the main memory, work area, and the like of the CPU 21. That is, when executing processing, the CPU 21 loads the necessary programs and the like from the ROM 22 into the RAM 23 and executes the programs and the like to realize various functional operations. The learning model storage unit 110 and the attribute data storage unit 120 shown in FIG. 1 may be configured using the RAM 23.

[0035] The HDD 24 stores, for example, various data and information required when the CPU 21 performs processing using a program. The HDD 24 also stores, for example, various data and information obtained when the CPU 21 performs processing using a program. The input unit 25 is composed of a pointing device such as a keyboard and a mouse. The display unit 26 is composed of a monitor such as a liquid crystal display (LCD). The display unit 26 may function as a GUI (Graphical User Interface) when configured in combination with the input unit 25.

[0036] The communication I / F 27 is an interface that controls communication between the information processing device 100 and an external device. The communication I / F 27 provides an interface with a network and executes communication with the external device via the network. Various data, parameters, and the like are transmitted and received between the information processing device 100 and the external device via the communication I / F 27. In this embodiment, the communication I / F 27 may execute communication via a wired LAN (Local Area Network) or a dedicated line that conforms to a communication standard such as Ethernet (registered trademark). However, the network that can be used in this embodiment is not limited to this, and may be configured as a wireless network. This wireless network includes a wireless PAN (Personal Area Network) such as Bluetooth (registered trademark), ZigBee (registered trademark), or UWB (Ultra Wide Band). It also includes wireless LANs (Local Area Networks) such as Wi-Fi (Wireless Fidelity) (registered trademark) and wireless MANs (Metropolitan Area Networks) such as WiMAX (registered trademark). It also includes wireless WANs (Wide Area Networks) such as 4G and 5G defined by 3GPP (Third Generation Partnership Project) (registered trademark). Note that the network only needs to connect devices to each other so that communication is possible, and the communication standard, scale, and configuration are not limited to those described above.

[0037] At least some of the functions of the information processing device 100 shown in Fig. 1 can be realized by the CPU 21 executing a program. However, at least some of the functions of the information processing device 100 shown in Fig. 1 may be operated as dedicated hardware. In this case, the dedicated hardware may operate under the control of the CPU 21.

[0038] [Learning Procedure of Learning Model] Next, the learning procedure of the learning model by the learning unit 103 will be described with reference to Figs. 5A to 5D. In this embodiment, the learning unit 103 performs incremental learning, in which the learning model is trained by continuously using additional training data. Fig. 5A shows the learning procedure of the learning model for the first attribute in the first time step. Fig. 5B shows the learning procedure of the learning model for the first attribute to the second attribute in the second time step. Fig. 5C shows the learning procedure of the learning model for the first attribute to the third attribute in the third time step. Fig. 5D is a diagram for explaining the learning procedure of the learning model for the first attribute to the Nth attribute in the Nth time step. Here, the term "time step" is understood to be a term indicating a relative time, and not a term indicating a specific time (or time width).

[0039] First, the learning procedure in the first time step will be described with reference to FIG. 5A . In the first time step, the first learning unit 1031 applies the first attribute learning data 121-1 stored in the attribute data storage unit 120 to the first attribute-specific model 111-1 to learn the first attribute-specific model 111-1. A full set of first attribute learning data 121-1 is used to learn the first attribute-specific model 111-1. Therefore, for example, the first learning unit 1031 may request the full set of first attribute learning data 121-1 from the learning data management unit 102. In response, the learning data management unit 102 may supply the full set of first attribute learning data 121-1 to the first learning unit 1031.

[0040] The first learning unit 1031 inputs product images included in the first attribute learning data 121-1 into the first attribute-specific model 111-1 and obtains, as output 501 (estimation result), a probability distribution of possible features of the first attribute (hereinafter also referred to as an estimated probability distribution). For example, if the first attribute is "pattern," as described above, possible features of the "pattern" are, for example, "border," "plain," "check," "dot," and "print." The output 501 is an estimated probability distribution in this order, for example, {0.6, 0.1, 0.1, 0.1, 0.1}. Furthermore, as described above, in the case of data 35 shown in FIG. 3B, the correct probability distribution, which is the correct data, is {1, 0, 0, 0, 0}.

[0041] The first learning unit 1031 acquires an output 501 (estimated probability distribution) and a correct probability distribution for all of the first attribute learning data 121-1, and calculates an evaluation value (evaluation function) 502 based on the output 501 and the correct probability distribution. In this embodiment, cross-entropy (CE) (also referred to as cross-entropy loss) is used as an evaluation value for training an attribute-specific model such as the first attribute-specific model 111-1. The first learning unit 1031 trains the first attribute-specific model 111-1 based on the evaluation value 502 (for example, so as to minimize the evaluation value 502). The first learning unit 1031 stores the trained first attribute-specific model 111-1 in the learning model storage unit 110. The trained first attribute-specific model 111-1 is used in the training process in the second time step.

[0042] Next, the learning procedure in the second time step will be described with reference to FIG. 5B . In the second time step, the first learning unit 1031 applies the second attribute learning data 121-2 stored in the attribute data storage unit 120 to the second attribute-specific model 111-2 to train the second attribute-specific model 111-2. Similar to the procedure described with reference to FIG. 5A , a full set of second attribute learning data 121-2 is used to train the second attribute-specific model 111-2. Therefore, for example, the first learning unit 1031 may request the full set of second attribute learning data 121-2 from the learning data management unit 102. In response, the learning data management unit 102 may supply the full set of second attribute learning data 121-2 to the first learning unit 1031.

[0043] The first learning unit 1031 inputs product images included in the second attribute learning data 121-2 into the second attribute-specific model 111-2 and obtains an estimated probability distribution for the second attribute as the output 503. Similar to the processing in the first time step, the first learning unit 1031 obtains the output 503 (estimated probability distribution) and the correct probability distribution for all of the second attribute learning data 121-2, and calculates an evaluation value 504 (e.g., cross entropy) based on the output 503 and the correct probability distribution. The first learning unit 1031 trains the second attribute-specific model 111-2 based on the evaluation value 504 (e.g., so as to minimize the evaluation value 504). The first learning unit 1031 stores the trained second attribute-specific model 111-2 in the learning model storage unit 110. The trained second attribute-specific model 111-2 is used in the learning processing in the third time step.

[0044] Furthermore, in the second time step, the second learning unit 1032 trains the attribute prediction model 112. The attribute prediction model 112 in the second time step is a learning model for predicting the first attribute and the second attribute. The second learning unit 1032 applies the first attribute learning data 121-1 and the second attribute learning data 121-2 stored in the attribute data storage unit 120 to the attribute prediction model 112 to train the attribute prediction model 112. To train the attribute prediction model 112, a portion of the first attribute learning data 121-1 and the full set of second attribute learning data 121-2 are used. Therefore, for example, the second learning unit 1032 may request a portion of the first attribute learning data 121-1 and the full set of second attribute learning data 121-2 from the learning data management unit 102. In response to this, the learning data management unit 102 may supply a portion of the first attribute learning data 121-1 and the full set of second attribute learning data 121-2 to the second learning unit 1032. Note that in the figures of the present disclosure, a portion of an area of ​​attribute learning data that is shaded indicates that it is a portion of the attribute learning data.

[0045] The second learning unit 1032 inputs product images included in part of the first attribute learning data 121-1 to the attribute prediction model 112 and obtains an estimated probability distribution for the first attribute as output 505. Furthermore, the second learning unit 1302 inputs product images included in part of the first attribute learning data 121-1 to the first attribute-specific model 111-1 and obtains an estimated probability distribution for the first attribute as output 506. The second learning unit 1032 obtains outputs 505 (estimated probability distributions) and 506 (estimated probability distributions) for all of the part of the first attribute learning data 121-1, and calculates an evaluation value 507 representing the similarity based on the outputs 505 and 506. In this embodiment, Kullback-Leibler divergence (also referred to as KL divergence loss) is used as the evaluation value representing the similarity. The second learning unit 1032 trains the attribute prediction model 112 so that the output 505 and the output 506 match (i.e., so that the similarity is high). When the KL divergence is used as the evaluation value 507, the second learning unit 1032 trains the attribute prediction model 112 so as to minimize the evaluation value 507.

[0046] In this way, the second learning unit 1302 learns the attribute prediction model 112 so that the output 505 from the first attribute-dedicated model 111-1 matches the output 506 from the attribute prediction model 112. In other words, the second learning unit 1302 learns a dedicated model trained for a specific attribute (the first attribute-dedicated model 111-1 in the example of FIG. 5B ) so that the output from the dedicated model matches the output from the learning model to be trained (the attribute prediction model 112 in the example of FIG. 5B ), thereby performing so-called knowledge distillation.

[0047] Furthermore, the second learning unit 1032 inputs product images included in the full set of second attribute learning data 121-2 into the attribute prediction model 112 and obtains an estimated probability distribution for the second attribute as output 508. The second learning unit 1032 obtains the output 508 (estimated probability distribution) and a correct answer probability distribution for all of the second attribute learning data 121-2, and calculates an evaluation value 509 (e.g., cross entropy) based on the output 508 and the correct answer probability distribution. The second learning unit 1032 trains the attribute prediction model 112 based on the evaluation value 509 (e.g., so as to minimize the evaluation value 509).

[0048] In this way, the second learning unit 1032 uses knowledge distillation to train the attribute prediction model 112, thereby making it possible to prevent forgetting of tasks for the first attribute that have already been learned. The second learning unit 1032 stores the trained attribute prediction model 112 in the learning model storage unit 110.

[0049] Next, the learning procedure in the third time step will be described with reference to FIG. 5C . In the third time step, the first learning unit 1031 applies the third attribute learning data 121-3 stored in the attribute data storage unit 120 to the third attribute-specific model 111-3 to train the third attribute-specific model 111-3. Similar to the procedure described with reference to FIG. 5A , a full set of third attribute learning data 121-3 is used to train the third attribute-specific model 111-3. Therefore, for example, the first learning unit 1031 may request the full set of third attribute learning data 121-3 from the learning data management unit 102. In response, the learning data management unit 102 may supply the full set of third attribute learning data 121-3 to the first learning unit 1031.

[0050] The first learning unit 1031 inputs product images included in the third attribute learning data 121-3 into the third attribute-specific model 111-3 and obtains an estimated probability distribution for the third attribute as the output 510. Similar to the processing in the first time step, the first learning unit 1031 obtains the output 510 (estimated probability distribution) and the correct probability distribution for all of the third attribute learning data 121-3, and calculates an evaluation value 511 (e.g., cross entropy) based on the output 510 and the correct probability distribution. The first learning unit 1031 trains the third attribute-specific model 111-3 based on the evaluation value 511 (e.g., so as to minimize the evaluation value 511). The first learning unit 1031 stores the trained third attribute-specific model 111-3 in the learning model storage unit 110. The trained third attribute-specific model 111-3 is used in the learning processing in the fourth time step.

[0051] Furthermore, in a third time step, the second learning unit 1032 trains the attribute prediction model 112. The attribute prediction model 112 in the third time step is a learning model for predicting a third attribute from a first attribute. The second learning unit 1032 applies the first attribute learning data 121-1, the second attribute learning data 121-2, and the third attribute learning data 121-3 stored in the attribute data storage unit 120 to the attribute prediction model 112 to train the attribute prediction model 112. To train the attribute prediction model 112, a portion of the first attribute learning data 121-1, a portion of the second attribute learning data 121-2, and the full set of third attribute learning data 121-3 are used. Therefore, for example, the second learning unit 1032 may request a portion of the first attribute learning data 121-1, a portion of the second attribute learning data 121-2, and the full set of third attribute learning data 121-3 from the learning data management unit 102. In response to this, the learning data management unit 102 can supply a portion of the first attribute learning data 121-1, a portion of the second attribute learning data 121-2, and the full set of third attribute learning data 121-3 to the second learning unit 1032.

[0052] The second learning unit 1032 inputs product images included in a portion of the first attribute learning data 121-1 to the attribute prediction model 112 and obtains an estimated probability distribution for the first attribute as output 512. The second learning unit 1302 also inputs product images included in a portion of the first attribute learning data 121-1 to the first attribute-specific model 111-1 and obtains an estimated probability distribution for the first attribute as output 513. The second learning unit 1032 obtains output 512 (estimated probability distribution) and output 513 (estimated probability distribution) for all of the portion of the first attribute learning data 121-1 and calculates an evaluation value 514 (e.g., KL divergence) based on the output 512 and the output 513. The second learning unit 1032 trains the attribute prediction model 112 based on the evaluation value 514 (e.g., so as to minimize the evaluation value 514). The second learning unit 1032 also inputs product images included in part of the second attribute learning data 121-2 to the attribute prediction model 112 and obtains an estimated probability distribution for the second attribute as output 515. The second learning unit 1302 also inputs product images included in part of the second attribute learning data 121-2 to the second attribute-specific model 111-2 and obtains an estimated probability distribution for the second attribute as output 516. The second learning unit 1032 obtains outputs 515 (estimated probability distributions) and 516 (estimated probability distributions) for all of the part of the second attribute learning data 121-2, and calculates an evaluation value 517 (e.g., KL divergence) based on the outputs 515 and 516. The second learning unit 1032 trains the attribute prediction model 112 based on the evaluation value 517 (e.g., so as to minimize the evaluation value 517).

[0053] Furthermore, the second learning unit 1032 inputs product images included in the full set of third attribute learning data 121-3 into the attribute prediction model 112 and obtains an estimated probability distribution for the third attribute as output 518. The second learning unit 1032 obtains the output 518 (estimated probability distribution) and a correct answer probability distribution for all of the third attribute learning data 121-2, and calculates an evaluation value 519 (e.g., cross entropy) based on the output 518 and the correct answer probability distribution. The second learning unit 1032 trains the attribute prediction model 112 based on the evaluation value 519 (e.g., so as to minimize the evaluation value 519). The second learning unit 10312 stores the trained attribute prediction model 112 in the learning model storage unit 110.

[0054] FIG. 5D is a diagram illustrating the learning procedure performed by the first learning unit 1031 and the second learning unit 1302 at the Nth time step. The learning procedure shown in FIG. 5D corresponds to a generalized version of the learning procedures performed at the second to Nth time steps. Similar to the procedures described with reference to FIGS. 5B and 5C, the first learning unit 1301 inputs a full set of Nth attribute learning data 121-N to the Nth attribute-specific model 111-N and trains the Nth attribute-specific model 111-N based on the output 520 and the evaluation value 521. Similarly to the procedures described with reference to FIGS. 5B and 5C, the second learning unit 1032 trains the attribute prediction model 112. The attribute prediction model 112 at the Nth time step is a learning model for predicting the Nth attribute from the first attribute. Taking the first attribute, the second attribute, and the (N-1)th attribute as examples, the second learning unit 1032 trains the attribute prediction model 112 based on an evaluation value 524 based on outputs 522 and 523 for a portion of the first attribute learning data 121-1, an evaluation value 527 based on outputs 525 and 526 for a portion of the second attribute learning data 121-1, and an evaluation value 530 based on outputs 528 and 529 for a portion of the (N-1)th attribute learning data 121-1. Furthermore, the second learning unit 1302 inputs the full set of Nth attribute learning data 121-N to the attribute prediction model 112, and trains the attribute prediction model 112 based on outputs 531 and the evaluation value 532.

[0055] When cross entropy and KL divergence are used as evaluation values, the loss function L used by the second learning unit 1032 to train the attribute prediction model 112 in the Nth time step can be expressed as in equation (1). Here, CE represents cross entropy, KL represents KL divergence, τ represents temperature, σ represents softmax function, λ represents log-softmax function, and α represents balancing weight. ｏ ｉ represents the output from the i-th attribute specialized model (i is an integer equal to or greater than 1). ＾ ｏ ｉ represents the output for the i-th attribute from the attribute prediction model 112. ＾ Ｎ represents the output for the Nth attribute from the attribute prediction model 112. Ｎ represents the correct data for the Nth attribute. As can be seen from equation (1), the loss function is based on a value obtained by multiplying the evaluation values ​​(KL divergence) for the first attribute to the (N-1)th attribute by a weight α and adding them together, and a value obtained by multiplying the evaluation value (cross entropy) for the Nth attribute by a weight (1-α). The second learning unit 1032 trains the attribute prediction model 112 so as to optimize the value derived by the loss function (for example, to minimize the value).

[0056] In this way, the learning unit 103 trains the attribute prediction model 112 for predicting the first to N-th attributes for the first to (N-1)-th attributes according to knowledge distillation. Specifically, for the first to (N-1)-th attributes, the learning unit 103 trains the attribute prediction model 112 using the trained first attribute-specific models 111-1 to (N-1)-th attribute-specific models 111-(N-1), which have been trained using a full set of training data for the attributes. This makes it possible to train the attribute prediction model 112 so as to suppress catastrophic forgetting for the first to (N-1)-th attributes.

[0057] The attribute prediction models 112 trained in the second time step to the (N-1)th time step may be stored in the training model storage unit 110 separately from the attribute prediction model 112 trained in the Nth time step. The attribute prediction model 112 trained in the Nth time step is a training model for predicting the Nth attribute from the first attribute, and the attribute prediction model 112 trained in the nth time step (n is an integer of 2 or more and N-1 or less) is a training model for predicting the nth attribute from the first attribute.

[0058] 5A to 5D , the procedure for each time step has been described, but the second learning unit 1032 may train the attribute prediction model 112 for predicting the Nth attribute from the first attribute using a procedure different from the above. This modification corresponds to a process of training only the attribute prediction model 112 for predicting the Nth attribute from the first attribute.

[0059] In this modification, first, the first learning unit 1301 trains the first attribute-specific model 111-1 to the (N-1)th attribute-specific model 111-(N-1), respectively, using the first attribute learning data 121-1 to the (N-1)th attribute learning data 121-(N-1). This learning procedure corresponds, for example, to the learning procedure of the first learning unit 1301 described with reference to FIGS. 5A to 5D. Next, the second learning unit 1302 trains the attribute-specific model 112 according to the procedure described with reference to FIG. 5D. The learning procedure according to this modification also makes it possible to train the attribute prediction model 112 so as to suppress catastrophic forgetting for the first attribute to the (N-1)th attribute.

[0060] [Processing Flow] Next, a processing flow executed by the information processing device 100 according to this embodiment will be described with reference to Figures 6A, 6B, and 7. The processing shown in Figures 6A, 6B, and 7 can be realized by the CPU 21 of the information processing device 100 loading a program stored in the ROM 22 or the like into the RAM 23 and executing the program.

[0061] (1-1) Learning Process of Attribute Prediction Model 112 FIG. 6A shows a flowchart of the learning process of the attribute prediction model 112 for predicting the Nth attribute from the first attribute, which is executed by the information processing device 100. In S601, the learning unit 103 sets a parameter i to 1. i corresponds to the time step in the learning procedure described with reference to FIGS. 5A to 5D. In S602, the learning unit 103 determines whether i is 1. If i is 1 (Yes in S602), the process proceeds to S603; if not (No in S602), the process proceeds to S605.

[0062] S603 is a learning process in the first time step. In S603, the first learning unit 1031 uses the first attribute learning data 121-1 to train the first attribute specialized model 111-1. The learning procedure in S603 is as described with reference to FIG. 5A . After training the first attribute specialized model 111-1, the first learning unit 1031 stores the trained first attribute specialized model 111-1 in the learning model storage unit 110. Then, in S604, the learning unit 103 increments the parameter i. Thereafter, the process proceeds to S602.

[0063] Steps S605 to S606 are learning processes for the ith time step (i is 2 or greater). In step S605, the first learning unit 1031 uses the ith attribute learning data 121-i to train the ith attribute specialized model 111-i. After training the ith attribute specialized model 111-i, the first learning unit 1031 stores the trained ith attribute specialized model 111-i in the learning model storage unit 110. Next, in step S606, the second learning unit 1032 trains the attribute prediction model 112 using the trained first attribute specialized model 111-1 through the (i-1)th attribute specialized model 111-(i-1), a portion of each of the first attribute learning data 121-1 through the (i-1)th attribute learning data 121-(i-1), and the ith attribute learning data 121-i. The learning procedure in step S606 is as described with reference to FIGS. 5B to 5D . After learning the attribute prediction model 112, the second learning unit 1032 may store the attribute prediction model 112 in the learning model storage unit 110. The second learning unit 1302 may store the attribute prediction model 112 learned in the i-th time step in the learning model storage unit 110 so that the attribute prediction model 112 can be identified as a learning model for predicting the i-th attribute from the first attribute. Then, in S607, the learning unit 103 determines whether the parameter i is N. If i is N (Yes in S607), the process proceeds to S608. If not (No in S607), the process proceeds to S604.

[0064] In S608, the second learning unit 1032 stores the learned attribute prediction model 112 in the learning model storage unit 110 as a learning model for predicting the Nth attribute from the first attribute.

[0065] (1-2) Training Process of Attribute Prediction Model 112 (Variation) Next, a variation of the process described above in (1-1) will be described. Fig. 6B shows a flowchart of a variation of the training process of the attribute prediction model 112 executed by the information processing device 100. As described above, this variation is a process in which only the attribute prediction model 112 for predicting the Nth attribute from the first attribute is trained.

[0066] In S611, the first learning unit 1031 uses the first attribute learning data 121-1 to the (N-1)th attribute learning data 121-(N-1), respectively, to train the first attribute-specific models 111-1 to 111-(N-1). The training procedure in S611 is as described with reference to FIG. 5A. After training the first attribute-specific models 111-1 to 111-(N-1), the first learning unit 1031 stores the trained first attribute-specific models 111-1 to 111-(N-1) in the training model storage unit 110.

[0067] In S612, the second learning unit 1032 trains the attribute prediction model 112 using the trained first attribute dedicated model 111-1 to the (N-1)th attribute dedicated model 111-(N-1), a portion of each of the first attribute learning data 121-1 to the (N-1)th attribute learning data 121-(N-1), and the Nth attribute learning data 121-N. The learning procedure in S612 is as described with reference to FIG. 5D.

[0068] In S613, the second learning unit 1032 stores the learned attribute prediction model 112 in the learning model storage unit 110 as a learning model for predicting the Nth attribute from the first attribute.

[0069] (2) Attribute Prediction Process of Product Included in Product Image After the process of (1-1) or (1-2) described above, the information processing device 100 predicts the attribute of the product included in the product image. Fig. 7 shows a flowchart of the attribute prediction process executed by the information processing device 100. In this process, a trained attribute prediction model 112 for predicting the Nth attribute (N is an integer equal to or greater than 2) from the first attribute is used.

[0070] In S71, the image acquisition unit 104 acquires a product image including a target product (a product for which attributes are to be predicted). For example, the image acquisition unit 104 acquires the product image by having an operator of the information processing device 100 operate the information processing device 100 to access an arbitrary e-commerce site and then select a product image including the target product. The image acquisition unit 104 can also acquire a product image by acquiring a product image transmitted from an external device such as a user device or a URL (Uniform Resource Locator) indicating the product image.

[0071] In S72, the attribute prediction unit 105 inputs the product image acquired by the image acquisition unit 104 into the trained attribute prediction model 112, and predicts N attributes of the first attribute from the first attribute of the target product included in the product image. Specifically, the attribute prediction unit 105 extracts the N attribute types of the target product, and predicts the features of each of the N attribute types.

[0072] In S73, the output unit 106 outputs information on the attributes predicted by the attribute prediction unit 105 (attribute prediction results). Specifically, the output unit 106 outputs the N attribute types predicted by the attribute prediction unit 105 and the features (feature values) of each of the N attribute types. The output unit 106 may display the attribute prediction results on the display unit 26 (FIG. 2). Furthermore, when a product image is acquired from an external device such as a user device, the output unit 106 may transmit the product image to the external device via the communication I / F 27 (FIG. 2) so that the image is displayed on the display unit of the external device.

[0073] In this way, when the information processing device 100 according to this embodiment trains the attribute prediction model 112 for predicting multiple attributes of a product from a product image containing the product, it trains one or more attributes that have already been trained using a dedicated training model trained specifically for the one or more attributes. This makes it possible to train the attribute prediction model 112 with new attributes while suppressing catastrophic forgetting.

[0074] In this embodiment, the learning procedure of the learning model and the attribute prediction procedure for predicting multiple attributes of a product when the product image includes one product have been described. When the product image includes multiple products, the information processing device 100 may be configured to divide the product image into partial images including each of the multiple products using a known image recognition process, and predict multiple attributes for each partial image using the procedure described in this embodiment.

[0075] Although specific embodiments have been described above, these embodiments are merely examples and are not intended to limit the scope of the present invention. The devices and methods described herein may be embodied in forms other than those described above. Furthermore, appropriate omissions, substitutions, and modifications may be made to the above-described embodiments without departing from the scope of the present invention. Such omissions, substitutions, and modifications are included within the scope of the claims and their equivalents, and belong to the technical scope of the present invention.

[0076] The disclosure of this embodiment includes the following configuration: [1] An information processing device including a learning unit that trains an attribute prediction model for predicting N attributes (N is an integer equal to or greater than 2) of a product from an image including the product, the learning unit including: a first learning unit that trains an ith learning model for predicting an ith attribute (i is an integer from 1 to N-1) of the product using ith learning data for the ith attribute; and a second learning unit that trains the attribute prediction model using the ith learning model trained by the first learning unit, a portion of the ith learning data, and Nth learning data for an Nth attribute of the product.

[0077] [2] The information processing device described in [1], wherein the first learning unit trains the i-th learning model using the estimation result of the i-th attribute obtained by applying the i-th learning data to the i-th learning model.

[0078] [3] The information processing device described in [2], wherein the first learning unit trains the i-th learning model using an evaluation value obtained from the estimation result of the i-th attribute.

[0079] [4] The information processing device according to [3], wherein the evaluation value is cross entropy.

[0080] [5] The information processing device described in any of [1] to [4], wherein the second learning unit trains the attribute prediction model using first estimation results for each of the first attribute to the (N-1)th attribute obtained by applying a portion of the i-th learning data to the i-th learning model, and second prediction results for each of the first attribute to the N-th attribute obtained by applying a portion of the i-th learning data and the N-th learning data to the attribute prediction model.

[0081] [6] The information processing device described in [5], wherein the second learning unit trains the attribute prediction model using a first estimation result for each of the first attribute to the (N-1) attributes, (N-1) first evaluation values ​​obtained by a second prediction result for each of the first attribute to the (N-1) attributes, and a second evaluation value obtained by a second prediction result for the N attribute.

[0082] [7] The information processing device according to [6], wherein each of the (N-1) first evaluation values ​​is a KL divergence, and the second evaluation value is a cross entropy.

[0083] [8] The information processing device described in [7], wherein the second learning unit trains the attribute prediction model using a loss function based on a value obtained by multiplying each of the (N-1) first evaluation values ​​by a weight α and adding them up, and a value obtained by multiplying the second evaluation value by a weight (1-α).

[0084] [9] An information processing device described in any of [1] to [8], further comprising an acquisition unit that acquires an image including a target product, and a prediction unit that inputs the image acquired by the acquisition unit into the attribute prediction model trained by the second learning unit, and predicts the first attribute to the Nth attribute of the target product.

[0085]

[10] An information processing method for training an attribute prediction model for predicting N attributes (N is an integer equal to or greater than 2) of a product from an image including the product, the information processing method including: training an i-th learning model for predicting an i-th attribute (i is an integer from 1 to N-1) of the product using i-th learning data for the i-th attribute; and training the attribute prediction model using the i-th learning model trained by the first learning unit, a portion of the i-th learning data, and N-th learning data for the N-th attribute of the product.

[0086]

[11] An information processing program for training an attribute prediction model for predicting N attributes (N is an integer equal to or greater than 2) of a product from an image including the product, the information processing program causing a computer to execute the following steps: training an i-th learning model for predicting an i-th attribute (i is an integer from 1 to N-1) of the product using i-th learning data for the i-th attribute; and training the attribute prediction model using the i-th learning model trained by the first learning unit, a portion of the i-th learning data, and N-th learning data for the N-th attribute of the product.

[0087] 100: Information processing device, 101: Learning data acquisition unit, 102: Learning data management unit, 103: Learning unit, 1301: First learning unit, 1302: Second learning unit, 104: Image acquisition unit, 105: Attribute prediction unit, 106: Output unit, 110: Learning model storage unit, 111-1: First attribute specialized model, 111-N: Nth attribute specialized model, 112: Attribute prediction model, 120: Attribute data storage unit, 121-1: First attribute learning data, 121-N: Nth attribute learning data

Claims

1. a learning unit that learns an attribute prediction model for predicting N attributes (N is an integer equal to or greater than 2) of a product from an image including the product, The learning unit is a first learning unit that learns an i-th learning model for predicting an i-th attribute (i is an integer from 1 to N-1) of the product using i-th learning data for the i-th attribute; a second learning unit that learns the attribute prediction model by using the i-th learning model learned by the first learning unit, a portion of the i-th learning data, and N-th learning data for the N-th attribute of the product; An information processing device having the above configuration.

2. The first learning unit learns the i-th learning model using an estimation result of the i-th attribute obtained by applying the i-th learning data to the i-th learning model. The information processing device according to claim 1 .

3. The first learning unit learns the i-th learning model by using an evaluation value obtained from the estimation result of the i-th attribute. The information processing device according to claim 2 .

4. The information processing apparatus according to claim 3 , wherein the evaluation value is a cross entropy.

5. The second learning unit is a first estimation result for each of the first attribute to the (N−1)th attribute obtained by applying a portion of the i-th training data to the i-th training model; and second estimation results of the first attribute to the N-th attribute obtained by applying a portion of the i-th training data and the N-th training data to the attribute prediction model; and The attribute prediction model is trained using The information processing device according to claim 1 .

6. the second learning unit learns the attribute prediction model using a first estimation result for each of the first attribute to the (N-1) attributes, (N-1) first evaluation values ​​obtained by a second estimation result for each of the first attribute to the (N-1) attributes, and a second evaluation value obtained by a second estimation result for the N attribute; The information processing device according to claim 5 .

7. Each of the (N-1) first evaluation values ​​is a KL divergence, and each of the second evaluation values ​​is a cross entropy. The information processing device according to claim 6.

8. The information processing device according to claim 7, wherein the second learning unit trains the attribute prediction model using a loss function based on a value obtained by multiplying each of the (N-1) first evaluation values ​​by a weight α and adding them up, and a value obtained by multiplying the second evaluation value by a weight (1-α).

9. An acquisition unit that acquires an image including a target product; a prediction unit that inputs the image acquired by the acquisition unit into the attribute prediction model trained by the second learning unit to predict a first attribute to an Nth attribute of the target product; Further comprising The information processing device according to claim 1 .

10. An information processing method executed by an information processing device for learning an attribute prediction model for predicting N attributes (N is an integer equal to or greater than 2) of a product from an image including the product, the method comprising: training an i-th learning model for predicting an i-th attribute (i is an integer from 1 to N-1) of the product using i-th learning data for the i-th attribute; training the attribute prediction model using the trained i-th learning model, a portion of the i-th learning data, and N-th learning data for the N-th attribute of the product; An information processing method comprising:

11. An information processing program for training an attribute prediction model for predicting N attributes (N is an integer equal to or greater than 2) of a product from an image including the product, the information processing program comprising: training an i-th learning model for predicting an i-th attribute (i is an integer from 1 to N-1) of the product using i-th learning data for the i-th attribute; training the attribute prediction model using the trained i-th learning model, a portion of the i-th learning data, and N-th learning data for the N-th attribute of the product; An information processing program that causes a computer to execute the following.