Information processing device, information processing method, and information processing program

Abstract

To provide thumbnail images reflecting intent of a search by a user.SOLUTION: An information processing device of the present invention comprises a search unit, a selection unit, and a provision unit. The search unit searches for a product based on a search query specified by a user. The selection unit selects a candidate image most relevant to the search query among multiple candidate images associated with search targets included in a result of the search by the search unit from each search target. The provision unit provides the candidate images selected by the selection unit in the form of thumbnail images of the search targets.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, a technique for image search using a VSE (Visual-Semantic Embedding) model has been disclosed. For example, in an image search using a VSE model, a specified search keyword and a search target image are converted into feature vectors, and an image according to the similarity between the two feature vectors is output as a search result (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180941 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional techniques merely search for images that have similar feature vectors to a search keyword, and do not take into consideration providing thumbnail images that reflect the user's search intent.

[0005] The present invention has been made in consideration of the above, and has an object to provide an information processing device, an information processing method, and an information processing program that are capable of providing thumbnail images that reflect a user's search intent. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the information processing device of the present invention includes a search unit that searches for search targets based on a search query specified by a user, a selection unit that selects from each of the search targets a candidate image that has the highest relationship with the search query from among a plurality of candidate images linked to the search targets included in the search results by the search unit, and a provision unit that provides the candidate image selected by the selection unit as a thumbnail image of the search target. Effect of the Invention

[0007] According to the present invention, it is possible to provide thumbnail images that reflect the user's search intent. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a provision system according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of information processing according to the embodiment. [Diagram 3] FIG. 3 is an explanatory diagram of a distributed representation space according to the embodiment. [Figure 4] FIG. 4 is a schematic diagram of thumbnail image generation processing according to the embodiment. [Diagram 5] FIG. 5 is a block diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of information stored in a user information storage unit according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of information stored in a product information storage unit according to the embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a method for collecting learning data according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of learning data according to the embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a composite image according to the embodiment. [Figure 11]FIG. 11 is a flowchart illustrating an example of a processing procedure of information processing according to the embodiment. [Figure 12] FIG. 12 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the information processing device, the information processing method, and the information processing program according to the present application will be described in detail with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to the embodiments. In addition, the following description will be given of a case where the search target is a product handled in an online shopping mall.

[0010] [Embodiment] [1.1 System] First, a provision system shown in Fig. 1 will be described. Fig. 1 is a diagram showing a configuration example of a provision system according to an embodiment. As shown in Fig. 1, the provision system includes an information processing device 1 and a user terminal 10. The information processing device 1 and the user terminal 10 are connected to each other via a predetermined communication network (network N) so as to be able to communicate with each other via wired or wireless communication.

[0011] The information processing device 1 provides an electronic commerce service that provides (searches, sells, etc.) products including clothing (also called fashion items, including clothes, underwear, footwear (also called shoes), headwear (caps, hats, etc.), bags, accessories (also called accessories), etc.), cosmetics (also called fashion items), etc. The information processing device 1 also provides a coordination service that accepts posts from users of content (images, videos, articles, etc.) showing outfits using multiple clothing items, content showing makeup using one or more cosmetics, etc., and provides (searches, distributes, etc.) the posted content to other users. For example, the information processing device 1 can be realized by a server device, a cloud system, etc.

[0012] Moreover, the information processing device 1 provides various services to users by using a distributed representation space learned in advance, as described later. Moreover, images and keywords (including tags, etc.) are projected in the distributed representation space according to the embodiment, and the closeness between images, images and keywords, and keywords and keywords are projected. Moreover, the closer the two are semantically, the closer they are learned to be in the distributed representation space (and vice versa). That is, the similarity between images, images and keywords, and keywords and keywords can be measured as a distance in the distributed representation space. Note that the closeness in the distributed representation space according to the embodiment includes, for example, closeness based on Euclidean distance and closeness based on cosine distance.

[0013] The information processing device 1 may also have a function as a web server that provides a website related to the service. The information processing device 1 may also be a device that distributes information to be displayed on applications related to various services installed in the user terminal 10 to the user terminal 10. The information processing device 1 may also be a device that distributes the application data itself.

[0014] Furthermore, the information processing device 1 may function as a distribution device that distributes control information to the user terminal 10. Here, the control information is described in, for example, a script language such as JavaScript (registered trademark) or a style sheet language such as CSS (Cascading Style Sheets). Note that the application itself distributed from the information processing device 1 may be regarded as the control information.

[0015] The user terminal 10 is an information processing device used by a user. The user terminal 10 is realized, for example, by a smartphone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), etc. The user terminal 10 displays information distributed by the information processing device 1, a server device that provides a predetermined service, etc., using a web browser or an application.

[0016] [1.2 Information Processing] Next, information processing according to an embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of information processing according to an embodiment. As shown in Fig. 2, the information processing device 1 functions as a web server that provides information on fashion items, and first accepts a search query specified by a user from a user terminal 10 (step S1).

[0017] The information processing device 1 searches for products based on the search query (step S2). For example, products tagged with keywords or the like corresponding to the search query are output as search results.

[0018] Next, the information processing device 1 selects a thumbnail image for each product included in the search result (step S3). More specifically, the information processing device 1 selects a candidate image that has the highest relationship with the search query from among multiple candidate images associated with each product for each product included in the search result.

[0019] For example, the information processing device 1 selects thumbnail images of each product based on the similarity between the search query (keyword) and each candidate image in the distributed representation space. More specifically, the search query indicates the user's search intent, and the thumbnail image clicked (viewed) by the user who specified the search query (it may be a thumbnail image that was not only clicked but also led to a purchase) can be considered to be an image that better reflects the user's search intent. For example, the information processing device 1 can project each search query, candidate images, etc. into the distributed representation space by using a technology related to the distributed representation space, such as VSE (Visual-Semantic Embedding) or CLIP (Contrastive Language-Image Pre-training).

[0020] Therefore, for example, the information processing device 1 can select a thumbnail image that is highly related to the search query, that is, that reflects the user's search intent, by linking the search query specified by the user with the thumbnail image clicked by the user and using the linked data as learning data. Also, for example, the information processing device 1 may link the search query specified by the user with the thumbnail image not clicked by the user and use the linked data as negative example learning data. Note that in this embodiment, in addition to the search query and the candidate images, a thumbnail image is selected from the candidate images based on the attributes of the user who specified the search query.

[0021] Here, the distributed representation space according to the embodiment will be described with reference to Fig. 3. Fig. 3 is an explanatory diagram of the distributed representation space according to the embodiment. Note that, here, a process of selecting a thumbnail image from a plurality of candidate images associated with one product will be described.

[0022] As shown in Fig. 3, first, the information processing device 1 embeds a plurality of candidate images linked to a product in a distributed representation space (step S11), and embeds a search query and user attributes in the distributed representation space (steps S12 and S13). The user attributes include, for example, information on the user's age, sex, body type, foot type, skin color, residence, click log, search log (search history), purchase log (purchase history), browsing log (browsing history), etc. The user attributes may also include, for example, information on the age, sex, body type, foot type, skin color, residence, click log, search log, purchase log, browsing log, etc. of a user who has a predetermined relationship (including a parent-child relationship, etc.) with the user.

[0023] In the example shown in Fig. 3, the feature vectors of the candidate images are indicated as "image vectors Vi1 to Vi5", the feature vector of the search query is indicated as "query vector Vq", and the feature vector of the user attribute is indicated as "user vector Vu". In the following, when there is no need to distinguish between the image vectors Vi1 to Vi5, they will be referred to as "image vector Vi".

[0024] For example, the information processing device 1 creates a ranking of the image vectors Vi1 to Vi5 according to their positional relationship with the query vector Vq and the user vector Vu. For example, the information processing device 1 creates a ranking in descending order of the image vector Vi having the smallest area of ​​the space (triangle) connecting each of the image vectors Vi1 to Vi5, the query vector Vq, and the user vector Vu. Note that the information processing device 1 may generate a ranking of the image vectors Vi1 to Vi5 in descending order of similarity to either the query vector Vq or the user vector Vu.

[0025] Then, the information processing device 1 selects the first ranked candidate image as a thumbnail image. That is, the information processing device 1 selects, from among the image vectors Vi, a candidate image corresponding to an image vector Vi similar to the query vector Vq and the user vector Vu, as a thumbnail image.

[0026] That is, the information processing device 1 selects a candidate image to be a thumbnail image from among the candidate images associated with the product, according to the search query and the user attributes. More specifically, from among the image vectors Vi corresponding to the candidate images, an image vector Vi similar to a query vector Vq corresponding to the search query and a user vector Vu corresponding to the user attributes is selected as the thumbnail image.

[0027] This allows the information processing device 1 to select a thumbnail image that reflects the user's search intent from multiple candidate images linked to the product. In the example shown in Fig. 3, it is shown that the area of ​​the space connecting the image vector Vi4 with the query vector Vq and the user vector Vu is the smallest, indicating that the candidate image corresponding to the image vector Vi4 is selected as the thumbnail image.

[0028] When the information processing device 1 selects a thumbnail image from each candidate image for each product included in the search result in step S2, it provides the selected thumbnail image to the user terminal 10 as shown in FIG. 2 (step S4).

[0029] Furthermore, the information processing device 1 can also generate thumbnail images for products that do not have image vectors Vi that satisfy the selection conditions for thumbnail images from the image vectors Vi in the distributed representation space. That is, for products for which no appropriate thumbnail image exists, the information processing device 1 generates a new thumbnail image based on the search query. Note that the information processing device 1 may register multiple candidate images generated in advance, and select and provide a thumbnail image from the candidate images registered in advance when no thumbnail image satisfies the selection conditions.

[0030] 4 is a schematic diagram of thumbnail image generation processing according to an embodiment. For example, the information processing device 1 generates a composite image Ic, which is a thumbnail image obtained by combining a product image It of a product for which a suitable thumbnail image is not available and a compositing image Is corresponding to a search query.

[0031] For example, the composite image Is includes candidate images associated with other products and posted images (e.g., coordinated images) posted by each user to a coordination service, etc. The posted images include photos tagged with fashion items worn by each user.

[0032] For example, the information processing device 1 extracts each compositing image Is from candidate images linked to other products and posted images posted by each user based on the search query. For example, the information processing device 1 extracts a compositing image Is from each image according to the similarity between the image vector Vi of each image in the distributed representation space and the query vector Vq of the search query.

[0033] Next, the information processing device 1 uses various image generation AIs to synthesize a composite image Ic by combining the product image It and the composite image Is.

[0034] In this case, for example, the information processing device 1 repeatedly generates the composite image Ic until the area of ​​the space connecting the image vector Vi of the composite image Ic, the query vector Vq of the search query, and the user vector Vu of the user attribute becomes less than or equal to a threshold value, thereby generating a composite image Ic adapted to the search query specified by the user.

[0035] In this way, the information processing device 1 automatically generates thumbnail images for products that do not have thumbnail images corresponding to the search query. This allows the information processing device 1 to provide an appropriate thumbnail image corresponding to the search query entered by the user.

[0036] [2. Information Processing Device] Next, a configuration example of the information processing device 1 according to the embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing a configuration example of the information processing device 1 according to the embodiment. As shown in Fig. 5, the information processing device 1 includes a communication unit 2, a storage unit 3, and a control unit 4. Note that the information processing device 1 may also include an input unit (e.g., a keyboard, a mouse, etc.) that receives various operations from an administrator who uses the information processing device 1, and a display unit (e.g., a liquid crystal display, etc.) that displays various information.

[0037] The communication unit 2 is realized by, for example, a network interface card (NIC) etc. The communication unit 2 is connected to a communication network such as 4G (4th Generation) or 5G (5th Generation) by wire or wirelessly, and transmits and receives information to and from each of the user terminals 10 etc. via the communication network.

[0038] The storage unit 3 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 3 has a user information storage unit 31, a product information storage unit 32, and a model storage unit 33.

[0039] The user information storage unit 31 stores information about each user. Fig. 6 is a diagram showing an example of information stored in the user information storage unit 31 according to the embodiment. As shown in Fig. 6, the user information storage unit 31 stores information items such as "user ID" and "user information" in association with each other.

[0040] "User ID" indicates identification information for identifying a user. "User information" indicates user information. In the example shown in FIG. 6, conceptual information such as "User information #1" and "User information #2" is stored in "User information", but in reality, information on user attributes such as age and gender, and identification information for products (items) owned by the user are stored.

[0041] The product information storage unit 32 stores product information. The product information storage unit 32 is information about products handled in the online shopping mall provided by the information processing device 1. Fig. 7 is a diagram showing an example of information stored in the product information storage unit 32 according to the embodiment.

[0042] As shown in FIG. 7, the product information storage unit 32 stores information items such as "product ID" and "image information" in association with each other. "Product ID" indicates identification information for identifying a product. "Image information" is image information of a product identified by the corresponding product ID. In this embodiment, multiple candidate images are stored as image information for each product.

[0043] The model storage unit 33 stores the model. For example, the model is a learning model that projects each image, a search query (keyword), and user information into the same distributed representation space. For example, the learning model is a model that has been learned by a learning unit 42 described later, and the learning model may be, for example, a VSE model.

[0044] Next, the control unit 4 will be described. The control unit 4 is a controller, and is realized, for example, by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) executing various programs (corresponding to an example of an information processing program) stored in a storage device inside the information processing device 1 using a RAM as a working area. The control unit 4 is also, for example, a controller, and is realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0045] As shown in Fig. 5, the control unit 4 includes an acquisition unit 41, a learning unit 42, a search unit 43, a selection unit 44, a generation unit 45, and a provision unit 46, and realizes or executes the functions and actions of information processing described below. The internal configuration of the control unit 4 is not limited to the configuration shown in Fig. 5, and may be other configurations as long as they perform the information processing described below. Furthermore, the connection relationship between the processing units in the control unit 4 is not limited to the connection relationship shown in Fig. 5, and may be other connection relationships.

[0046] The acquisition unit 41 acquires various information. For example, the acquisition unit 41 acquires user information from the user terminal 10 and registers it in the user information storage unit 31. The acquisition unit 41 also acquires product information from terminals of shop owners (not shown) in the online shopping mall and registers it in the product information storage unit 32.

[0047] The acquisition unit 41 also accepts posts from users, such as content (images, videos, articles, etc.) showing outfits using multiple clothing accessories, and content showing makeup using one or more cosmetics. For example, such content is provided (searched, distributed, etc.) to other users through a platform such as an SNS. Note that such a platform may be a platform provided by the information processing device 1.

[0048] The learning unit 42 learns a model (learning model) stored in the model storage unit 33. First, a method of collecting learning data will be described. Fig. 8 is a diagram showing an example of a method of collecting learning data according to an embodiment.

[0049] 8, for example, the information processing device 1 provides an annotation app to the user terminal 10 and acquires learning data of a learning model through the annotation app. For example, as shown in FIG 8, the annotation app displays a search query to be learned and multiple images.

[0050] Then, the user selects an image that matches the search query through the annotation app, and the information processing device 1 acquires learning data that links the search query with the image selected by the user. In the example shown in FIG. 8, the search query is "golf", and the user selects an image that is associated with "golf". Note that the search query may be "golf knit sweater", and the user may select an image that is associated with "golf knit sweater".

[0051] The information processing device 1 can collect learning data of images corresponding to the search queries by having the user select an image corresponding to each search query through the annotation app.

[0052] Note that the annotation app is not limited to the example shown in Fig. 8, and may display two images at the same time, may change the arrangement of the images depending on the user, may display the images in a circular arrangement, or may display the images while constantly moving the arrangement. This can eliminate the bias that images are more likely to be selected depending on their arrangement.

[0053] In addition to the above example, the information processing device 1 may collect learning data from log information of each user in an online shopping mall, for example. Fig. 9 is a diagram showing an example of learning data according to the embodiment.

[0054] As shown in FIG. 9, the learning data includes items such as "user," "search query," "clicked image," and "unclicked image." "User" indicates the attributes of a user. "Search query" indicates a search query specified by a corresponding user. The learning data may also include information regarding the time it took a user to select an image, images displayed at the same time, the position (display position) of the image selected by the user, and the like.

[0055] A "clicked image" refers to an image of a product included in the search results for the corresponding search query by the corresponding user and that was clicked on (e.g., a thumbnail image), while an "unclicked image" refers to an image of a product included in the search results for the corresponding search query by the corresponding user and that was not clicked on.

[0056] That is, in this case, it is possible to collect learning data based on the logs of each user in the online shopping mall. After collecting this learning data, the learning unit 42 trains the learning model based on the collected learning data. Specifically, the learning unit 42 trains the learning model so that the feature vector of the search query and the feature vector of the image linked to the search query become similar feature vectors in the distributed representation space.

[0057] More specifically, the learning unit 42 trains the learning model so that similar images have similar feature vectors, similar search queries (keywords) have similar feature vectors, and the search queries that serve as learning data and the images linked to the search queries have similar feature vectors.

[0058] The learning unit 42 also learns a learning model capable of projecting the user attributes of each user into the distributed representation space. For example, the user attributes include so-called demographic attributes such as the user's age, sex, and residence, and the user's log in the online shopping mall, and the user's log includes a search history, a browsing history, a purchase history, and the like.

[0059] Furthermore, the learning unit 42 generates a pseudo ranking of thumbnail images expressing the search intent using a method such as Serial Rank, and evaluates the learning model. Furthermore, the learning unit 42 trains the learning model so that similar user attributes become similar feature vectors in the distributed representation space. Note that the learning data is not limited to the above example, and may include logs of each user in the coordination service. The user's log in the coordination service includes browsing history, search history, posting history, product purchase history through the coordination service, etc.

[0060] The search unit 43 searches for a search target (product) based on a search query specified by a user. Specifically, the search unit 43 acquires a user ID and a search query from the user terminal 10, and searches the product information storage unit 32 for products that match the search query specified by the user. Then, the search unit 43 passes the user ID and the product information included in the search result to the selection unit 44.

[0061] The selection unit 44 selects, for each product, a candidate image having the highest relationship with the search query from among multiple candidate images linked to the products included in the search results by the search unit 43. Specifically, the selection unit 44 uses a learning model to project the candidate image, user attribute, and search query for each product into a distributed representation space, and calculates the area of ​​the space connecting each image vector Vi, user vector Vu, and query vector Vq in the distributed representation space (see FIG. 2).

[0062] For example, the selection unit 44 selects as the thumbnail image a candidate image corresponding to the image vector Vi having the smallest area of ​​the space. At this time, for example, when the area of ​​the space connecting the image vector Vi, the user vector Vu, and the query vector Vq is equal to or larger than a threshold, that is, when there is no thumbnail image corresponding to the search query, the selection unit 44 passes information about the product to the generation unit 45 and instructs the generation of a thumbnail image.

[0063] The generation unit 45 generates thumbnail images of products included in the search results by the search unit 43 based on the relationship between the search query and the images. The generation unit 45 generates thumbnail images for products for which generation of thumbnail images is instructed by the selection unit 44, that is, for products for which multiple candidate images linked to the products do not satisfy the selection conditions for the thumbnail image.

[0064] For example, the generation unit 45 generates, as a thumbnail image, a coordinated image including products included in the search results by the search unit 43. The coordinated image is an image using a plurality of clothing items. For example, the generation unit 45 generates the coordinated image by combining a plurality of candidate images linked to products corresponding to each category such as tops, bottoms, and shoes.

[0065] First, the generation unit 45 extracts an image for synthesis based on the search query. For example, the generation unit 45 uses a learning model to project product images of products handled in the online shopping mall and posted images posted by users into a distributed representation space, and extracts an image similar to a query vector Vq of the search query in the distributed representation space as an image for synthesis.

[0066] For example, based on the product image It for which a thumbnail image is to be generated and the search query, the generation unit 45 extracts an image for synthesis that is similar to the search query in the distributed representation space and suits the product as an image for synthesis. For example, the generation unit 45 extracts a plurality of patterns of images for synthesis that differ according to the bone structure, style, season, etc. of the model appearing in the posted image. The generation unit 45 may extract a plurality of patterns of images for synthesis that differ according to the compatibility of the product (e.g., a top, etc.) with the model appearing in the posted image, the compatibility of the product with other products (e.g., a skirt, etc.) worn by the model appearing in the posted image, and the compatibility of the product with the background appearing in the posted image.

[0067] When extracting a compositing image from posted images posted by a user, the generation unit 45 may exclude posted images that include products of a brand that is not handled in the online shopping mall from the candidates for the compositing image.

[0068] For example, the posted images are tagged with the products (items) that each user is wearing, and based on such tags, the generation unit 45 can identify posted images that include products that are not available in the online shopping mall.

[0069] Note that the generating unit 45 may generate a composite image by overwriting products not handled in the online shopping mall when generating a composite image using the images to be composited, without excluding all posted images from the target images for composite. In other words, by not displaying products of brands not handled in the online shopping mall in thumbnail images, it is possible to suppress, for example, copyright infringement.

[0070] The generation unit 45 may prepare images corresponding to each search query in advance and extract (select) an image to be synthesized from among these images.

[0071] After extracting the composite image, the generating unit 45 generates a composite image from the product image It of the target product and the composite image using an image generation AI. For example, the generating unit 45 generates a collage image by arranging each product image (each item image) on the composite image, and then generates a natural thumbnail image from the collage image. At this time, the generating unit 45 may extract cutout images of the items used, background, etc. from each composite image, and then generate a thumbnail image.

[0072] In addition, the generation unit 45 may change the instruction to the image generation AI according to the user or the search query. For example, when the user is a man, the generation unit 45 instructs the image generation AI to generate an image in which a man is a model.

[0073] Also, suppose that the user who originated the search is a man and the search query is "women, one-piece dress, fashionable," and the man is searching for a present for a woman. In this case, the generation unit 45 instructs the image generation AI to generate an image of a woman as a model based on the search query "women." Note that whether the user or the search query is to be prioritized may be determined in advance, for example, on a rule basis.

[0074] Furthermore, when generating a composite image, the generating unit 45 may include, for example, products previously purchased by the user in the composite image. Note that the products previously purchased by the user can be identified, for example, from the user's purchase history in the online shopping mall, but may also be identified from posted images (coordinated images) posted by the user.

[0075] FIG. 10 is a diagram showing an example of a composite image according to an embodiment. For example, as shown in FIG. 10, the generating unit 45 extracts product images of products previously purchased by the user from the user information storage unit 31, and generates a composite image including the extracted product images. In this case, the generating unit 45 may generate a composite image including product images of products selected from the user's log. In this case, the generating unit 45 may extract product images related to products registered in the user's favorites, products viewed in the past, or products purchased by users similar to the target user. In the example shown in FIG. 10, the products previously purchased by the user are a bag and a necklace. The generating unit 45 generates a composite image Ic by further using the product image Ib corresponding to the bag and the product image In corresponding to the necklace as images for synthesis.

[0076] In other words, in this case, the composite image Ic is generated by coordinating a product image (a dress in Figure 10) that corresponds to the search query specified by the user, in other words, a product image that matches the user's search intent, with a bag or necklace that the user previously purchased.

[0077] That is, the composite image Ic can present the user with a sample of the one-piece dress currently being sold worn with a bag or necklace that the user currently owns.

[0078] In this way, the generating unit 45 can generate a composite image Ic that is individually optimized for the user, and this is expected to increase the user's desire to purchase. For example, the generating unit 45 may generate the composite image Ic by combining products designated by the user. In this case, the user may select a product from products purchased in the past, for example, an image of the product photographed by the user may be selected.

[0079] The providing unit 46 provides the user with the thumbnail image selected by the selecting unit 44. At this time, for a product for which a thumbnail image was not selected by the selecting unit 44, the providing unit 46 provides the composite image Ic generated by the generating unit 45 as a thumbnail image.

[0080] This allows the providing unit 46 to provide thumbnail images that reflect the user's search intent.

[0081] [3. Processing flow] Next, a processing procedure executed by the information processing device 1 according to the embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of the processing procedure of the information processing according to the embodiment. The processing shown below is repeatedly executed by the information processing device 1 at a predetermined cycle each time a search query is acquired.

[0082] 11, the information processing device 1 receives a search query from a user (step S101). Then, the information processing device 1 searches for products that match the search query (step S102).

[0083] Next, the information processing device 1 embeds multiple candidate images associated with the products included in the search results, the search query, and user attributes into the distributed representation space (step S103). Next, the information processing device 1 selects thumbnail images from among the candidate images based on each feature vector in the distributed representation space (step S104).

[0084] Next, the information processing device 1 determines whether there is a product whose candidate image does not satisfy the selection conditions (step S105), and if there is a product whose candidate image does not satisfy the selection conditions (step S105: Yes), it generates a thumbnail image of the product (step S106).

[0085] If thumbnail images can be selected from all products (step S105; No), the information processing device 1 proceeds to the process of step S107. Then, the information processing device 1 provides the thumbnail images to the user (step S107) and ends the process.

[0086] 4. Modifications In the above-described embodiment, the information processing device 1 provides thumbnail images related to products, but the present invention is not limited thereto. For example, the present invention may be applied when providing thumbnail images for various Web services such as SNS and video sites. Furthermore, the information processing device 1 may provide the present invention when providing thumbnail images related to outfits in a coordination service. For example, the selection unit may select a candidate image that has the highest relationship with the search query from among multiple candidate images (posted images) associated with outfits included in the search results by the search unit, from each outfit, and provide the selected candidate image as a thumbnail image.

[0087] In the above embodiment, the providing unit 46 provides the composite image Ic generated by the generating unit 45 as a thumbnail image for a product for which a thumbnail image was not selected by the selecting unit 44, but the present invention is not limited to this. Even if there is no candidate image that reflects the user's search intent among the multiple candidate images linked to the product, if a posted image showing an outfit in which the product is used is posted on a coordination site and there is a posted image that reflects the user's search intent among the multiple candidate images, the posted image may be provided as a thumbnail image. Specifically, the information processing device 1 embeds multiple posted images showing outfits using the products included in the search results, the search query, and user attributes in the distributed representation space. Next, the information processing device 1 selects a thumbnail image from the posted images based on each feature vector in the distributed representation space.

[0088] In addition, instead of embedding all posted images showing coordination using products included in the search results into the distributed representation space, the information processing device 1 may first search the coordination site using the same search query and embed the posted images included in the search results into the distributed representation space.

[0089] That is, the search target by the search unit 43 may be expanded to, for example, coordinated images posted on a coordinate service in addition to the products handled in the online shopping mall. In this way, by expanding the search target, it is possible to provide search results that reflect the user's search intent.

[0090] 5. Effects The information processing device 1 of the embodiment includes a search unit 43 that searches for products based on a search query specified by a user, a selection unit 44 that selects from each product a candidate image that has the highest relationship with the search query from among a plurality of candidate images linked to products included in the search results by the search unit 43, and a provision unit that provides the candidate image selected by the selection unit 44 as a thumbnail image of the product.

[0091] The selection unit 44 also projects the search query and the candidate images onto the distributed representation space, and selects candidate images similar to the search query in the distributed representation space. The selection unit 44 also selects candidate images based on user information of the user who specified the search query, and selects the candidate image with the smallest area of ​​space connecting the user information, the search query, and the candidate images in the distributed representation space onto which the user information is projected.

[0092] In addition, during the learning stage, the selection unit 44 selects candidate images using a model that has been trained so that a candidate image that has been pre-annotated as an image corresponding to a search query and the search query become similar vectors in the distributed representation space.

[0093] In addition, during the learning stage, the selection unit 44 selects candidate images using a model that has been trained so that a search query specified by a user and an image selected by the user when using the search query are similar vectors in the distributed representation space.

[0094] In addition, the information processing device 1 is provided with a generation unit 45 that generates a composite image for a product that does not satisfy the selection conditions of the candidate image by the selection unit 44, using an image for synthesis that satisfies the search conditions, and the provision unit 46 provides the composite image generated by the generation unit 45 as a thumbnail image.

[0095] By performing any one or a combination of the above-described processes, the information processing device according to the present application can provide thumbnail images that reflect the user's search intent.

[0096] [6. Hardware Configuration] The information processing device 1 according to the embodiment described above is realized by a computer 1000 having a configuration as shown in Fig. 12. Fig. 12 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device according to the embodiment. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, a HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0097] The CPU 1100 operates and controls each unit based on a program stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, programs that depend on the hardware of the computer 1000, and the like.

[0098] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, etc. The communication interface 1500 receives data from other devices via a network (communication network) N and sends the data to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the network N.

[0099] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse (in FIG. 12, the output devices and the input devices are collectively referred to as "input / output devices") via an input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input / output interface 1600.

[0100] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0101] For example, when the computer 1000 functions as an information processing device according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 4. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via the network N.

[0102] [7. Other] Although the embodiments of the present application have been described above, the present invention is not limited to the contents of these embodiments. The above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-described components can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments.

[0103] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0104] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0105] For example, the above-mentioned information processing device may be realized using multiple server computers, and depending on the functions, the configuration can be flexibly changed, such as by calling an external platform using an API (Application Programming Interface) or network computing.

[0106] Furthermore, the above-described embodiments and modifications can be appropriately combined as long as the processing contents are not contradictory.

[0107] Furthermore, the above-mentioned "section, module, unit" can be read as "means" or "circuit", etc. For example, an acquisition section can be read as an acquisition means or an acquisition circuit. [Explanation of symbols]

[0108] 1. Information processing device 2. Communications Department 3 Storage section 4. Control section 10 User terminal 31 User information storage unit 32 Product information storage section 33 Model Memory Section 41 Acquisition Department 42 Learning Department 43 Search Department 44 Selection section 45 Generation part 46 Providing Department

Claims

1. a search unit that searches for search targets based on a search query specified by a user; a selection unit that selects, from among a plurality of candidate images associated with the search target included in the search results by the search unit, a candidate image that has the highest relationship with the search query from each of the search targets; a providing unit that provides the candidate image selected by the selecting unit as a thumbnail image to be searched; Equipped with The selection unit is In a learning stage, the search query and the candidate images are projected into the distributed representation space using a model trained so that the search query specified by the user and an image selected by the user in a search result based on the search query have similar feature vectors in the distributed representation space, and the candidate images similar to the search query are selected. An information processing device comprising:

2. a search unit that searches for search targets based on a search query specified by a user; a selection unit that selects, from among a plurality of candidate images associated with the search target included in the search results by the search unit, a candidate image that has the highest relationship with the search query from each of the search targets; a providing unit that provides the candidate image selected by the selecting unit as a thumbnail image to be searched; Equipped with The selection unit is In a learning stage, the search query and the candidate images are projected into a distributed representation space using a model that has been trained so that the candidate images, which have been annotated in advance as images corresponding to the search query, and the search query have similar feature vectors in the distributed representation space, and the candidate images similar to the search query are selected. An information processing device comprising:

3. a search unit that searches for search targets based on a search query specified by a user; a selection unit that selects, from among a plurality of candidate images associated with the search target included in the search results by the search unit, a candidate image that has the highest relationship with the search query from each of the search targets; a providing unit that provides the candidate image selected by the selecting unit as a thumbnail image to be searched; Equipped with The selection unit is Selecting a coordinated image using a plurality of clothing items as the candidate image, The providing unit is Providing the coordinated image as the thumbnail image. An information processing device comprising:

4. a search unit that searches for search targets based on a search query specified by a user; a selection unit that selects, from among a plurality of candidate images associated with the search target included in the search results by the search unit, a candidate image that has the highest relationship with the search query from each of the search targets; a generation unit that generates a composite image from an image of the search target that does not satisfy the selection condition of the candidate image by the selection unit; a providing unit that provides the candidate image selected by the selecting unit or the composite image generated by the generating unit as a thumbnail image of the search target; An information processing device comprising:

5. 1. A computer-implemented information processing method, comprising: a search step of searching for a search target based on a search query specified by a user; a selection step of selecting, from among a plurality of candidate images associated with the search target included in the search results by the search step, a candidate image having the highest relevance to the search query from each of the search targets; a providing step of providing the candidate image selected by the selecting step as a thumbnail image to be searched; Including, The selection step includes: In a learning stage, the search query and the candidate images are projected into the distributed representation space using a model trained so that the search query specified by the user and an image selected by the user in a search result based on the search query have similar feature vectors in the distributed representation space, and the candidate images similar to the search query are selected. An information processing method comprising:

6. 1. A computer-implemented information processing method, comprising: a search step of searching for a search target based on a search query specified by a user; a selection step of selecting, from among a plurality of candidate images associated with the search target included in the search results by the search step, a candidate image having the highest relevance to the search query from each of the search targets; a providing step of providing the candidate image selected by the selecting step as a thumbnail image to be searched; Including, The selection step includes: In a learning stage, the search query and the candidate images are projected into a distributed representation space using a model that has been trained so that the candidate images, which have been annotated in advance as images corresponding to the search query, and the search query have similar feature vectors in the distributed representation space, and the candidate images similar to the search query are selected. An information processing method comprising:

7. 1. A computer-implemented information processing method, comprising: a search step of searching for a search target based on a search query specified by a user; a selection step of selecting, from among a plurality of candidate images associated with the search target included in the search results by the search step, a candidate image having the highest relevance to the search query from each of the search targets; a providing step of providing the candidate image selected by the selecting step as a thumbnail image to be searched; Including, The selection step includes: Selecting a coordinated image using a plurality of clothing items as the candidate image, The providing step includes: Providing the coordinated image as the thumbnail image. An information processing method comprising:

8. 1. A computer-implemented information processing method, comprising: a search step of searching for a search target based on a search query specified by a user; a selection step of selecting, from among a plurality of candidate images associated with the search target included in the search results by the search step, a candidate image having the highest relevance to the search query from each of the search targets; a generating step of generating a composite image from an image of the search target that does not satisfy the selection condition of the candidate image by the selection step; a providing step of providing the candidate image selected by the selecting step or the composite image generated by the generating step as a thumbnail image to be searched; 13. An information processing method comprising:

9. a search step for searching for a search target based on a search query specified by a user; a selection step of selecting, from among a plurality of candidate images associated with the search targets included in the search results by the search step, a candidate image having the highest relevance to the search query from each of the search targets; a providing step of providing the candidate image selected by the selecting step as a thumbnail image to be searched; Run the following on your computer: The selection step comprises: In a learning stage, the search query and the candidate images are projected into the distributed representation space using a model trained so that the search query specified by the user and an image selected by the user in a search result based on the search query have similar feature vectors in the distributed representation space, and the candidate images similar to the search query are selected. An information processing program characterized by:

10. a search step for searching for a search target based on a search query specified by a user; a selection step of selecting, from among a plurality of candidate images associated with the search targets included in the search results by the search step, a candidate image having the highest relevance to the search query from each of the search targets; a providing step of providing the candidate image selected by the selecting step as a thumbnail image to be searched; Run the following on your computer: The selection step comprises: In a learning stage, the search query and the candidate images are projected into a distributed representation space using a model that has been trained so that the candidate images, which have been annotated in advance as images corresponding to the search query, and the search query have similar feature vectors in the distributed representation space, and the candidate images similar to the search query are selected. An information processing program characterized by:

11. a search step for searching for a search target based on a search query specified by a user; a selection step of selecting, from among a plurality of candidate images associated with the search targets included in the search results by the search step, a candidate image having the highest relevance to the search query from each of the search targets; a providing step of providing the candidate image selected by the selecting step as a thumbnail image to be searched; Run the following on your computer: The selection step comprises: Selecting a coordinated image using a plurality of clothing items as the candidate image, The providing step includes: Providing the coordinated image as the thumbnail image. An information processing program characterized by:

12. a search step for searching for a search target based on a search query specified by a user; a selection step of selecting, from among a plurality of candidate images associated with the search targets included in the search results by the search step, a candidate image having the highest relevance to the search query from each of the search targets; a generating step of generating a composite image from an image of the search target that does not satisfy the selection condition of the candidate image by the selection step; a providing step of providing the candidate image selected by the selecting step or the composite image generated by the generating step as a thumbnail image of the search target; An information processing program characterized by causing a computer to execute the above.

Images