Information processing device, information processing method, and information processing program

Abstract

To provide thumbnail images reflecting intent of a search by a user.SOLUTION: An information processing device of the present invention comprises a search unit, a selection unit, and a provision unit. The search unit searches for a product based on a search query specified by a user. The selection unit selects a candidate image most relevant to the search query among multiple candidate images associated with search targets included in a result of the search by the search unit from each search target. The provision unit provides the candidate images selected by the selection unit in the form of thumbnail images of the search targets.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, techniques related to image search using a VSE (Visual-Semantic Embedding) model have been disclosed. For example, in image search using the VSE model, a specified search keyword and a search target image are converted into feature vectors, and an image corresponding to the similarity between the two feature vectors is output as a search result (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180941 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional techniques merely search for images that have similar feature vectors to a search keyword, and do not take into consideration providing thumbnail images that reflect the user's search intent.

[0005] The present invention has been made in view of the above, and aims to provide an information processing device, an information processing method, and an information processing program that can provide thumbnail images that reflect the user's search intent. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the information processing device of the present invention comprises a search unit that searches for a search target based on a search query specified by a user, a selection unit that selects from each of the search targets, among a plurality of candidate images linked to the search targets included in the search results by the search unit, the candidate image that has the highest relationship with the search query, and a provision unit that provides the candidate image selected by the selection unit as a thumbnail image of the search target. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide thumbnail images that reflect the user's search intent. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a provision system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of information processing according to the embodiment. [Figure 3] FIG. 3 is an explanatory diagram of a distributed representation space according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an outline of the thumbnail image generation process according to the embodiment. [Figure 5] FIG. 5 is a block diagram illustrating an example of the configuration of an information processing device according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of information stored in a user information storage unit according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of information stored in a product information storage unit according to the embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a method for collecting learning data according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of learning data according to the embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a composite image according to the embodiment. [Figure 11]FIG. 11 is a flowchart illustrating an example of a processing procedure for information processing according to the embodiment. [Figure 12] FIG. 12 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an information processing device, an information processing method, and an information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to these embodiments. In addition, the following description will be given assuming that the search target is a product sold in an online shopping mall.

[0010] [Embodiment] [1.1 System] First, a description will be given of the provision system shown in Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the provision system according to the embodiment. As shown in Fig. 1, the provision system includes an information processing device 1 and a user terminal 10. The information processing device 1 and the user terminal 10 are connected to each other via a predetermined communication network (network N) so as to be able to communicate with each other via wired or wireless communication.

[0011] The information processing device 1 provides an e-commerce service that offers (searches, sells, etc.) products including clothing (also called fashion items, including clothes, underwear, footwear (also called shoes), headwear (such as caps and hats), bags, and ornaments (also called accessories)), cosmetics (also called fashion items), etc. The information processing device 1 also provides a coordination service that accepts posts from users of content (images, videos, articles, etc.) showing outfits using multiple clothing items, content showing makeup using one or more cosmetics, and provides (searches, distributes, etc.) the posted content to other users. For example, the information processing device 1 can be realized by a server device, a cloud system, etc.

[0012] Furthermore, as will be described later, the information processing device 1 provides various services to users by using a distributed representation space that has been trained in advance. Images and keywords (including tags, etc.) are projected into the distributed representation space according to the embodiment, and the closeness between images, images and keywords, and keywords is projected. The closer two items are semantically, the closer they are trained to be in the distributed representation space (and vice versa). That is, the similarity between images, images and keywords, and keywords can be measured as a distance in the distributed representation space. The closeness in the distributed representation space according to the embodiment includes, for example, closeness based on Euclidean distance and closeness based on cosine distance.

[0013] The information processing device 1 may also have a function as a web server that provides a website related to the service. The information processing device 1 may also be a device that distributes information to be displayed in applications related to various services installed in the user terminal 10 to the user terminal 10. The information processing device 1 may also be a device that distributes the application data itself.

[0014] Furthermore, the information processing device 1 may function as a distribution device that distributes control information to the user terminal 10. Here, the control information is written in, for example, a script language such as JavaScript (registered trademark) or a style sheet language such as CSS (Cascading Style Sheets). Note that the application itself distributed from the information processing device 1 may also be considered as control information.

[0015] The user terminal 10 is an information processing device used by a user. The user terminal 10 is realized by, for example, a smartphone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), etc. The user terminal 10 displays information distributed by the information processing device 1 or a server device that provides a predetermined service, using a web browser or an application.

[0016] [1.2 Information Processing] Next, information processing according to the embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of information processing according to the embodiment. As shown in Fig. 2, the information processing device 1 functions as a web server that provides information about fashion items, and first accepts a search query specified by a user from a user terminal 10 (step S1).

[0017] The information processing device 1 searches for products based on the search query (step S2). For example, products tagged with keywords corresponding to the search query are output as search results.

[0018] Next, the information processing device 1 selects a thumbnail image for each product included in the search result (step S3). More specifically, the information processing device 1 selects, from each product included in the search result, a candidate image that has the highest relationship with the search query among multiple candidate images associated with each product.

[0019] For example, the information processing device 1 selects thumbnail images of each product based on the similarity between the search query (keyword) and each candidate image in the distributed representation space. More specifically, the search query indicates the user's search intent, and the thumbnail image clicked (viewed) by the user who specified the search query (it may be a thumbnail image that not only clicked but also led to a purchase) can be considered to be an image that better reflects the user's search intent. For example, the information processing device 1 can project each search query, candidate image, etc. into the distributed representation space using technologies related to distributed representation spaces, such as VSE (Visual-Semantic Embedding) and CLIP (Contrastive Language-Image Pre-training).

[0020] Therefore, for example, the information processing device 1 can select thumbnail images that are highly related to the search query, i.e., that reflect the user's search intent, by linking a search query specified by the user with a thumbnail image clicked by the user and using the linked data as learning data. Furthermore, for example, the information processing device 1 may link a search query specified by the user with a thumbnail image not clicked by the user and use the linked data as negative example learning data. In this embodiment, in addition to the search query and candidate images, a thumbnail image is selected from the candidate images based on the attributes of the user who specified the search query.

[0021] Here, the distributed representation space according to the embodiment will be described with reference to Fig. 3. Fig. 3 is an explanatory diagram of the distributed representation space according to the embodiment. Note that the process of selecting a thumbnail image from a plurality of candidate images linked to one product will be described here.

[0022] 3, first, the information processing device 1 embeds a plurality of candidate images linked to a product into a distributed representation space (step S11), and also embeds a search query and user attributes into the distributed representation space (steps S12 and S13). The user attributes include, for example, information about the user's age, gender, body type, foot type, skin color, residence, and the user's click log, search log (search history), purchase log (purchase history), and browsing log (browsing history). The user attributes may also include, for example, information about the age, gender, body type, foot type, skin color, residence, click log, search log, purchase log, and browsing log of users who have a predetermined relationship (including a parent-child relationship) with the user.

[0023] In the example shown in Figure 3, the feature vectors of each candidate image are indicated as "image vectors Vi1 to Vi5," the feature vector of the search query is indicated as "query vector Vq," and the feature vector of the user attribute is indicated as "user vector Vu." Note that, hereinafter, when there is no need to distinguish between the image vectors Vi1 to Vi5, they will be referred to as "image vector Vi."

[0024] For example, the information processing device 1 generates a ranking of the image vectors Vi1 to Vi5 according to their positional relationship with the query vector Vq and the user vector Vu. For example, the information processing device 1 generates a ranking in descending order of the image vector Vi, starting from the image vector Vi1 to Vi5 with the smallest area of ​​the space (triangle) connecting the query vector Vq and the user vector Vu. Note that the information processing device 1 may generate a ranking of the image vectors Vi1 to Vi5 in descending order of similarity to either the query vector Vq or the user vector Vu.

[0025] Then, the information processing device 1 selects the candidate image ranked first as the thumbnail image. That is, the information processing device 1 selects, from among the image vectors Vi, the candidate images corresponding to the image vectors Vi that are respectively similar to the query vector Vq and the user vector Vu as the thumbnail image.

[0026] That is, the information processing device 1 selects a candidate image to be used as a thumbnail image from among the candidate images associated with the product in accordance with the search query and the user attributes. More specifically, from the image vectors Vi corresponding to the candidate images, the information processing device 1 selects an image vector Vi that is similar to a query vector Vq corresponding to the search query and a user vector Vu corresponding to the user attributes, and selects the image vector Vi as the thumbnail image.

[0027] This allows the information processing device 1 to select a thumbnail image that reflects the user's search intent from multiple candidate images linked to the product. In the example shown in Fig. 3, the area of ​​the space connecting the image vector Vi4 with the query vector Vq and the user vector Vu is the smallest, indicating that the candidate image corresponding to the image vector Vi4 will be selected as the thumbnail image.

[0028] When the information processing device 1 selects a thumbnail image from each candidate image for each product included in the search results in step S2, it provides the selected thumbnail image to the user terminal 10 as shown in FIG. 2 (step S4).

[0029] Furthermore, the information processing device 1 can also generate thumbnail images for products for which there is no image vector Vi that satisfies the selection conditions for a thumbnail image from among the image vectors Vi in the distributed representation space. That is, for products for which there is no appropriate thumbnail image, the information processing device 1 generates a new thumbnail image based on the search query. Note that the information processing device 1 may register multiple candidate images generated in advance, and if there is no thumbnail image that satisfies the selection conditions, select and provide a thumbnail image from the candidate images registered in advance.

[0030] 4 is a schematic diagram of a thumbnail image generation process according to an embodiment. For example, the information processing device 1 generates a composite image Ic, which is a thumbnail image obtained by combining a product image It for a product that does not have a suitable thumbnail image with a compositing image Is corresponding to the search query.

[0031] For example, the composite image Is includes candidate images associated with other products and posted images (e.g., coordinated images) posted by each user to a coordination service, etc. The posted images include photos tagged with fashion items worn by each user.

[0032] For example, the information processing device 1 extracts each compositing image Is based on the search query from candidate images associated with other products or posted images posted by each user. For example, the information processing device 1 extracts each compositing image Is from each image according to the similarity between the image vector Vi of each image in the distributed representation space and the query vector Vq of the search query.

[0033] Next, the information processing device 1 uses various image generation AIs to synthesize a composite image Ic by combining the product image It and the composite image Is.

[0034] In this case, for example, the information processing device 1 repeatedly generates the composite image Ic until the area of ​​the space connecting the image vector Vi of the composite image Ic, the query vector Vq of the search query, and the user vector Vu of the user attribute becomes less than a threshold, thereby generating a composite image Ic that is adapted to the search query specified by the user.

[0035] In this way, the information processing device 1 automatically generates thumbnail images for products that do not have thumbnail images corresponding to the search query, allowing the information processing device 1 to provide appropriate thumbnail images corresponding to the search query entered by the user.

[0036] [2. Information Processing Device] Next, a configuration example of the information processing device 1 according to the embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration example of the information processing device 1 according to the embodiment. As shown in Fig. 5, the information processing device 1 includes a communication unit 2, a storage unit 3, and a control unit 4. Note that the information processing device 1 may also include an input unit (e.g., a keyboard or a mouse) that accepts various operations from an administrator who uses the information processing device 1, and a display unit (e.g., a liquid crystal display) that displays various information.

[0037] The communication unit 2 is realized by, for example, a network interface card (NIC), etc. The communication unit 2 is connected to a communication network such as 4G (4th Generation) or 5G (5th Generation) by wire or wirelessly, and transmits and receives information to and from each of the user terminals 10, etc. via the communication network.

[0038] The storage unit 3 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 3 has a user information storage unit 31, a product information storage unit 32, and a model storage unit 33.

[0039] The user information storage unit 31 stores information about each user. Fig. 6 is a diagram showing an example of information stored in the user information storage unit 31 according to the embodiment. As shown in Fig. 6, the user information storage unit 31 stores information items such as "user ID" and "user information" in association with each other.

[0040] "User ID" indicates identification information for identifying a user. "User information" indicates user information. In the example shown in Figure 6, conceptual information such as "User information #1" and "User information #2" is stored in "User information," but in reality, information on user attributes such as age and gender, and identification information for products (items) owned by the user are stored.

[0041] The product information storage unit 32 stores product information. The product information storage unit 32 stores information about products handled in the online shopping mall provided by the information processing device 1. Fig. 7 is a diagram showing an example of information stored in the product information storage unit 32 according to the embodiment.

[0042] As shown in FIG. 7, the product information storage unit 32 stores information items such as "product ID" and "image information" in association with each other. "Product ID" indicates identification information for identifying a product. "Image information" is image information of a product identified by the corresponding product ID. In this embodiment, multiple candidate images are stored as image information for each product.

[0043] The model storage unit 33 stores a model. For example, the model is a learning model that projects each image, a search query (keyword), and user information into the same distributed representation space. For example, the learning model is a model that has been learned by the learning unit 42, which will be described later, and the learning model may be, for example, a VSE model.

[0044] Next, a description will be given of the control unit 4. The control unit 4 is a controller, and is realized, for example, by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) using RAM as a work area to execute various programs (corresponding to examples of information processing programs) stored in a storage device inside the information processing device 1. The control unit 4 is also, for example, a controller, and is realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0045] As shown in Fig. 5, the control unit 4 includes an acquisition unit 41, a learning unit 42, a search unit 43, a selection unit 44, a generation unit 45, and a provision unit 46, and realizes or executes the functions and actions of the information processing described below. Note that the internal configuration of the control unit 4 is not limited to the configuration shown in Fig. 5, and may be other configurations as long as they perform the information processing described below. Furthermore, the connection relationship between the processing units included in the control unit 4 is not limited to the connection relationship shown in Fig. 5, and may be other connection relationships.

[0046] The acquisition unit 41 acquires various types of information. For example, the acquisition unit 41 acquires user information from the user terminal 10 and registers it in the user information storage unit 31. The acquisition unit 41 also acquires product information from terminals of shop owners (not shown) in the online shopping mall and registers it in the product information storage unit 32.

[0047] The acquisition unit 41 also receives posts from users, such as content (images, videos, articles, etc.) showing outfits using multiple clothing items, and content showing makeup using one or more cosmetics. For example, such content is provided (searched, distributed, etc.) to other users through a platform such as a social networking service. Note that such a platform may be a platform provided by the information processing device 1.

[0048] The learning unit 42 learns the model (learning model) stored in the model storage unit 33. First, a method for collecting learning data will be described. Fig. 8 is a diagram showing an example of a method for collecting learning data according to an embodiment.

[0049] 8, for example, the information processing device 1 provides an annotation app to the user terminal 10 and acquires learning data for a learning model through the annotation app. For example, as shown in FIG. 8, the annotation app displays a search query to be learned and multiple images.

[0050] Then, the user selects an image that matches the search query through the annotation app, and the information processing device 1 acquires learning data that links the search query with the image selected by the user. In the example shown in FIG. 8, the search query is "golf," and the user selects an image that is associated with "golf." Alternatively, the search query may be "golf knit sweater," and the user may select an image that is associated with "golf knit sweater."

[0051] The information processing device 1 can collect learning data of images corresponding to search queries by having the user select an image corresponding to each search query through the annotation app.

[0052] Note that the annotation app is not limited to the example shown in Fig. 8, and may simultaneously display two images, change the layout of the images depending on the user, display the images in a circular layout, or display the images while constantly moving the layout. This eliminates the bias that makes images more likely to be selected depending on their layout.

[0053] In addition to the above example, the information processing device 1 may collect learning data from, for example, log information of each user in an online shopping mall. Fig. 9 is a diagram showing an example of learning data according to the embodiment.

[0054] As shown in FIG. 9, the learning data includes items such as "user," "search query," "clicked image," and "unclicked image." "User" indicates the attributes of the user. "Search query" indicates the search query specified by the corresponding user. The learning data may also include information such as the time it took the user to select an image, images displayed at the same time, and the position (display position) of the image selected by the user.

[0055] "Clicked image" refers to an image of a product that was included in the search results for the corresponding search query by the corresponding user and that was clicked on (e.g., a thumbnail image), and "Unclicked image" refers to an image of a product that was included in the search results for the corresponding search query by the corresponding user and that was not clicked on.

[0056] That is, in this case, it is possible to collect learning data based on the logs of each user in the online shopping mall. After collecting this learning data, the learning unit 42 trains the learning model based on the collected learning data. Specifically, the learning unit 42 trains the learning model so that, in the distributed representation space, the feature vector of the search query and the feature vector of the image linked to the search query become similar feature vectors in the distributed representation space.

[0057] More specifically, the learning unit 42 trains the learning model so that similar images have similar feature vectors, similar search queries (keywords) have similar feature vectors, and the search queries that serve as learning data and the images linked to those search queries have similar feature vectors.

[0058] The learning unit 42 also learns a learning model that can project the user attributes of each user into the distributed representation space. For example, the user attributes include so-called demographic attributes such as the user's age, gender, and residence, as well as the user's log in the online shopping mall, which includes search history, browsing history, purchase history, and so on.

[0059] The learning unit 42 also generates a pseudo-ranking of thumbnail images that express search intent using a method such as Serial Rank, and evaluates the learning model. The learning unit 42 also trains the learning model so that similar user attributes become similar feature vectors in the distributed representation space. Note that the learning data is not limited to the above example, and may include logs of each user in the coordination service. The user's log in the coordination service includes browsing history, search history, posting history, product purchase history through the coordination service, etc.

[0060] The search unit 43 searches for a search target (product) based on a search query specified by a user. Specifically, the search unit 43 acquires a user ID and a search query from the user terminal 10, and searches the product information storage unit 32 for products that match the search query specified by the user. The search unit 43 then passes the user ID and product information included in the search result to the selection unit 44.

[0061] The selection unit 44 selects, for each product, a candidate image that has the highest relationship with the search query from among multiple candidate images linked to the product included in the search results by the search unit 43. Specifically, the selection unit 44 uses a learning model to project the candidate image, user attribute, and search query for each product into a distributed representation space, and calculates the area of ​​the space connecting each image vector Vi, user vector Vu, and query vector Vq in the distributed representation space (see FIG. 2).

[0062] For example, the selection unit 44 selects as the thumbnail image the candidate image corresponding to the image vector Vi that has the smallest area of ​​the space. At this time, for example, if the area of ​​the space connecting the image vector Vi, the user vector Vu, and the query vector Vq is equal to or greater than a threshold, that is, if there is no thumbnail image that corresponds to the search query, the selection unit 44 passes information about the product to the generation unit 45 and instructs the generation of a thumbnail image.

[0063] The generation unit 45 generates thumbnail images of products included in the search results by the search unit 43 based on the relationship between the search query and the image. The generation unit 45 generates thumbnail images for products for which the selection unit 44 has instructed the generation of thumbnail images, i.e., for products for which multiple candidate images linked to the products do not satisfy the selection conditions for thumbnail images.

[0064] For example, the generation unit 45 generates, as thumbnail images, coordinated images including products included in the search results by the search unit 43. The coordinated images are images using a plurality of clothing items. For example, the generation unit 45 generates coordinated images by combining a plurality of candidate images linked to products corresponding to each category, such as tops, bottoms, and shoes.

[0065] First, the generation unit 45 extracts images to be synthesized based on the search query. For example, the generation unit 45 uses a learning model to project product images of products sold in the online shopping mall and images posted by users into a distributed representation space, and extracts images in the distributed representation space that are similar to the query vector Vq of the search query as images to be synthesized.

[0066] For example, based on the product image It for which a thumbnail image is to be generated and the search query, the generation unit 45 extracts, as an image to be synthesized, an image that is similar to the search query in the distributed representation space and that matches the product. For example, the generation unit 45 extracts a plurality of patterns of images to be synthesized that differ depending on the bone structure, style, season, etc. of the model appearing in the posted image. Note that the generation unit 45 may extract a plurality of patterns of images to be synthesized that differ depending on the compatibility between the product (e.g., a top) and the model appearing in the posted image, the compatibility between the product (e.g., a top) and other products (e.g., a skirt) worn by the model appearing in the posted image, and the compatibility with the background appearing in the posted image.

[0067] When extracting a composite image from posted images posted by a user, the generation unit 45 may, for example, exclude posted images that include products of a brand that is not sold in the online shopping mall from the candidates for the composite image.

[0068] For example, posted images are tagged with the products (items) that each user is wearing, and based on these tags, the generation unit 45 can identify posted images that include products that are not available in the online shopping mall.

[0069] The generation unit 45 may generate a composite image by overwriting products that are not sold in the online shopping mall when generating a composite image using the images to be composited, without excluding all posted images from the target images for composite. In other words, by not displaying products of brands that are not sold in the online shopping mall in thumbnail images, it is possible to prevent copyright infringement, for example.

[0070] The generation unit 45 may prepare images corresponding to each search query in advance and extract (select) an image to be synthesized from among these images.

[0071] After extracting the composite image, the generation unit 45 uses an image generation AI to generate a composite image from the product image It of the target product and the composite image. For example, the generation unit 45 generates a collage image by arranging each product image (each item image) on the composite image, and then generates a natural thumbnail image from the collage image. In this case, the generation unit 45 may extract, for example, cutout images of the items used, the background, etc. from each composite image, and then generate the thumbnail image.

[0072] Furthermore, the generation unit 45 may change the instruction to the image generation AI depending on the user or the search query. For example, if the user is a male, the generation unit 45 instructs the image generation AI to generate an image in which a male is the model.

[0073] Also, suppose the user who originated the search is a man and the search query is "women, dress, fashionable," and the man is searching for a gift for a woman. In this case, the generation unit 45 instructs the image generation AI to generate an image of a woman as a model based on the search query "women." Note that whether the user or the search query is to be given priority may be determined in advance, for example, on a rule basis.

[0074] Furthermore, when generating a composite image, the generating unit 45 may include, for example, products previously purchased by the user in the composite image. Note that products previously purchased by the user can be identified from, for example, the user's purchase history in an online shopping mall, but may also be identified from posted images (coordinated images) posted by the user.

[0075] FIG. 10 is a diagram illustrating an example of a composite image according to an embodiment. For example, as illustrated in FIG. 10, the generation unit 45 extracts product images of products previously purchased by a user from the user information storage unit 31 and generates a composite image including the extracted product images. In this case, the generation unit 45 may generate a composite image including product images of products selected from the user's log. In this case, the generation unit 45 may extract product images related to products registered in the user's favorites, products previously viewed, or products purchased by users similar to the target user. In the example illustrated in FIG. 10, the products previously purchased by the user are a bag and a necklace. The generation unit 45 generates a composite image Ic by further using the product image Ib corresponding to the bag and the product image In corresponding to the necklace as images for synthesis.

[0076] In other words, in this case, the composite image Ic is generated by coordinating a product image (a dress in Figure 10) that corresponds to the search query specified by the user, in other words, a product image that matches the user's search intent, with a bag or necklace that the user has previously purchased.

[0077] That is, the composite image Ic can present the user with a sample of what the one-piece dress currently being sold will look like when worn with a bag or necklace that the user currently owns.

[0078] In this way, the generation unit 45 can generate a composite image Ic that is individually optimized for the user, which is expected to increase the user's purchasing motivation. Note that, for example, the generation unit 45 may generate the composite image Ic by combining products specified by the user. In this case, the user may select a product from products that they have purchased in the past, or, for example, may select an image of the product that they have photographed.

[0079] The providing unit 46 provides the user with the thumbnail image selected by the selecting unit 44. At this time, for a product for which a thumbnail image was not selected by the selecting unit 44, the providing unit 46 provides the composite image Ic generated by the generating unit 45 as the thumbnail image.

[0080] This allows the providing unit 46 to provide thumbnail images that reflect the user's search intent.

[0081] [3. Processing flow] Next, a processing procedure executed by the information processing device 1 according to the embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of the processing procedure of information processing according to the embodiment. The processing shown below is repeatedly executed by the information processing device 1 at a predetermined cycle each time a search query is acquired.

[0082] 11, the information processing device 1 receives a search query from a user (step S101). Subsequently, the information processing device 1 searches for products that match the search query (step S102).

[0083] Next, the information processing device 1 embeds multiple candidate images linked to products included in the search results, the search query, and user attributes into a distributed representation space (step S103). Next, the information processing device 1 selects thumbnail images from the candidate images based on each feature vector in the distributed representation space (step S104).

[0084] Next, the information processing device 1 determines whether there are any products whose candidate images do not satisfy the selection conditions (step S105), and if there are any products that do not satisfy the selection conditions (step S105: Yes), it generates thumbnail images of the products (step S106).

[0085] If thumbnail images can be selected from all products (step S105; No), the information processing device 1 proceeds to the process of step S107. Then, the information processing device 1 provides the thumbnail images to the user (step S107) and ends the process.

[0086] [4. Modifications] In the above-described embodiment, the information processing device 1 provides thumbnail images related to products, but the present invention is not limited thereto. For example, the present invention may be applied when providing thumbnail images for various web services such as social networking sites and video sites. Furthermore, the information processing device 1 may provide the present invention when providing thumbnail images related to outfits in a coordination service. For example, the selection unit may select, from among multiple candidate images (posted images) associated with outfits included in the search results by the search unit, a candidate image that has the highest relationship with the search query from each outfit and provide it as a thumbnail image.

[0087] In the above-described embodiment, the providing unit 46 provides the composite image Ic generated by the generating unit 45 as a thumbnail image for a product for which the selecting unit 44 did not select a thumbnail image. However, this is not limited to this. Even if there is no candidate image that reflects the user's search intent among the multiple candidate images associated with the product, if a posted image showing an outfit using the product has been posted to a coordination site and among the posted images reflects the user's search intent, the posted image may be provided as a thumbnail image. Specifically, the information processing device 1 embeds multiple posted images showing outfits using the products included in the search results, the search query, and user attributes into a distributed representation space. Next, the information processing device 1 selects a thumbnail image from the posted images based on each feature vector in the distributed representation space.

[0088] In addition, instead of embedding all posted images showing coordinations using products included in the search results into the distributed representation space, the information processing device 1 may search the coordination site in advance using the same search query and embed the posted images included in the search results into the distributed representation space.

[0089] That is, the search target of the search unit 43 may be expanded to include, for example, coordinated images posted on a coordinate service in addition to products handled in the online shopping mall. By expanding the search target in this way, it is possible to provide search results that reflect the user's search intent.

[0090] [5. Effects] The information processing device 1 according to the embodiment includes a search unit 43 that searches for products based on a search query specified by a user, a selection unit 44 that selects, from among multiple candidate images linked to products included in the search results by the search unit 43, a candidate image that has the highest relationship with the search query for each product, and a provision unit that provides the candidate image selected by the selection unit 44 as a thumbnail image of the product.

[0091] The selection unit 44 also projects the search query and candidate images into the distributed representation space, and selects candidate images in the distributed representation space that are similar to the search query. The selection unit 44 also selects candidate images based on user information of the user who specified the search query, and selects the candidate image that has the smallest area of ​​the space connecting the user information, the search query, and the candidate images in the distributed representation space onto which the user information is projected.

[0092] In addition, during the learning stage, the selection unit 44 selects candidate images using a model that has been trained so that candidate images that have been pre-annotated as images corresponding to the search query and the search query become similar vectors in the distributed representation space.

[0093] In addition, during the learning stage, the selection unit 44 selects candidate images using a model that has been trained so that the search query specified by the user and the image selected by the user when using the search query become similar vectors in the distributed representation space.

[0094] In addition, the information processing device 1 is provided with a generation unit 45 that generates a composite image for a product that does not satisfy the selection conditions of the candidate image by the selection unit 44 using an image to be synthesized that satisfies the search conditions, and the provision unit 46 provides the composite image generated by the generation unit 45 as a thumbnail image.

[0095] By performing any one or a combination of the above-described processes, the information processing device according to the present application can provide thumbnail images that reflect the user's search intent.

[0096] [6. Hardware Configuration] The information processing device 1 according to the embodiment described above is realized by, for example, a computer 1000 configured as shown in Fig. 12. Fig. 12 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device according to the embodiment. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0097] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 starts up, programs that depend on the hardware of the computer 1000, and the like.

[0098] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, etc. The communication interface 1500 receives data from other devices via a network (communication network) N and sends the data to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the network N.

[0099] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse (in FIG. 12, the output devices and input devices are collectively referred to as "input / output devices") via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input / output interface 1600.

[0100] Media interface 1700 reads a program or data stored in recording medium 1800 and provides it to CPU 1100 via RAM 1200. CPU 1100 loads the program or data from recording medium 1800 onto RAM 1200 via media interface 1700 and executes the loaded program. Recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0101] For example, when the computer 1000 functions as the information processing device according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 4. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via the network N.

[0102] [7. Other] Although the embodiments of the present application have been described above, the present invention is not limited to the contents of these embodiments. Furthermore, the above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the scope of so-called equivalents. Furthermore, the above-described components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments.

[0103] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0104] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0105] For example, the above-mentioned information processing device may be realized using multiple server computers, and depending on the function, the configuration can be flexibly changed, such as by calling an external platform using an API (Application Programming Interface) or network computing.

[0106] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0107] Furthermore, the above-mentioned "section, module, unit" can be read as "means" or "circuit," etc. For example, an acquisition unit can be read as an acquisition means or an acquisition circuit. [Explanation of symbols]

[0108] 1. Information processing equipment 2. Communications Department 3 Storage section 4. Control section 10 User terminal 31 User information storage unit 32 Product information storage section 33 Model memory section 41 Acquisition Department 42 Learning Department 43 Search Section 44 Selection section 45 Generation part 46 Providing Department

Claims

1. a search unit that searches for search targets based on a search query specified by a user; a selection unit that selects, from among a plurality of candidate images associated with the search target included in the search results by the search unit, the candidate image that has the highest relationship with the search query and the user information of the user from each of the search targets; a providing unit that provides the candidate image selected by the selecting unit as a thumbnail image to be searched; An information processing device comprising:

2. 1. A computer-implemented information processing method, comprising: a search step of searching for a search target based on a search query specified by a user; a selection step of selecting, from each of the search targets included in the search results by the search step, a candidate image that has the highest relationship between the search query and the user information of the user, from among a plurality of candidate images linked to the search targets; a providing step of providing the candidate image selected by the selecting step as a thumbnail image to be searched; An information processing method comprising:

3. a search procedure for searching for a search target based on a search query specified by a user; a selection step of selecting, from each of the search targets, a candidate image that has the highest relationship between the search query and the user information of the user from among a plurality of candidate images associated with the search targets included in the search results of the search step; a providing step of providing the candidate image selected by the selecting step as a thumbnail image to be searched; An information processing program characterized by causing a computer to execute the above.

Images