Search system, search method, and information processing device

Abstract

This search system acquires a query for searching for a desired item, generates an image by using the query and a trained machine learning model, and searches for the item on the basis of the image.

Description

Search system, search method, and information processing device

[0001] The present invention relates to a search system, a search method, and an information processing device, and more particularly to a technique for searching for items corresponding to an input query.

[0002] In recent years, electronic commerce (EC) has become widespread, and as a result, many EC sites have been established on the web. EC sites perform searches based on search queries (hereinafter simply referred to as queries) entered by users to present users with items they desire. For example, Patent Literature 1 discloses a search system that performs a product search by searching for product data that includes a character string that matches text entered by the user (an example of a query).

[0003] Japanese Patent Application Laid-Open No. 2001-265781

[0004] In conventional search systems, depending on the query entered by the user, appropriate search results may not be obtained. For example, in a system such as that disclosed in Patent Literature 1 that searches for product data containing a character string that matches a text entered as a query, if a long piece of text is entered as a query, it becomes difficult to match the character string in the product data with the query. Therefore, in such cases, appropriate search results may not be obtained.

[0005] In view of the above-described problems, the present disclosure provides a technique for flexibly accepting a query input from a user and obtaining appropriate search results in accordance with the query.

[0006] A search system according to one aspect of the present disclosure includes a query acquisition unit that acquires a query for searching for a desired item, an image generation unit that generates an image using the query and a trained machine learning model, and a search unit that searches for an item based on the image.

[0007] A search method according to one aspect of the present disclosure includes a query acquisition process for acquiring a query for searching for a desired item, an image generation process for generating an image using the query and a trained machine learning model, and a search process for searching for the item based on the image.

[0008] An information processing device according to one aspect of the present disclosure includes a query acquisition unit that acquires a query for searching for a desired item, an image generation unit that generates a plurality of images using the query and a trained diffusion model, and a feature vector acquisition unit that acquires a feature vector for searching for the item based on the plurality of images.

[0009] According to the present invention, it is possible to flexibly accept a query input from a user and obtain appropriate search results in response to the query.

[0010] FIG. 1 shows an example of the configuration of an information processing system. FIG. 2 shows an example of the functional configuration of an information processing device. FIG. 3 shows an example of the functional configuration of a search device. FIG. 4 is a diagram for explaining an image generation model in the learning stage. FIG. 5 shows an example of the hardware configuration of an information processing device and a search device according to an embodiment. FIG. 6A is a diagram for explaining the procedure for generating a first type of image. FIG. 6B is a diagram for explaining the procedure for generating a third type of image. FIG. 7 is a sequence diagram showing the flow of processing executed in an information processing system. FIG. 8A shows an example of an image generated by the information processing device. FIG. 8B shows an example of a search result page generated by the search device.

[0011] Hereinafter, with reference to the accompanying drawings, an embodiment for carrying out the present invention will be described in detail. Among the components disclosed below, those having the same function will be given the same reference numerals, and their description will be omitted. Note that the embodiment disclosed below is an example of a means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions. The present invention is not limited to the following embodiment. Furthermore, not all of the combinations of features described in the present embodiment are necessarily essential to the solution of the present invention.

[0012] 1 shows an example of the configuration of an information processing system 1 according to this embodiment. As an example, the information processing system 1 includes a user device 10 and a search system 20. The search system 20 includes an information processing device 21 and a search device 22. In the information processing system 1, the user device 10, in response to a user operation, transmits a search query (hereinafter simply referred to as a query) for a desired item to the search system 20, and the search system 20, upon receiving the query, operates to transmit search results that match the query to the user device 10.

[0013] The user device 10 is a device such as a smartphone or tablet, and is configured to be able to communicate with the search system 20 (i.e., the information processing device 21 and the search device 22) via a public network such as LTE (Long Term Evolution) or a wireless communication network such as a wireless LAN (Local Area Network). The user device 10 has a display unit (display surface) such as a liquid crystal display, and a user of the user device 10 can perform various operations using a GUI (Graphical User Interface) equipped on the liquid crystal display. These operations include various operations on content such as images displayed on the screen, such as tapping, sliding, and scrolling using a finger or a stylus. Note that the user device 10 is not limited to the device shown in FIG. 1 , but may also be a device such as a desktop personal computer (PC) or a notebook PC. In this case, user operations may be performed using input devices such as a mouse and a keyboard. The user device 10 may also include a separate display unit.

[0014] In the search system 20, the search device 22 is a server device that provides an electronic commerce site (EC site). A user of the user device 10 can access the EC site provided by the search device 22, input a query to search for a desired item in a search window (input field) on the site, and confirm the query. Accordingly, the user device 10 transmits the query input by the user to the search system 20. The information processing device 21 receives the query transmitted from the user device 10 and generates one or more images based on the query. Alternatively, the information processing device 21 may generate one or more images from information other than the query. Furthermore, the information processing device 21 acquires feature vectors (hereinafter also referred to as image features) representing the characteristics of the one or more images and outputs the image features to the search device 22. The search device 22 acquires the image features acquired by the information processing device 21 and generates a web page (search result page) displaying search results based on the image features. The search device 22 transmits (distributes) the generated search result page to the user device 10. Although the information processing device 21 and the search device 22 are configured as separate devices in FIG. 1, they may be configured as a single information processing device.

[0015] In this embodiment, the query input by the user of the user device 10 may be text (e.g., text information describing an item), tags indicating attributes (e.g., the color, brand, category, genre, size, or gender of the target (female, male, unisex, etc.) of the item) (hereinafter also referred to as attribute tags), or images (e.g., images representing the item). Alternatively, the query may be a combination of two or more of text, attribute tags, and images. Note that the image used as a query may be an incomplete image, such as an image with missing parts, a blurred image, or a partial image.

[0016] Furthermore, a query input by a user of the user device 10 is linked to information that identifies the user device 10 and information on the date and time of input. Furthermore, the query may be linked to characteristics of the user device 10 or the user (hereinafter also referred to as user characteristics). The user characteristics may include demographic information. Demographic information is demographic user characteristics (attributes) such as gender, age, residential area, occupation, and family structure.

[0017] In this embodiment, items to be searched for may be tangible or intangible goods or services that can be provided for various services. For example, with regard to digital content services, there are items such as video content such as movies and animations, and still image content such as photographs and illustrations. With regard to e-commerce services, there are items such as intangible or tangible goods handled through online shopping. With regard to travel services, there are items such as information and reservations for hotels, package tours, and transportation. With regard to mobile services, there are items such as mobile devices, public network / Internet connections, and communication usage fees. With regard to sports and cultural services, there are items such as events such as sporting events and concerts, and goods sold at events.

[0018] [Functional Configuration of Information Processing Device 21] An example of the functional configuration of the information processing device 21 according to this embodiment is shown in Fig. 2. As an example of its functional configuration, the information processing device 21 is configured with a query acquisition unit 101, an image generation unit 102, a classifier selection unit 103, an image feature extraction unit 104, an output unit 105, and a learning model storage unit 110. The learning model storage unit 110 stores an image generation model 111, a classifier group 112 including a plurality of classifiers (image recognition models), and an image feature extraction model 113, which are trained machine learning models.

[0019] The query acquisition unit 101 acquires a query transmitted from the user device 10 in response to a user operation. As described above, the query may be text, attribute tags, or an image (e.g., an image representing an item). Alternatively, the query may be a combination of two or more of text, attribute tags, and images. Note that the query is not limited to these and may be any information that the user device 10 can transmit to search for a desired item.

[0020] The image generation unit 102 generates one or more images based on the query acquired by the query acquisition unit 101. Alternatively, the image generation unit 102 may generate one or more images based on a classifier (described later) without the query. In this embodiment, the image generation unit 102 generates an image using a trained image generation model 111 stored in the learning model storage unit 110.

[0021] Here, the image generation model 111 in the learning stage will be described with reference to FIG. 4 . FIG. 4 is a diagram for explaining the image generation model 111 in the learning stage. The image generation model 111 corresponds to a diffusion model in the field of machine learning. When an image is the target, the diffusion model is a model that generates an image by gradually removing noise from a noisy image, and a stable diffusion model is known, for example. The image generation model 111 is configured to also reflect query information, and therefore can be referred to as a query-conditioned diffusion model. Note that while a generative adversarial network (GAN) is also known as a machine learning model for image generation, the diffusion model is capable of generating multiple images with higher accuracy and greater variety than a GAN.

[0022] In FIG. 4 , an image 41 (a reference image for learning) and a query 42 are used as input data in the learning stage. The image 41 is input to an image encoder 401 in the image generation model 111. A variational autoencoder (VAE) may be used as the image encoder 401. The image encoder 401 converts the image 41 (pixel image) into a latent representation in a latent space (a latent image embedding representation). This embeds a high-dimensional image into a low-dimensional latent space, reducing the load on computational processing. The latent representation converted by the image encoder 401 is then subjected to a diffusion process 402 in which noise (e.g., Gaussian noise) is repeatedly added at multiple time steps to generate a noise image 403.

[0023] In the de-diffusion process 404, the noise added in the diffusion process 402 is gradually removed from the noise image 403. A U-Net may be used for noise removal in the de-diffusion process 404. The U-Net is a type of FCN (Fully Convolution Network) and is configured to include multiple blocks including convolution layers and multiple blocks including attention layers. The de-diffusion process 404 also uses information about the query 42 input to the image generation model 111. Specifically, a feature vector (embedding vector) embedded in a feature space is extracted from the query 42 by the query encoder 406. The feature vector extracted by the query encoder 406 represents the features of the query 42 and will hereinafter also be referred to as the query feature. The query encoder 406 is a trained model (Transformer) configured to extract query features from the query 42. If the query 42 is text, for example, CLIP (Contrastive Language-Image Pre-training) can be used as the query encoder 406. The de-diffusion process 404 derives noise to be removed from the input noise image 403 based on the query features obtained by the query encoder 406, and subtracts the derived noise from the noise image 403. When a U-Net is used in the de-diffusion process 404, the query features obtained by the query encoder 406 are input to each of multiple blocks including an attention layer, and noise is derived. In the de-diffusion process 404, the output obtained by subtracting the derived noise from the noise image 403 is finally converted into an image by the image decoder 405, and the image 43 is generated. Thereafter, the error (difference) between the image 43 (generated image) and the image 41 (teacher image) is calculated, and the image generation model 111 is trained to reduce the error. For example, the mean absolute error (MAE) or mean square error (MSE) is calculated from the pixel values ​​of the image 43 and the image 41, and the image generation model 111 is trained so that the error is reduced.

[0024] The images 41 may be at least a portion of all images representing items available on the EC site provided by the search device 22. The images 41 are not limited to images representing items, but may also include images not directly related to items available on the EC site. The learning process (noise image generation and image generation process) described above is performed on the at least a portion of the images. The trained image generation model 111 is stored in the learning model storage unit 110. When an item available on the EC site is updated and the image representing the item is changed (i.e., when a change occurs in the input data), the image generation model 111 is further trained accordingly and stored in the learning model storage unit 110. The image generation model 111 may be trained by a predetermined functional block of the information processing device 21 and stored in the learning model storage unit 110, or may be trained by an external device and stored in the learning model storage unit 110.

[0025] 2, the image generation unit 102 generates one or more images using the image generation model 111 learned according to the above-described flow. The image generation procedure by the image generation unit 102 will be described later.

[0026] When a classifier is used in image generation by the image generation unit 102, the classifier selection unit 103 selects a classifier to be used from a classifier group 112 including multiple classifiers stored in the learning model storage unit 110. The multiple classifiers included in the classifier group 112 are prepared in advance, and each classifier is a trained classifier (image recognition model) configured to classify images. The classifier group 112 and the selection of a classifier will be described later.

[0027] The image feature extraction unit 104 uses a trained image feature extraction model 113 stored in the learning model storage unit 110 to extract a feature vector (embedded vector) embedded in a feature space from each of one or more images generated by the image generation unit 102. The feature vector extracted by the image feature extraction unit 104 corresponds to an image feature that represents a characteristic of each of the images. The image feature extraction model 113 is a learning model that has been trained to extract and output image features of an input image. The image feature extraction unit 104 inputs the image generated by the image generation unit 102 to the feature extraction model 221 and extracts and obtains the image features of the image.

[0028] The output unit 105 outputs the image features of each of the one or more images generated by the image generation unit 102, which are acquired by the image feature extraction unit 104, to the search device 22. Furthermore, the output unit 105 may output image data including the one or more images generated by the image generation unit 102 to the search device 22. Furthermore, the output unit 105 may be configured to output (present) the image data including the one or more images generated by the image generation unit 102 to the user device 10.

[0029] [Functional Configuration of Search Device 22] FIG. 3 shows an example of the functional configuration of the search device 22 according to this embodiment. As an example of its functional configuration, the search device 22 includes an image feature acquisition unit 201, a search unit 202, an output unit 203, and a search database 210. A trained feature extraction model 211 is stored in the learning model storage unit 210. The search database 210 stores data related to items (hereinafter also referred to as item data), including images representing each item. Each item data is assigned identification information that identifies each item and is linked to the image features of each image. Each item data may include attribute information (related information) such as the price and description of each item. The item data stored in the search database 210 may be updated in accordance with the addition and / or deletion of items on the e-commerce site provided by the search device 22 or the addition and / or deletion of search conditions.

[0030] The image feature acquisition unit 201 acquires image features (feature vectors) generated by the information processing device 21. That is, the image feature acquisition unit 201 acquires image features representing features of one or more images generated in the information processing device 21 based on a query transmitted from the user device 10. Alternatively, the image feature acquisition unit 201 may acquire image features representing features of one or more images generated in the information processing device 21 based on a selected classifier. Additionally or alternatively, the image feature acquisition unit 201 may acquire image data including one or more images generated by the information processing device 21.

[0031] The search unit 202 performs an item search based on the image features acquired by the image feature acquisition unit 201. For example, the search unit 202 searches the search database 210 using the image features as search conditions, and acquires item data having the same or similar features as the image features as search results. Furthermore, the search unit 202 creates (constructs) a web page (search result page) including the acquired item data.

[0032] In addition to or instead of this, the search unit 202 may extract image features from the images acquired by the image feature acquisition unit 201, search the search database 210 using the image features as search conditions, and acquire item data. In this case, the search device 22 has a trained image feature extraction model 113. The search unit 202 may extract and acquire image features from the images acquired by the image feature acquisition unit 201 using the image feature extraction model 113, and search the search database 210 using the image features as search conditions to acquire item data. Furthermore, the search unit 202 may create (configure) a search result page including the acquired item data.

[0033] The output unit 203 transmits (distributes) the search result page created by the search unit 202 to the user device 10. The search result page is response data to the query transmitted from the user device 10, and can be displayed on the display unit of the user device 10.

[0034] [Hardware Configuration of Information Processing Device 21 and Search Device 22] Next, an example of the hardware configuration of the information processing device 21 and the search device 22 will be described. Because both devices may have similar hardware configurations, the information processing device 21 will be described here. The user device 10 may also have a similar hardware configuration. FIG. 5 is a block diagram showing an example of the hardware configuration of the information processing device 21 according to this embodiment. The information processing device 21 according to this embodiment can be implemented on a single or multiple computers, mobile devices, or any other processing platform. While FIG. 5 illustrates an example in which the information processing device 21 is implemented on a single computer, the information processing device 21 according to this embodiment may also be implemented in a computer system including multiple computers. The multiple computers may be connected to each other via a wired or wireless network so that they can communicate with each other.

[0035] 5, the information processing device 21 may include a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM 503, an HDD (Hard Disk Drive) 504, an input unit 505, a display unit 506, a communication I / F 507, a GPU (Graphics Processing Unit) 508, and a system bus 509. The information processing device 21 may also include an external memory. The CPU 501 comprehensively controls the operations of the information processing device 21, and controls each of the components (502 to 508) via the system bus 509, which is a data transmission path.

[0036] The ROM 502 is a non-volatile memory that stores control programs and the like necessary for the CPU 501 to execute processing. The programs may be stored in a non-volatile memory such as an HDD 504 or an SSD (Solid State Drive) or in an external memory such as a removable storage medium (not shown). The RAM (Random Access Memory) 503 is a volatile memory that functions as the main memory, work area, and the like of the CPU 501. That is, when executing processing, the CPU 501 loads the necessary programs and the like from the ROM 502 into the RAM 503 and executes the programs and the like to realize various functional operations. In the case of the information processing device 21, the RAM 503 may include the learning model storage unit 110 shown in FIG. 1 .

[0037] The HDD 504 stores, for example, various data and information required when the CPU 501 performs processing using a program. The HDD 504 also stores, for example, various data and information obtained when the CPU 501 performs processing using a program. The input unit 505 is configured with a pointing device such as a keyboard and a mouse. The display unit 506 is configured with a monitor such as a liquid crystal display (LCD). The display unit 506 may function as a GUI (Graphical User Interface) when configured in combination with the input unit 505.

[0038] The communication I / F 507 is an interface that controls communication between the information processing device 21 and an external device. The communication I / F 507 provides an interface with a network and executes communication with the external device via the network. Various data, parameters, and the like are transmitted and received between the information processing device 21 and the external device via the communication I / F 507. In this embodiment, the communication I / F 507 may execute communication via a wired LAN (Local Area Network) or a dedicated line that conforms to a communication standard such as Ethernet (registered trademark). However, the network that can be used in this embodiment is not limited to this, and may be configured as a wireless network. This wireless network includes a wireless PAN (Personal Area Network) such as Bluetooth (registered trademark), ZigBee (registered trademark), or UWB (Ultra Wide Band). The term also includes wireless LANs (Local Area Networks) such as Wi-Fi (Wireless Fidelity) (registered trademark) and wireless MANs (Metropolitan Area Networks) such as WiMAX (registered trademark). It also includes wireless WANs (Wide Area Networks) such as 4G and 5G. Note that the network may be any network that connects devices to each other and enables communication, and the communication standard, scale, and configuration are not limited to those described above. The GPU 508 is a processor specialized for image processing. The GPU 508 works in cooperation with the CPU 501 to perform, for example, image generation processing according to this embodiment. Note that the GPU 508 may perform processing other than image processing.

[0039] At least some of the functions of the elements of the information processing device 21 shown in Fig. 2 can be realized by the CPU 501 executing a program. However, at least some of the functions of the elements of the information processing device 21 shown in Fig. 2 may be operated as dedicated hardware. In this case, the dedicated hardware operates under the control of the CPU 501.

[0040] [Image Generation Procedure] Next, a description will be given of the image generation procedure executed by the image generation unit 102 of the information processing device 21. In this embodiment, three types of image generation procedures (first type, second type, and third type) will be described. For the configuration of the image generation model 111, see FIG. 4.

[0041] 6A is a diagram illustrating a procedure for the first type of image generation. In the first type of image generation, the image generation unit 102 uses the trained image generation model 111 to generate one or more images based on a query transmitted from the user device 10 and acquired by the query acquisition unit 101.

[0042] 6A , the image generation unit 102 inputs a query 61 acquired by the query acquisition unit 101 to the trained image generation model 111. The query 61 is converted into query features by the query encoder 406 and input to the dediffusion process 404. The dediffusion process 404 derives noise to be removed from a noise image 403 based on the query features acquired by the query encoder 406, and subtracts the derived noise from the noise image 403. The output obtained by subtracting the derived noise from the noise image 403 in the dediffusion process 404 is finally converted into an image 62 by the image decoder 405 and output. By performing such processing on different noise images 403, multiple images including the image 62 are generated. Alternatively, if the query 61 is composed of multiple queries (e.g., multiple texts or multiple images), multiple images can be generated by performing such processing on each query feature. Alternatively, multiple images can be generated by changing the configuration or parameters of the image decoder 405. Alternatively, multiple images may be generated by changing the configuration or parameters in the query encoder 406. Note that in this disclosure, generating the image 62 may mean generating multiple images including the image 62 using the above-described procedure.

[0043] As described above, in the first type of image generation, the image generation unit 102 acquires the query 61 input by the user of the user device 10 to search for a desired item, and generates one or more images based on the query 61 and the trained image generation model 111. For image generation, by using query features (feature vectors) indicating the features of the query 61 generated by the query encoder 406, it becomes possible to generate one or more images based on the query 61.

[0044] As mentioned above, the image serving as the query 61 may be an incomplete image, such as an image with missing parts, a blurred image, or a partial image. For example, in the case of an image with missing parts, the missing parts are masked in the image, and noise is added to the area other than the masked part. Then, by combining the generated noise image with the masked part (i.e., the missing parts) and removing the noise from the combined image using a de-diffusion process, a more complete image can be generated in which the missing parts are predicted (inpainted).

[0045] (2) Image Generation by Sampling The second type of image generation is image generation by sampling. Referring to Fig. 6A, the de-diffusion process 404 generates a large amount of output from a single query feature obtained by the query encoder 406 by sampling. The output is converted into multiple images including the image 62 by the image decoder 405 and output. This allows a large amount of images to be generated. The large amount of images corresponds to images forming the distribution of the single query 61.

[0046] (3) Image Generation Using a Classifier FIG. 6B is a diagram illustrating the procedure for the third type of image generation. In the third type of image generation, the image generation unit 102 generates one or more images using a trained image generation model 111 including a classifier. Specifically, the image generation unit 102 generates one or more images using the trained image generation model 111 and gradients from the classifier. This type of image generation corresponds to classifier-guided image generation. In this embodiment, a classifier selected by the classifier selection unit 103 from a classifier group 112 stored in the learning model storage unit 110 is used. Each classifier included in the classifier group 112 is a trained classifier (image recognition model) configured to classify images having predetermined characteristics. Each classifier is assigned identification information that identifies the classifier. Furthermore, each classifier is associated with one or more characteristics (keywords) related to the image to be classified.

[0047] 6B, this type of image generation differs from the first type of image generation (see FIG. 6A) in that the gradient output from a classifier 601 is input to the de-diffusion process 404. The classifier 601 is selected from the classifier group 121 by the classifier selection unit 103.

[0048] In this embodiment, the classifier selection unit 103 can select a classifier using the following procedure. For example, the classifier selection unit 103 can analyze the query 61, extract features included in the query 61, and select a classifier having the features. For example, when the query 61 is text, the classifier selection unit 103 can extract (predict) features such as item names and item genres included in the text, and select a classifier associated with the features. The classifier selection unit 103 can extract the features based on rules or by using machine learning (e.g., the query encoder 406).

[0049] Alternatively, the classifier selection unit 103 may select a classifier in accordance with a user instruction 63. The user may be the user of the user device 10. For example, when the classifier selection unit 103 receives identification information of a classifier from the user as the user instruction 63, the classifier selection unit 103 may select a classifier corresponding to the identification information. Furthermore, when the classifier selection unit 103 receives text from the user as the user instruction 63, including features of an image to be classified or excluded, the classifier selection unit 103 may select a classifier associated with the features indicated by the text. Specifically, when the classifier selection unit 103 receives text from the user, such as "80s fashion style" or "shirts that do not contain words," the classifier selection unit 103 may extract (predict) features indicated by the text and select a classifier associated with the features. The classifier selection unit 103 may extract the features based on a rule or by using machine learning. The user instruction 63 for selecting a classifier may be included in the query 61 or may be transmitted from the user device 10 separately from the query 61.

[0050] Alternatively, the classifier selection unit 103 may select a classifier according to preset setting information 64. The setting information 64 may be set in the classifier selection unit 103 in advance or may be set from an external device. For example, when the setting information 64 includes the type of a web page, the classifier selection unit 103 may select a classifier according to the type of the web page. Specifically, when the setting information 64 includes information about a food-related web page, the classifier selection unit 103 may select a classifier associated with a feature related to food.

[0051] The selected classifier 601 outputs the gradient from the classifier 601 as a weight to the de-diffusion process 404 so as to increase the probability that an image that can be classified by the classifier 601 will be generated. The de-diffusion process 404 removes noise from the noise image 403 in accordance with the query feature output from the query encoder 406 and the gradient output from the classifier 601, thereby generating an image 62. This allows the image generation unit 102 to generate images limited to those that can be classified by the selected classifier 601.

[0052] Note that when a classifier is used, the image generation unit 102 may remove noise from the noise image 403 according to the gradient from the classifier 601 without the query 61, to generate the image 62. For example, the classifier selection unit 103 may select the classifier 601 according to a user instruction 63 transmitted from the user device 10, and the image generation unit 102 may remove noise from the noise image 403 according to the gradient of the classifier 601, to generate the image 62. Alternatively, the classifier selection unit 103 may select the classifier 601 based on the setting information 64 at a predetermined timing, and the image generation unit 102 may remove noise from the noise image 403 according to the gradient from the classifier 601, to generate the image 62.

[0053] As described above, in the third type of image generation, the image generation unit 102 generates one or more images based on the query 61 input by the user of the user device 10 to search for a desired item, the gradients from the classifier 601, and the trained image generation model 111. The classifier 601 can be selected by the classifier selection unit 103 based on the query 61, the user instruction 63, or the setting information 64. This allows the image generation unit 102 to generate an image that can be classified by the selected classifier 601, i.e., an image targeted by the classifier 601. In addition, in this type of image generation, the image generation unit 102 can generate an image according to the classifier selected based on the user instruction 63 or the setting information 64, even without the query 61.

[0054] [Processing Flow] The processing flow executed in the information processing system 1 according to this embodiment will be described. Fig. 7 is a sequence diagram showing the processing flow executed in the information processing system 1. As shown in Fig. 1, the information processing system 1 is configured from a user device 10 and a search system 20 including an information processing device 21 and a search device 22. Here, the processing flow will be described when a query for searching for a desired item is sent from the user device 10 in response to an operation by the user.

[0055] When the user device 10 transmits a query, the query acquisition unit 101 of the information processing device 21 receives and acquires the query (S701). The query may be text (e.g., text information describing an item) or attribute tags indicating attributes (e.g., the item's color, brand, category, genre, size, or target gender (female, male, unisex, etc.)). Alternatively, the query may be a combination of two or more of text, attribute tags, and images. The image generation unit 102 of the information processing device 21 generates multiple images based on the query and the trained image generation model 111 (S702). The image generation process is as described above. Once the multiple images are generated, the image feature extraction unit 104 of the information processing device 21 extracts and acquires image features (feature vectors) for each of the generated images using the trained image feature extraction model 113 (S703). Once the image features are acquired, the output unit 105 of the information processing device 21 outputs the image features to the search device 22 (S704).

[0056] 8A shows an example of images generated in S702 in response to the query transmitted from the user device 10 in S701. FIG. 8A shows an image group 81 generated by the image generation unit 102 of the information processing device 21 when the query 80 transmitted from the user device 10 in S701 is the text "black and white striped V-neck T-shirt." The image group 81 includes four images (images 811, 812, 813, and 814). The query 80 is composed of a relatively long sentence (a long query without breaks). However, in the image generation model 111, the query encoder 406 extracts query features of the query 80, and then generates images that reflect the query features. This makes it possible to generate an image group 81 that takes into account the features of the query 80.

[0057] Returning to FIG. 7 , when the image features are output from the output unit 105 of the information processing device 21, the image feature acquisition unit 201 of the search device 22 acquires the image features (S704). Subsequently, the search unit 202 of the search device 22 performs an item search using the search database 210 based on the image features (S705). For example, the search unit 202 searches the search database 210 using the image features as search criteria and acquires item data having the same or similar features as the image features as search results. Furthermore, the search unit 202 creates (constructs) a search result page including the acquired item data (S706). The output unit transmits (distributes) the created search result page to the user device 10 (S707).

[0058] FIG. 8B shows an example of a search result page created in S706 for the query sent in S701. FIG. 8B shows a search result page 82 including item data searched based on image features extracted from images 811-814 included in the image group 81 generated for the query 80 shown in FIG. 8A. Taking item data 821 in the search result page 82 as an example, the item data 821 may include an image 822 representing the item and attribute information 823 such as the price and description of the item. Since the search result page 82 includes item data searched based on image features extracted from each of the images 811-814 included in the image group 81, the search result page 82 is composed of item data reflecting features explicitly and / or implicitly included in the images 811-814. The search result page 81 is sent to the user device 10 and displayed on the display unit of the user device 10. The user of the user device 10 can receive the search result page 81 as a search result for the query 80.

[0059] In this manner, the search system 20 according to this embodiment accepts a query for a desired item from a user, generates one or more images that reflect the characteristics of the query, and generates search results based on the characteristics of the images, and provides them to the user. Here, the query is not limited to text, but can be any form of query, such as attribute tags or images (including incomplete images). This enables the search system 20 to more flexibly accept query input from users and provide users with appropriate search results in response to the query, which can improve customer satisfaction on e-commerce sites and lead to increased purchasing motivation.

[0060] 7 describes the processing flow when a query is transmitted from the user device 10. However, the information processing device 21 may generate an image based on the query from the user device 10 and the gradient from the selected classifier (S702), as described with reference to FIG. 6B . In this case, the classifier selection unit 103 of the information processing device 21 may select a classifier based on the query received in S701. Alternatively, the classifier selection unit 103 may select a classifier based on a user instruction (user instruction 63 in FIG. 6B ) or setting information (setting information 64).

[0061] Alternatively, as described with reference to FIG. 6B , the information processing device 21 may generate an image based on the gradient from the selected classifier without a query from the user device 10 (S702). In this case, the classifier selection unit 103 may select a classifier based on a user instruction (user instruction 63 in FIG. 6B ) or setting information (setting information 64). When using the gradient from the classifier selected without a query, the information processing device 21 can generate an image regardless of the timing of receiving a query. In response, the search device 22 can create a search result page based on the generated image (S705, S706) and transmit it to the user device 10 (S707). If the user instruction or setting information is based on the user's past web service usage history of the user of the user device 10, the search result page transmitted to the user may include items that match the user's potential needs, which can serve as an effective push advertisement.

[0062] 8A , the image group 81 generated from the text 80 is not presented to the user. However, in other embodiments, the information processing device 21 may be configured to transmit the generated image group 81 to the user device 10 for presentation to the user. A user who has viewed the image group 81 may operate the user device 10 to delete unintended images from the image group 81 and transmit one or more remaining images to the information processing device 21, and the information processing device 21 may acquire image features of the remaining images and transmit them to the search device 22. Alternatively, a user who has viewed the image group 81 may operate the user device 10 to identify one image from the image group 81 that matches the user's intention and transmit it to the information processing device 21, and the information processing device 21 may acquire image features of the identified image and transmit them to the search device 22. This may result in the search results 22 being closer to the results intended by the user.

[0063] Although specific embodiments have been described above, these embodiments are merely examples and are not intended to limit the scope of the present invention. The devices and methods described herein may be embodied in forms other than those described above. Furthermore, appropriate omissions, substitutions, and modifications may be made to the above-described embodiments without departing from the scope of the present invention. Such omissions, substitutions, and modifications are included within the scope of the claims and their equivalents, and belong to the technical scope of the present invention.

[0064] The disclosure of this embodiment includes the following configuration: [1] A search system including: a query acquisition unit that acquires a query for searching for a desired item, an image generation unit that generates an image using the query and a trained machine learning model, and a search unit that searches for an item based on the image.

[0065] [2] The search system described in [1], wherein the trained machine learning model is a diffusion model in which machine learning is performed so as to reduce the error between a generated image generated by removing noise from a noisy image to which noise has been added to a training image and the training image.

[0066] [3] The search system described in [2], wherein the image generation unit generates a plurality of images using the query and the diffusion model, and the search unit searches for items based on the plurality of images.

[0067] [4] The search system described in [1], wherein the trained machine learning model is a diffusion model in which machine learning has been performed to remove noise from training images to which noise has been added and generate the training images, the image generation unit generates multiple images using the query and the diffusion model, and the search unit searches for items based on the multiple images.

[0068] [5] The search system described in [3] or [4], wherein the image generation unit generates the plurality of images using the query and the diffusion model including a trained classifier configured to classify images.

[0069] [6] The search system described in [5], further comprising a selection unit that selects the classifier from a plurality of pre-prepared, trained classifiers configured to classify images.

[0070] [7] A search system described in any of [1] to [6], wherein the query includes any one of text, tags indicating attributes, and images, or a combination of two or more of text, tags indicating attributes, and images.

[0071] [8] A search method including: a query acquisition step of acquiring a query for searching for a desired item; an image generation step of generating an image using the query and a trained machine learning model; and a search step of searching for an item based on the image.

[0072] [9] An information processing device comprising: a query acquisition unit that acquires a query for searching for a desired item; an image generation unit that generates a plurality of images using the query and a trained diffusion model; and a feature vector acquisition unit that acquires a feature vector for searching for the item based on the plurality of images.

[0073]

[10] An information processing method including: a query acquisition step of acquiring a query for searching for a desired item; an image generation step of generating a plurality of images using the query and a trained diffusion model; and a feature vector acquisition step of acquiring a feature vector for searching for the item based on the plurality of images.

[0074]

[11] An information processing program for causing a computer to execute information processing, the program causing the computer to execute processing including a query acquisition process for acquiring a query for searching for a desired item, an image generation process for generating an image using the query and a trained machine learning model, and a search process for searching for an item based on the image.

[0075]

[12] An information processing program for causing a computer to execute information processing, the program causing the computer to execute processes including a query acquisition process for acquiring a query for searching for a desired item, an image generation process for generating a plurality of images using the query and a trained diffusion model, and a feature vector acquisition process for acquiring a feature vector for searching for an item based on the plurality of images.

[0076] 1: Information processing system, 10: User device, 20: Search system, 21: Information processing unit, 22: Search device, 101: Query acquisition unit, 102: Image generation unit, 103: Classifier selection unit, 104: Image feature extraction unit, 105: Output unit, 110: Learning model storage unit, 111: Image generation model, 112: Classifier group, 113: Image feature extraction model, 201: Image feature acquisition unit, 202: Search unit, 203: Output unit, 210: Search database

Claims

1. a query acquisition unit that acquires a query for searching for a desired item; an image generation unit that generates an image using the query and a trained first machine learning model; a search unit that searches for an item based on the image; A search system comprising:

2. the trained first machine learning model is a diffusion model in which machine learning is performed so as to reduce an error between a generated image generated by removing noise from a noisy image in which noise is added to a training image and the training image; The search system according to claim 1 .

3. the image generation unit generates a plurality of images using the query and the diffusion model; the search unit searches for an item based on the plurality of images. The search system according to claim 2 .

4. the trained first machine learning model is a diffusion model in which machine learning is performed to remove noise from training images to which noise has been added and generate the training images; the image generation unit generates a plurality of images using the query and the diffusion model; the search unit searches for an item based on the plurality of images. The search system according to claim 1 .

5. the image generator generates the plurality of images using the query and the diffusion model, the diffusion model including a trained classifier configured to classify images.

5. The search system according to claim 3 or 4.

6. a selection unit that selects the classifier from a plurality of pre-prepared, trained classifiers configured to classify images; The search system according to claim 5 .

7. The query includes any one of text, tags indicating attributes, and images, or a combination of two or more of text, tags indicating attributes, and images. The search system according to any one of claims 1 to 4.

8. The search unit extracts image features from the image using a trained second machine learning model, and searches for the item based on the image features. The search system according to claim 1 .

9. a query acquisition step of acquiring a query for searching for a desired item; an image generation process for generating an image using the query and a trained machine learning model; a search step of searching for an item based on the image; Including, how to search.

10. a query acquisition unit that acquires a query for searching for a desired item; an image generation unit that generates a plurality of images using the query and a trained diffusion model; a feature vector acquisition unit that acquires a feature vector for searching for an item based on the plurality of images; An information processing device comprising: