Product search system, product search method, and program

Abstract

To enable a user to easily find a desired commodity in a store.SOLUTION: A product search system (1) includes an input information acquisition unit (111) that acquires input information including a natural language sentence or an image input to a terminal by a user in a store, a product identifying unit (211) that identifies a product corresponding to the input information using a machine learning model machine-learned to output product identifying information for identifying any one of a plurality of products using the natural language sentence or the image as an input, and a product information presentation unit (213) that presents product information including a situation of the identified product to the user via the terminal with reference to store information including a situation of each product that can be sold in the store among the plurality of products.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a product search system, a product search method, and a program. [Background technology]

[0002] Patent Document 1 describes a system that presents suggested words when a user searches for a product they wish to purchase. The system extracts other search words that the user has used in past searches and that are presumed to have the same search purpose as the most recent search word, and presents at least a portion of the other extracted search words and the most recent search word as suggested words. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 29852 Brochure Summary of the Invention [Problem to be solved by the invention]

[0004] In a store selling products, customers may be unable to find the product they want. Furthermore, due to a lack of manpower, employees may be unable to properly suggest the product the customer is looking for. When using the system described in Patent Document 1 in a store, if the most recent search term is unclear or if previously used search terms cannot be retrieved, appropriate suggested words cannot be presented. Therefore, the system has room for improvement in terms of making it easier for customers or employees to find the desired product in the store.

[0005] One aspect of the present invention has been made to solve the above-mentioned problems, and its purpose is to provide a technology that makes it easier for users to find desired products in a store. [Means for solving the problem]

[0006] In order to solve the above problem, a product search system according to one embodiment of the present invention includes an input information acquisition unit that acquires input information including a natural language sentence or an image input by a user into a terminal in a store; a product identification unit that identifies a product corresponding to the input information using a machine learning model that has been trained to input the natural language sentence or the image and output product identification information that identifies one of a plurality of products; and a product information presentation unit that refers to store information including the status of each of the plurality of products that are available for sale in the store, and presents product information including the status of the identified product to the user via the terminal.

[0007] In order to solve the above problem, a product search method according to one embodiment of the present invention includes an input information acquisition process in which one or more processors acquire input information including a natural language sentence or an image input by a user into a terminal in a store; a product identification process in which the one or more processors identify a product corresponding to the input information using a machine learning model that has been trained to input the natural language sentence or the image and output product identification information that identifies one of a plurality of products; and a product information presentation process in which the one or more processors refer to store information including the status of each product among the plurality of products that is available for sale in the store, and present product information including the status of the identified product to the user via the terminal.

[0008] In order to solve the above-mentioned problems, a program according to one embodiment of the present invention causes one or more processors to execute an input information acquisition process for acquiring input information including a natural language sentence or an image input by a user into a terminal at a store; a product identification process for identifying a product corresponding to the input information using a machine learning model trained by machine learning to input the natural language sentence or the image and output product identification information that identifies one of a plurality of products; and a product information presentation process for referring to store information including the status of each of the plurality of products that are available for sale at the store and presenting product information including the status of the identified product to the user via the terminal. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of the configuration of a product search system according to an embodiment of the present invention. [Figure 2] 2 is a block diagram showing an example of the configuration of the terminal and the server shown in FIG. 1. FIG. [Figure 3] FIG. 2 is a diagram illustrating inputs and outputs of a first model and a second model according to an embodiment of the present invention. [Figure 4] 1 is a flowchart showing the flow of a product search method according to one embodiment of the present invention. [Figure 5] FIG. 10 is a diagram showing an example of a screen according to an embodiment of the present invention. [Figure 6] FIG. 10 is a diagram showing another example of a screen according to an embodiment of the present invention. [Figure 7] FIG. 10 is a flowchart showing the flow of another product search method according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] [Embodiment] A product search system 1 according to one embodiment of the present invention will be described in detail below with reference to the drawings. The product search system 1 is a system that allows a user to search for a desired product in a store. The user may be a customer visiting the store or a store staff member.

[0011] <Configuration of Product Search System 1> Fig. 1 is a block diagram showing an example of the configuration of a product search system 1. As shown in Fig. 1, the product search system 1 includes a terminal 10, a server 20, an analysis device 30, a product image database 70, a product information database 80, and a store information database 90. The terminal 10 can be placed in a store and is communicatively connected to the server 20 via a network NW1 in the store. The network NW1 is a network to which the terminal 10 can be connected in the store, and may include, for example, wireless communication such as a wireless LAN (local area network) or mobile data communication, but is not limited to these.

[0012] The server 20, the analysis device 30, the product image database 70, the product information database 80, and the store information database 90 may be partly or entirely located in the store, but are not necessarily located in the store. The server 20, the analysis device 30, the product image database 70, the product information database 80, and the store information database 90 are communicatively connected via a network NW2. The network NW2 includes, but is not limited to, a wireless LAN, a wired LAN, a WAN, or a combination thereof. Note that the product image database 70, the product information database 80, and the store information database 90 may be partly or entirely located in an external device connected to the product search system 1, instead of being included in the product search system 1.

[0013] (Configuration of Terminal 10) The terminal 10 is a computer used by a user and includes at least a processor and a memory. The terminal 10 may be, for example, a robot, digital signage, a personal computer, a smart device, or the like. The terminal 10 may be placed in a store. The term "placeable in a store" includes, for example, being installed so that it can be moved around the store, being fixed in the store, being loaned to a user in the store, being carried by a user visiting the store, and the like. The "store" is an area where products are sold. The "store" may be the entire area within a building, or may be a partial area within a building (e.g., a tenant, a floor, etc.). The "store" may also include an area attached to the building (e.g., a garden area, an area in front of the entrance, etc.). The "store" may also be an outdoor area.

[0014] 2 is a block diagram showing an example of the configuration of the terminal 10 and the server 20. As shown in FIG. 2, the terminal 10 includes a control unit 110, a storage unit 120, a communication unit 130, an input unit 140, a display unit 150, an imaging unit 160, an audio input unit 170, an audio output unit 180, a drive unit 190, and a movement mechanism 195. The control unit 110 controls each unit of the terminal 10. Details of the control unit 110 will be described later. The storage unit 120 stores various data used by the control unit 110. The communication unit 130 connects to a network NW1 and communicates with the server 20.

[0015] The input unit 140 accepts input operations to the terminal 10. The input unit 140 may be, but is not limited to, a keyboard, a mouse, a controller, a touchpad, etc. The display unit 150 displays an image generated by the terminal 10. The display unit 150 may be, but is not limited to, a liquid crystal display, an organic EL (Electro Luminescence) display, etc. The input unit 140 and the display unit 150 may be integrally formed as, for example, but is not limited to, a touch panel. The imaging unit 160 captures an image of the surroundings and generates a captured image. The audio input unit 170 detects surrounding audio and inputs it to the terminal 10. The audio output unit 180 outputs audio generated by the terminal 10. The drive unit 190 drives a movement mechanism 195 for moving the terminal 10. Note that some or all of the input unit 140, display unit 150, photographing unit 160, audio input unit 170, and audio output unit 180 do not necessarily have to be included in the terminal 10, and may be connected to the terminal 10 as peripheral devices. Also, the drive unit 190 and the movement mechanism 195 do not necessarily have to be included in the terminal 10, depending on the method for realizing the guidance control unit 113, which will be described later.

[0016] The control unit 110 includes an input information acquisition unit 111, a product information output unit 112, and a guidance control unit 113. The input information acquisition unit 111 acquires input information including a natural language sentence or an image input by a user to the terminal 10 in a store. The natural language sentence may be input using a keyboard, which is an example of the input unit 140, or may be input using a software keyboard displayed on the display unit 150. The natural language sentence may also be input as a voice using the voice input unit 170. The image may also be input by having the imaging unit 160 capture an image. The image may also be input by reading it from the storage unit 120. The input information acquisition unit 111 transmits the acquired input information to the server 20.

[0017] The product information output unit 112 presents the user with product information received from the server 20 in response to transmission of input information. The product information is information about a product identified using a machine learning model in response to the input information, and includes the product status. The product status includes the location of the product. The product status may further include the number of products in stock. The product information may also include the product category, product price, etc. in addition to the product status. Note that specific examples of product status and product information are not limited to the examples described above. The product information may be displayed on the display unit 150, may be output from the audio output unit 180, or may be output from both the display unit 150 and the audio output unit 180.

[0018] The guidance control unit 113 controls the terminal 10 to guide the user to the location of the product identified in accordance with the input information. For example, the guidance control unit 113 may control the drive unit 190 to move the device to the location using the movement mechanism 195. Also, for example, the guidance control unit 113 may display a store map on the display unit 150 and superimpose the current location and the location of the product on the store map. Furthermore, the guidance control unit 113 may provide route guidance from the current location to the location using one or both of the display unit 150 and the audio output unit 180. This allows the user to more easily reach the location of the desired product. As a result, it becomes even easier for the user to find the desired product in the store.

[0019] (Server 20 configuration) The server 20 is a computer that transmits product information to the terminal 10 in response to input information received from the terminal 10, and includes at least a processor and a memory. The server 20 may be located in a store, but is not necessarily located in a store.

[0020] As shown in Fig. 2, the server 20 includes a control unit 210, a storage unit 220, and a communication unit 230. The control unit 210 controls each unit of the server 20. The storage unit 220 stores various data used by the control unit 210. The communication unit 230 connects to networks NW1 and NW2 and communicates with other devices. The control unit 210 includes a product identification unit 211, a similar product identification unit 212, and a product information presentation unit 213.

[0021] The product identification unit 211 identifies a product corresponding to the input information using a machine learning model that has been trained to input a natural language sentence or an image and output product identification information that identifies one of a plurality of products. Here, the plurality of products may be, for example, a plurality of products that can be sold in a particular region (e.g., Japan), but is not limited to this. In this embodiment, the machine learning model is stored in the analysis device 30, which will be described later. Therefore, the product identification unit 211 acquires the product identification information from the analysis device 30 by transmitting the natural language sentence or the image to the analysis device 30. Details of the analysis device 30 will be described later.

[0022] Here, the natural language sentence or image to be sent to the analysis device 30 is included in the input information received from the terminal 10. If the input information includes speech, the natural language sentence is extracted by analyzing the speech and sent to the analysis device 30. The process of extracting the natural language sentence from the speech may be performed in the terminal 10 or in the server 20.

[0023] Furthermore, when the product identification unit 211 identifies multiple product candidates corresponding to the input information using a machine learning model, it may present the identified multiple candidates to the user via the terminal 10 and identify the product indicated by the candidate selected by the user. This allows the user to select a desired product from multiple candidates corresponding to their input information, making it easier for them to find the desired product. Note that when the product identification unit 211 identifies a single product corresponding to the input information, it may identify the single product without presenting the single product as a candidate to the user.

[0024] The similar product identification unit 212 identifies, among multiple products, other products that are similar to the identified product and that are available for sale in the store as similar products. For example, the similar product identification unit 212 may identify the other product when the status of the identified product satisfies a predetermined condition. Examples of the predetermined condition include, but are not limited to, when the inventory quantity is below a threshold (e.g., zero), when other products similar to the identified product are running a campaign, etc.

[0025] The product information presentation unit 213 refers to store information including the status of each product available for sale in the store among multiple products, and presents product information including the status of the identified product to the user via the terminal 10. Specific examples of product status and product information have been described above, so detailed explanations will not be repeated. This allows the user to know the status of the desired product in the store using natural language sentences or images, even if they do not know details of the product, such as the product name or category. As a result, it becomes easier for users to find the desired product in the store. Furthermore, making it easier for users to find the desired product also has the advantage for the store, as it alleviates labor shortages and reduces lost purchasing opportunities.

[0026] Furthermore, when similar products are identified, the product information presentation unit 213 presents the user with product information including the status of the similar products. Specific examples of the status of similar products and specific examples of similar product information are similarly explained by replacing the product with a similar product in the explanation of the specific example of the product's status and the specific example of the product information. This allows the user to receive recommendations for other products similar to the desired product that are available in the store. As a result, even if the desired product is not available in the store or is out of stock, for example, similar products can be easily found. This may also lead to the purchase of similar products, which has the benefit of increasing purchasing opportunities and reducing lost purchasing opportunities for the store.

[0027] (Configuration of analysis device 30) The analysis device 30 is a computer that analyzes natural language sentences or images and includes at least a processor and a memory. The analysis device 30 stores a machine learning model in the memory. The machine learning model is a model trained by machine learning to input a natural language sentence or an image and output product identification information that identifies one of a plurality of products. In this embodiment, the machine learning model includes a first model and a second model.

[0028] FIG. 3 is a diagram schematically illustrating input and output of a first model and a second model. In FIG. 3, first model A and first model B are each an example of a first model. First model A has been machine-trained to receive a natural language sentence as input and output product feature information indicating product features. First model B has been machine-trained to receive an image as input and output product feature information. In the following, when there is no need to particularly distinguish between first model A and first model B, they will each be simply referred to as the first model.

[0029] Here, the product feature information includes, for example, information indicating the appearance, classification, ingredients, or sensory characteristics of a product. For example, the appearance characteristics of a product indicate the visual characteristics of the product, such as the shape, pattern, size, color, etc. of the package or the product itself. Furthermore, the sensory characteristics indicate, for example, the taste, smell, feel, etc. The product feature information output from the first model may be a natural language sentence or vector data in a feature space.

[0030] Furthermore, the product feature information output from the first model is not limited to one, and multiple candidates may be output. For example, multiple product feature information candidates such as a "red box" and a "pink box" may be output for a certain image input. Specifically, when multiple product feature information and their respective accuracy levels are output from the first model, the product feature information with the highest accuracy level may be output from the first model, or multiple product feature information with accuracy levels equal to or higher than a threshold level may be selected as candidates.

[0031] The first model can be realized by, for example, a neural network, but is not limited to this. For example, the first model A is machine-learned using training data that associates sample natural language sentences with correct product feature information. The first model B is machine-learned using training data that associates sample images with correct product feature information. It is desirable that the first model A and the first model B are updated by re-learning or additional learning using additional training data provided in response to product updates, etc.

[0032] The second model is machine-trained to output product identification information using the product feature information output from the first model as input. The product identification information is information that identifies one of a plurality of products. As described above, the plurality of products may be products that can be sold in a particular region (e.g., Japan). The product identification information may be, for example, a product code, but is not limited to this.

[0033] Furthermore, the product specifying information output from the second model is not limited to one, and multiple candidates may be output. For example, multiple product specifying information candidates such as "Drink AA" and "Sweets BB" may be output in response to input of product characteristic information such as "red box." For example, when multiple pieces of product specifying information and their respective accuracy are output from the second model, the product specifying information with the highest accuracy may be output from the second model, or multiple pieces of product specifying information with accuracy equal to or higher than a threshold may be output as candidates.

[0034] The second model can be realized by, for example, but is not limited to, a neural network. The second model is machine-learned using training data that associates sample product feature information with correct product identification information. It is desirable that the second model be updated by re-learning or additional training using additional training data provided in response to product updates, etc.

[0035] The analysis device 30 inputs the natural language sentence or image received from the server 20 into the first model to acquire product feature information, and inputs the acquired product feature information into the second model to acquire product identification information. The analysis device 30 also transmits the acquired product identification information to the server 20.

[0036] By configuring the machine learning model with the first and second models described above, it is possible to identify the product desired by the user with greater accuracy because it is based on the product features extracted from the natural language sentence or image input by the user. Furthermore, it is possible to identify the product desired by the user with even greater accuracy because it is based on the product's appearance, classification, ingredients, or sensory features extracted from the natural language sentence or image input by the user.

[0037] (Product image database 70) The product image database 70 stores product images in association with product identification information. It is desirable that the product image database 70 stores an image for each of the multiple products described above, but it is not necessary that images be stored for all of the multiple products. The number of images associated with certain product identification information may be one or more. For example, an image of the product's packaging and an image of the product itself inside the package may be associated with certain product identification information. Furthermore, images of the product's packaging (or the product itself) taken from multiple directions (e.g., front and back, 360-degree directions, etc.) may be associated with certain product identification information. It is desirable that the product image database 70 be updated in response to changes in the product's appearance (e.g., package renewal, etc.).

[0038] (Product Information Database 80) The product information database 80 stores information indicating product attributes in association with the product identification information. It is desirable that the product information database 80 stores information indicating the attributes for each of the above-mentioned multiple products, but it is not necessary that information indicating the attributes for all of the multiple products be stored. Product attributes may include, but are not limited to, classification, manufacturer, size, weight, etc. Furthermore, if the product is a food product, the product attributes may further include, but are not limited to, ingredients, taste characteristics, etc. It is desirable that the product information database 80 be updated in response to changes in product attributes (for example, changes in product size, etc.).

[0039] (Store Information Database 90) The store information database 90 stores information indicating the status of products in a store. As described above, the status of a product includes the location of the product, the number of items in stock, etc. For example, if the product search system 1 is applicable to each of a plurality of stores, the store information database 90 stores store information about the store in association with store identification information that identifies the store. It is desirable that the store information database 90 be updated in response to changes in the status of the product (for example, changes in the location of the product, changes in the number of items in stock).

[0040] <Product search method S1 flow> The product search system 1 configured as above executes a product search method S1. Figure 4 is a flow diagram showing the flow of the product search method S1. As shown in Figure 4, the product search method S1 includes steps S101 to S117.

[0041] In step S101, the input information acquisition unit 111 of the terminal 10 acquires input information including a natural language sentence or an image input by a user to the terminal 10 in a store. The input information acquisition unit 111 transmits the acquired input information to the server 20. Note that the input information acquisition unit 111 may transmit store identification information that identifies the store where the input information was input, together with the input information, to the server 20. Step S101 is an example of an input information acquisition process.

[0042] For example, a customer visiting a store (an example of a user) may input a natural language sentence or an image to a robot (an example of a terminal 10) that is movably arranged within the store. Alternatively, for example, the customer may input a natural language sentence or an image to a smart device (an example of a terminal 10) that is loaned to the store. Alternatively, for example, the customer may input a natural language sentence or an image to a product search application that runs on a smart device (an example of a terminal 10) that the customer carries with them. Alternatively, a store staff member (an example of a user) may input a natural language sentence or an image to a personal computer (an example of a terminal 10) that is arranged in the back room.

[0043] Examples of natural language sentences input by the user include, for example, "I'm looking for sweets in a red package" or "I'm looking for a product with a picture of an animal on it." Examples of images input by the user include, but are not limited to, images of products previously purchased, images of products included as subjects in images obtained from the Internet, etc. Specific examples of the method for inputting natural language sentences or images into the terminal 10 have been described above, and therefore details will not be repeated. In this way, the input information transmitted to the server 20 includes natural language sentences or images.

[0044] In step S102, if the input information includes voice, the product identification unit 211 of the server 20 extracts a natural language sentence by analyzing the voice. If the input information includes text or an image, the process of step S102 is omitted.

[0045] In step S103, the product identification unit 211 transmits a natural language sentence included as text in the input information, a natural language sentence extracted from a voice included in the input information, or an image included in the input information to the analysis device 30. The analysis device 30 inputs the received natural language sentence into a first model A, or inputs the received image into a first model B, thereby acquiring one or more candidates for product feature information.

[0046] In step S104, the analysis device 30 acquires one or more product identification information candidates by inputting each of the product feature information candidates output from the first model A or the first model B into the second model. The analysis device 30 returns the one or more product identification information candidates to the server 20. As a result, the product identification unit 211 acquires one or more product identification information candidates.

[0047] In step S105, the product identification unit 211 acquires, from the product image database 70, an image associated with the product identification information indicated by each of the acquired candidates.

[0048] In step S106, the product identification unit 211 determines whether there are multiple candidates. If the determination in step S106 is Yes (i.e., there are multiple candidates), in step S107, the product identification unit 211 transmits an image of each of the multiple candidates to the terminal 10.

[0049] In step S108, the control unit 110 of the terminal 10 displays the received images of the multiple candidates in a selectable manner on the display unit 150. The control unit 110 also accepts a user operation to select one of the multiple candidates. The control unit 110 also transmits information indicating the candidate selected by the user to the server 20.

[0050] FIG. 5 is a diagram showing an example of a screen displayed on the display unit 150 of the terminal 10 in step S108. In FIG. 5, the example screen G1 includes a plurality of candidate images G11 to G13 and an operation object G14. If the desired product corresponds to one of the images G11 to G13, the user performs an operation to select the candidate indicated by the image. For example, if the input unit 140 and the display unit 150 are configured as touch panels, the selection operation may be an operation of touching the display area of ​​each of the images G11 to G13. However, the selection operation is not limited to this. As a result, information indicating the candidate corresponding to the image in the touched display area is transmitted to the server 20. Furthermore, if the desired product does not correspond to any of the images G11 to G13, the user performs an operation on the operation object G14 "Wrong." When the operation object G14 is operated, information indicating that none of the candidates was selected is transmitted to the server 20.

[0051] The description of step S109 and subsequent steps will be continued with reference to FIG. 4 again. In step S109, the product identification unit 211 of the server 20 determines whether or not a product corresponding to the input information has been identified. For example, if the product candidates corresponding to the input information have been narrowed down to one (No in step S106), the product corresponding to that candidate is identified, and therefore the result is determined as Yes. Also, if the user selects one of the multiple product candidates corresponding to the input information (step S108), the product corresponding to that candidate is identified, and therefore the result is determined as Yes. On the other hand, if the user does not select any of the multiple product candidates corresponding to the input information (step S108), the product is not identified, and therefore the result is determined as No. If the result is determined as Yes in step S109, step S109 is an example of a product identification process.

[0052] If the determination in step S109 is No, step S113 described below is executed If the determination in step S109 is Yes, the next step S110 is executed.

[0053] In step S110, the product information presentation unit 213 acquires store information from the store information database 90. For example, if store identification information has been received from the terminal 10 in step S101, store information associated with the store identification information is acquired.

[0054] In step S111, the product information presentation unit 213 refers to the store information and acquires information indicating the status of the product identified in step S109. For example, information indicating the status of the product, including the location and inventory of the product in the store, is acquired.

[0055] In step S112, the product information presentation unit 213 refers to the store information and determines whether the identified product is available for sale. Being available for sale may mean, for example, that the product is available at the store and in stock, but is not limited to this. If the determination in step S112 is Yes, step S114, which will be described later, is executed.

[0056] Step S113 is executed when step S112 is judged as No (i.e., the identified product is not available for sale) or when step S109 is judged as No (i.e., the product cannot be identified).

[0057] In step S113, the similar product identification unit 212 identifies similar products that are similar to the identified product (or, if no product is identified, a candidate product) and that are available for sale in the store. "Similar products" may mean, for example, that the products have similar characteristics related to appearance, classification, ingredients, or sensation. The similar products can be searched for in the product information database 80. Whether the similar products are available for sale is determined based on the store information.

[0058] In step S114, the product information presentation unit 213 transmits product information including the status of the product identified in step S109 or product information including the status of the similar product identified in step S113 to the terminal 10. Step S114 and the next step S115 are an example of a product information presentation process.

[0059] In step S115, the product information output unit 112 of the terminal 10 displays the product information received from the server 20 on the display unit 150. This allows the user to know the status (location, etc.) of the product available for sale in the store, which corresponds to the natural language sentence or image input in step S101. Furthermore, if the product corresponding to the input natural language sentence or image is not available for sale in the store, the user can receive recommendations of similar products that are available for sale.

[0060] FIG. 6 is a diagram showing an example of a screen displayed on the display unit 150 of the terminal 10 in step S115. Screen example G2 shown in FIG. 6 is displayed, for example, when the user touches the display area of ​​image G13 on screen example G1 of FIG. 5. Screen example G2 includes image G21, information G22, and operation objects G23 and G24. Image G21 shows an image of a product selected by the user. Information G22 shows product information including the status of the product. For example, the product information includes the product name "CC," the product category "confectionery," the price "118 yen," and the display location (an example of a placement location) "shelf A-1." If image G21 and information G22 show the desired product, the user may operate operation object G23 "Guide." If image G21 and information G22 do not show the desired product, the user operates operation object G24 "Wrong." When the operation object G24 is operated, the processing from step S113 (processing to identify and recommend similar products) may be repeated, or the processing from step S101 (processing to input a natural language sentence or an image) may be repeated.

[0061] The description of step S116 and subsequent steps will be continued with reference to Fig. 4 again. In step S116, the guidance control unit 113 of the terminal 10 accepts an operation to instruct guidance. For example, in the screen example G2, an operation on the operation object G23 "Guide" is accepted.

[0062] In step S117, the guidance control unit 113 controls the terminal 10 to guide the user to the location of the product included in the received product information. Specific examples of control for providing guidance are as described above, and therefore will not be repeated in detail.

[0063] Note that the operation for instructing guidance is not limited to the example in Fig. 6. Furthermore, guidance to the location where the product is to be placed does not necessarily have to be performed if the operation of the user instructing guidance is not received. Furthermore, guidance to the location where the product is to be placed may be started automatically without the operation of the user instructing guidance.

[0064] <Variations of input information> The above-described product search system 1 can be modified so that, instead of or in addition to inputting natural language sentences or images, information on classification, ingredients, and sensations is input to search for products. The product search system 1 modified in this way executes a product search method S2. Figure 7 is a flow diagram showing the flow of the product search method S2. As shown in Figure 7, the product search method S2 includes steps S201 to S203 and steps S105 to S117.

[0065] In step S201, the input information acquiring unit 111 of the terminal 10 acquires input information related to the product category, ingredients, or sensation that the user inputs to the terminal 10 at the store. The input information may include information related to some or all of the category, ingredients, and sensation. For example, the user may select a desired category, such as "confectionery," "vegetables," or "fruit," from multiple categories, or may input text. For example, the user may select a desired ingredient from multiple ingredients or may input text. For example, the user may be able to select whether the ingredient contains additives. For example, the user may select a desired taste (an example of a sensation), such as "sweet," "spicy," or "bitter," from multiple tastes, or may input text. The input information acquiring unit 111 transmits the acquired input information and store identification information to the server 20.

[0066] In step S202, the product identification unit 211 of the server 20 searches the product information database 80 to identify one or more product identification information candidates corresponding to the input information.

[0067] Subsequently, steps S105 to S117 are executed. Details of these steps are the same as those in the product search method S1, and therefore detailed description will not be repeated.

[0068] <Variations of input information acquisition> In the above-described product search system 1, the input information acquisition unit 111 may be capable of acquiring input information including natural language sentences or images input by a user to the terminal 10 outside a store. In this case, the terminal 10 may be any computer used by the user, such as a smartphone, tablet, laptop, or desktop personal computer. "Outside the store" may be, for example, the user's home, office, or public transportation, but is not limited to these, as long as it is outside the store. This has the advantage of making it easier for users to find desired products in the store even when they are outside the store. It also has the effect of increasing users' motivation to visit the store.

[0069] <Modifications of functional block configuration and device configuration> The functional block configuration and device configuration of the product search system 1 are not limited to the above-described configurations. For example, the control unit 110 of the terminal 10 may have some or all of the functional blocks of the control unit 210 of the server 20. Furthermore, if the control unit 110 has all of the functional blocks of the control unit 210, the product search system 1 does not necessarily include the server 20. Furthermore, for example, the control unit 210 of the server 20 may have a guidance control unit 113 of the control unit 110, and the terminal 10 may be controlled based on guidance control information received from the server 20. Furthermore, the server 20 and the analysis device 30 may be realized on the same computer.

[0070] <Effects of this embodiment> In this way, by using the product search system 1, even if a user does not know the details of a desired product in a store, the user can input a natural language sentence or an image related to the desired product and learn the location of the product identified using a machine learning model, etc. This has the effect of making it easier for the user to find the product in the store.

[0071] For example, the product search system 1 can present users with desired products based on ambiguous information, such as natural language sentences or language, rather than searches based on product names or categories. It also reduces lost purchasing opportunities for stores due to users being unable to find desired products. It also solves the problem of stores finding it difficult to guide users to desired products due to staff shortages or inexperienced staff. When the product search system 1 is applied to large stores, it reduces the difficulty for users in shopping around in large stores. It also reduces lost purchasing opportunities for stores due to users refraining from shopping because it is difficult to walk around.

[0072] According to each aspect of the present invention described above, the above-mentioned effects can be achieved, thereby contributing to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote inclusive and sustainable industrialization, and build resilient infrastructure."

[0073] 〔summary〕 The product search system according to aspect 1 includes an input information acquisition unit that acquires input information including a natural language sentence or an image input into a terminal by a user in a store; a product identification unit that identifies a product corresponding to the input information using a machine learning model that has been trained to input the natural language sentence or the image and output product identification information that identifies one of a plurality of products; and a product information presentation unit that refers to store information including the status of each product among the plurality of products that is available for sale in the store and presents product information including the status of the identified product to the user via the terminal.

[0074] The product search system of aspect 2 is the same as in aspect 1, in that the status of each product includes the location of the product, and further includes a guidance control unit that controls the terminal to guide the user to the identified location of the product.

[0075] In the product search system of aspect 3, in aspect 1 or 2, the machine learning model includes a first model and a second model, the first model is machine-learned to take the natural language sentence or image as input and output product feature information indicating the characteristics of a product, and the second model is machine-learned to take the product feature information output from the first model as input and output the product identification information.

[0076] A product search system according to a fourth aspect is the same as the third aspect, wherein the product characteristic information includes information indicating the appearance, classification, ingredients, or sensory characteristics of the product.

[0077] In the product search system of aspect 5, in any one of aspects 1 to 4, when the product identification unit identifies multiple product candidates corresponding to the input information using the machine learning model, it presents the identified multiple candidates to the user via the terminal and identifies the product indicated by the candidate selected by the user.

[0078] The product search system of aspect 6 is any one of aspects 1 to 5, and further includes a similar product identification unit that identifies, among the plurality of products, other products that are similar to the identified product and that are available for sale in the store as similar products, and the product information presentation unit presents product information including the status of the similar products to the user.

[0079] The product search system of aspect 7 is any one of aspects 1 to 6, wherein the input information acquisition unit is capable of acquiring the input information including a natural language sentence or an image entered by the user into the terminal outside the store.

[0080] The method of aspect 8 includes an input information acquisition process in which one or more processors acquire input information including a natural language sentence or an image input by a user into a terminal at a store; a product identification process in which the one or more processors identify a product corresponding to the input information using a machine learning model that has been trained to input the natural language sentence or the image and output product identification information that identifies one of a plurality of products; and a product information presentation process in which the one or more processors refer to store information including the status of each product among the plurality of products that is available for sale at the store and present product information including the status of the identified product to the user via the terminal.

[0081] The program of aspect 9 causes one or more processors to execute an input information acquisition process for acquiring input information including a natural language sentence or an image input by a user into a terminal at a store; a product identification process for identifying a product corresponding to the input information using a machine learning model that has been trained to input a natural language sentence or an image and output product identification information that identifies one of a plurality of products; and a product information presentation process for presenting product information including the status of the identified product to the user via the terminal, by referring to store information including the status of each of the plurality of products that are available for sale at the store.

[0082] [Software implementation example] The functions of each device (hereinafter referred to as "device") that constitutes product search system 1 can be realized by a program that causes a computer to function as the device, and a program that causes a computer to function as each control block of the device (particularly each part included in control units 110, 210).

[0083] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program, thereby realizing the functions described in each of the above embodiments.

[0084] The program may be non-transitory and may be recorded on one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

[0085] Furthermore, some or all of the functions of the control blocks can be realized by logic circuits. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of the control blocks can also be realized by, for example, a quantum computer.

[0086] Furthermore, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI ​​may run on the control device or on another device (for example, an edge computer or a cloud server).

[0087] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0088] 1. Product search system 10 devices 20 servers 30 Analyzer 70 Product Image Database 80 Product Information Database 90 Store Information Database 110, 210 control unit 111 Input information acquisition unit 112 Product information output section 113 Guidance control unit 120, 220 storage section 130, 230 Communications Department 140 Input section 150 Display section 160 Photography Department 170 Audio input section 180 Audio output section 190 Drive Unit 195 Moving mechanism 211 Product Specification Department 212 Similar product identification department 213 Product information presentation department

Claims

1. an input information acquisition unit that acquires input information including a natural language sentence or an image input by a user to a terminal in a store; a product identification unit that identifies a product corresponding to input information using a machine learning model that has been trained by machine learning to receive a natural language sentence or an image as input and output product identification information that identifies one of a plurality of products; a product information presentation unit that refers to store information including a status of each product available for sale at the store among the plurality of products, and presents product information including a status of the identified product to the user via the terminal; A product search system equipped with

2. The status of each product includes the location of the product, and a guidance control unit that controls the terminal to guide the user to the location of the identified product. The product search system according to claim 1 .

3. the machine learning models include a first model and a second model; the first model is machine-learned to output product feature information indicating a feature of a product using the natural language sentence or the image as an input; the second model is machine-learned to input the product feature information output from the first model and output the product identification information; The product search system according to claim 1 or 2.

4. The product characteristic information includes information indicating the appearance, classification, ingredients, or sensory characteristics of the product. The product search system according to claim 3 .

5. When the product identification unit identifies multiple product candidates corresponding to the input information using the machine learning model, it presents the identified multiple candidates to the user via the terminal and identifies the product indicated by a candidate selected by the user. The product search system according to claim 1 or 2.

6. a similar product specification unit that specifies, as a similar product, another product that is similar to the specified product among the plurality of products and that is available for sale in the store; The product information presentation unit presents product information including the status of the similar products to the user. The product search system according to claim 1 or 2.

7. the input information acquisition unit is capable of acquiring the input information including a natural language sentence or an image input by the user to the terminal outside the store; The product search system according to claim 1 or 2.

8. an input information acquisition process in which one or more processors acquire input information including a natural language sentence or an image input by a user to a terminal in a store; a product identification process in which the one or more processors identify a product corresponding to the input information using a machine learning model that has been trained by machine learning to input a natural language sentence or an image and output product identification information that identifies one of a plurality of products; a product information presentation process in which the one or more processors refer to store information including a status of each product available for sale at the store among the plurality of products, and present product information including a status of the identified product to the user via the terminal; Product search methods including.

9. one or more processors, an input information acquisition process for acquiring input information including a natural language sentence or an image input by a user to a terminal in a store; a product identification process for identifying a product corresponding to input information using a machine learning model that has been trained to input a natural language sentence or an image and output product identification information that identifies one of a plurality of products; a product information presentation process of referring to store information including a status of each product available for sale at the store among the plurality of products, and presenting product information including a status of the identified product to the user via the terminal; A program that executes the following.

Images