Information processing device and information processing method

Abstract

To provide an information processing device capable of reducing the time and effort of a user required for guidance to a commodity display location, and an information processing method.SOLUTION: The information processing device includes: a receiving unit that acquires an image of a target article; a processing unit that determines a second registered article similar to the target article out of first registered articles arranged in a facility based on images, and when there are multiple second registered articles, estimates a category of the target article based on a category each of the second registered articles belongs, and sets an arrangement location of the estimated target article in a facility corresponding to the category as a guidance location to perform guidance to a user; and a transmission unit that transmits the information about the set guidance location to a terminal owned by the user.SELECTED DRAWING: Figure 17

Description

[Technical field]

[0001] The present disclosure relates to an information processing device and an information processing method. [Background technology]

[0002] In stores such as supermarkets and convenience stores, many products are displayed and sold. Since it is difficult for a user (e.g., a store clerk or a customer) to search for a display location of a desired product in a store, a system for searching for a display location of a product has been considered.

[0003] For example, Patent Document 1 describes a store guidance system equipped with a memory means, a position detection means, a product designation means, a first identification means, a second identification means, and a display control means, in which the memory means stores the display positions of the products and promotional information set for the product display positions, the position detection means detects the position of the information processing terminal, the product designation means accepts designation of a product to be the subject of route guidance, the first identification means identifies from the memory means the promotional information set for a display position near the position detected by the position detection means, the second identification means identifies from the memory means the display position of the product designated by the product designation means, and the display control means displays the identified promotional information and route guidance to the display position on the display means of the information processing terminal.

[0004] For example, Patent Document 2 describes a shopping assistance system that includes a category designation means that allows a user to designate a category to which multiple types of products displayed and sold in a store belong, and a route setting means that sets a guidance route for guiding the user to the display location within the store of products that belong to the category based on the category designated via the category means, and the guidance means thereby guides the user along the guidance route set by the route setting means. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2012-038035 A [Patent Document 2] JP 2012-181767 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in Patent Document 1, the user has to specify the product to be the target of route guidance, which is troublesome for the user. Also, in Patent Document 2, the user has to specify a category to which multiple types of products belong, which is troublesome for the user.

[0007] Non-limiting examples of the present disclosure contribute to providing an information processing device and an information processing method that can reduce the effort required for a user to be guided to a display location of a product. [Means for solving the problem]

[0008] An information processing device according to one embodiment of the present disclosure includes a receiving unit that acquires an image of a target item, a processing unit that determines a second registered item similar to the target item from among first registered items placed in a facility based on the image, and if there are multiple second registered items, estimates a category of the target item based on the category to which each of the second registered items belongs, and sets a location in the facility corresponding to the estimated category of the target item as a guide location to guide a user, and a transmitting unit that transmits information regarding the set guide location to a terminal owned by the user.

[0009] In an information processing method according to one embodiment of the present disclosure, an information processing device determines a second registered item that is similar to a target item among first registered items placed in a facility based on an image of the target item, and if there are multiple second registered items, estimates a category of the target item based on the category to which each of the second registered items belongs, sets a location in the facility that corresponds to the estimated category of the target item as a guide location to guide a user, and transmits information regarding the set guide location to a terminal owned by the user.

[0010] In addition, these comprehensive or specific aspects may be realized by a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium, or may be realized by any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. Effect of the Invention

[0011] According to an embodiment of the present disclosure, the effort required for a user to be guided to a display location of a product can be reduced.

[0012] Further advantages and benefits of an embodiment of the present disclosure will become apparent from the specification and drawings, in which such advantages and / or benefits are provided by some of the embodiments and features described in the specification and drawings, respectively, but not necessarily all of which are provided to obtain one or more identical features. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram showing an example of a product guide system according to an embodiment; [Diagram 2] FIG. 13 is a diagram showing an example of a registration process. [Diagram 3] A diagram showing an example of a database of registered information. [Figure 4] FIG. 13 is a diagram showing an example of a process for setting a guidance position when a user uses the device; [Diagram 5] A first example of the object recognition result. [Figure 6]A second example of object recognition results. [Figure 7] A third example of object recognition results. [Figure 8] A fourth example of object recognition results. [Figure 9] A fifth example of object recognition results. [Figure 10] A sixth example of object recognition results. [Figure 11] Another example of category estimation processing [Figure 12] FIG. 13 is a diagram showing an example of a display screen that requests the user to select a location to be guided to. [Figure 13] FIG. 11 is a diagram showing a first example of a display for providing guidance to a user. [Figure 14] FIG. 11 is a diagram showing a second example of a display for providing guidance to a user. [Figure 15] A flowchart showing an example of a process flow of an information processing device according to an embodiment. [Figure 16] FIG. 1 is a diagram showing a configuration example of a terminal according to an embodiment; [Figure 17] FIG. 1 is a diagram showing an example of the configuration of an information processing device according to an embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functions are denoted by the same reference numerals, and redundant description will be omitted.

[0015] (One embodiment) In stores such as supermarkets and convenience stores, many products are displayed and sold. Because so many products are sold, it takes a lot of effort for a user (e.g., a customer shopping at the store or a store clerk) to search for a desired product. Therefore, a technology is desired that allows a user to navigate to where a desired product is located.

[0016] In this navigation technology, a user specifies a product whose location they want to check, and the location of the specified product is guided to the user. If the user is a customer, the product whose location they want to check corresponds, for example, to a product that the customer wishes to purchase. If the user is a store clerk, the product whose location they want to check corresponds, for example, to a product that the store clerk is about to display. In the following, a product whose location is to be guided to (for example, a product whose location the user wants to check) is referred to as a target product. The location of a product may also be simply referred to as the position of the product. The location may also be referred to as a display location.

[0017] In navigation technology, a method for specifying a target product may be a method in which the user inputs information about the target product (e.g., product name, manufacturer) into a terminal, etc. However, inputting information by the user is time-consuming for the user. Therefore, a method to simplify the specification of a target product by using an image-based object recognition technology is being considered.

[0018] For example, object recognition technology can be used based on an image of a target product to identify what the target product is, thereby simplifying the process of specifying the target product and providing guidance on the location of the target product.

[0019] However, in object recognition technology, the recognition accuracy of the object recognition process decreases depending on the angle at which the target product is photographed and the environment in which the target product is photographed (for example, the surrounding brightness, the direction of the surrounding light, etc.). In addition, since many products similar in color, shape, etc. are sold in a store, it is not possible to identify one that corresponds to the target product, and the recognition accuracy of the object recognition process decreases. When the recognition accuracy of the object recognition process decreases and it is not possible to identify what the target product is, it is difficult to guide the location of the target product. Note that object recognition is recognition using an image, and may therefore be referred to as image recognition.

[0020] Products are classified into several categories. A product category is a group for classifying products, and indicates a group for each type and characteristic of a product. Product categories include relatively narrow categories such as "carbonated drinks", "coffee drinks", "tea drinks", "soy sauce", and "salt", and relatively broad categories such as "soft drinks" and "seasonings". The category "soft drinks" includes "carbonated drinks", "coffee drinks", and "tea drinks". The category "seasonings" includes categories such as "soy sauce" and "salt". In this embodiment, a relatively small category such as "carbonated drinks", "coffee drinks", "tea drinks", "soy sauce", and "salt" is described as a "small category", and a relatively large category including multiple small categories is described as a "large category". At least one small category and at least one large category are associated with one product. It is possible that one product is associated with one of the small category and the large category, but not the other.

[0021] In a store, multiple products in the same category are likely to be displayed on the same shelf or on nearby shelves. Therefore, by showing the location of a shelf or the like corresponding to the category of the target product instead of the location of the target product, the location of the target product can be grasped by visual inspection from the shown shelf, and the location of the target product can be shown with a high probability.

[0022] Therefore, in this embodiment, when the recognition accuracy is lower than a predetermined level, instead of the position of the target product, the position of a shelf or the like corresponding to the category of the target product is provided. Note that, hereinafter, the position of a shelf or the like corresponding to the category of the target product may be described as the position of the category of the target product or the shelf of the category of the target product.

[0023] Fig. 1 is a diagram showing an example of a product guidance system 1 according to the present embodiment. The product guidance system 1 according to the present embodiment is introduced into a store. As shown in Fig. 1, the product guidance system 1 includes a terminal 10 and an information processing device 20.

[0024] The terminal 10 and the information processing device 20 are connected to each other, for example, via a network. The terminal 10 and the information processing device 20 may be wirelessly connected to the network via a base station or an access point that is omitted in FIG.

[0025] The terminal 10 is used by a user (e.g., a store clerk or a customer). The terminal 10 may be in the form of a smartphone, a tablet, or the like, or may be in the form of a wearable device having a head-mounted display. The user operates the terminal 10 to photograph a target product and perform an operation to request guidance to the location of the target product. In response to the user's operation, the terminal 10 transmits request information indicating a request for product guidance to the information processing device 20. The request information includes a product image of the photographed target product.

[0026] The subject of the photograph may be the actual product. For example, if the user is a store clerk, the product to be displayed is photographed. For example, if the user is a customer, the product previously purchased is photographed in order to repurchase the product. Alternatively, the subject of the photograph may be an image of the product printed on a paper medium such as an advertisement. Furthermore, the acquisition of the product image on the terminal 10 is not limited to a method of acquiring the product image by photographing it with a camera or the like. For example, the terminal 10 may acquire an image of the product included in an electronic medium such as a web page.

[0027] The information processing device 20 has the form of a server connected to the terminal 10 and the like via a network. Alternatively, the information processing device 20 may have the form of a cloud server provided on a network (e.g., the Internet). The information processing device 20 acquires request information from the terminal 10, and performs object recognition using a product image included in the request information. Based on the object recognition, the information processing device 20 sets a guidance position to which the user who owns the terminal 10 is guided, and transmits response information indicating the guidance position set to the terminal 10.

[0028] Based on the response information, the terminal 10 displays information to guide the user of the terminal 10. The user (for example, a store clerk or a customer) checks the display and moves to the location of the target product.

[0029] In the product information system 1 shown in Fig. 1, a registration process is carried out to register information about products sold in a store. Next, the registration process will be described.

[0030] <Registration process> Information on each product sold in the store and information on the location of the product (the position of the product) are registered in advance. Products for which image and other information is registered in advance may be all products sold in the store. Hereinafter, products for which image and other information is registered in advance are referred to as registered products. Note that the advance registration process may be performed by the information processing device 20, or may be performed by a processing device different from the information processing device 20.

[0031] Fig. 2 is a diagram showing an example of the registration process, in which the registration process for the placement location of the product is performed after the registration process for information related to the product.

[0032] In the process of registering information about a product, a camera photographs the product from multiple directions for object recognition, and the product images are registered in a product information database (hereinafter, product information DB). In addition to the photographed images, product-related information (e.g., JAN (Japanese Article Number) code, annotation data) is also registered in the product information DB.

[0033] In the process of registering information on the placement location of a product, a product to be registered as a placement location is photographed from one or more directions, and the product to be registered as a placement location included in the photographed image is identified by object recognition with reference to a product information DB, and the JAN code of the identified product is obtained. Then, a camera photographs the shelf on which the product to be registered as a placement location is displayed, and the photographed image is analyzed to detect position information indicating the three-dimensional position of the product to be registered as a placement location on the product shelf. The detected position information is registered in a product navigation database (hereinafter, product navigation DB) as an example of information on the product shelf position. The information on the product shelf position is registered in association with product information (e.g., JAN code) of the product to be registered as a placement location displayed on the product shelf.

[0034] In addition, in the process of registering information regarding the placement locations of products, instead of using image analysis, a store clerk or the like may register the positions of products in advance by manually inputting them.

[0035] The registration process of information related to the location of a product is performed at the stage of introducing the product information system 1. The registration process of information related to the location of a product is also performed when the location of a product is changed.

[0036] Next, an example of information obtained by the registration process shown in FIG. 2 will be described.

[0037] Fig. 3 is a diagram showing an example of a database of registered information, which includes a product information DB and a product navigation DB.

[0038] The product information DB is common to multiple stores, and the product navigation DB is provided for each store.

[0039] In the product information DB, product image information and product information are stored as examples of information related to products.

[0040] The product image information includes an image file and annotation information for performing object recognition of the product. The product image information is stored in the product information DB when a store manager or the like registers the product sold in the store.

[0041] The product image information includes an image file ID that identifies the file of the product image, the name of the image file, the product's JAN code, the shooting angle at which the image was shot, and annotation information.

[0042] The product information includes information for identifying the product. The product information is stored in the product information DB when a store manager or the like registers the products sold in the store. Alternatively, the product information may be imported into the product information DB from a retailer's DB.

[0043] The product information includes the product's JAN (Japanese Article Number) code, the product name, a major category name that is the name of a major category associated with the product, and a minor category name that is the name of a minor category associated with the product.

[0044] The product image information and the product information are linked by the JAN code.

[0045] The product navigation DB includes product position information and product shelf position information as examples of information related to product placement locations.

[0046] The product location information includes information about the location where the product is displayed in the store. The product location information is set when the product guide system is installed. The product location information is also updated when the location of the product is changed.

[0047] The product position information includes a JAN code and product position coordinates. The product position coordinates indicate a position in a three-dimensional space consisting of, for example, an X-axis, a Y-axis, and a Z-axis.

[0048] The product shelf position information includes information about the positions of the product shelves where products are displayed in a store. The product shelf position information is set when the product guide system is introduced. The product position information is also changed when there is a change in the position of the product shelf, the layout of the product shelf, or the products (or their categories) displayed on the product shelf.

[0049] The product shelf position information includes a product shelf ID that identifies the product shelf, a major category name associated with the product shelf, a minor category name associated with the product shelf, and product shelf coordinates. The product shelf coordinates indicate a position in a three-dimensional space consisting of, for example, an X-axis, a Y-axis, and a Z-axis.

[0050] Since a large category includes more products than a small category, the number of product shelves associated with one large category may be greater than the number of product shelves associated with the small categories contained in that large category. In addition, when products of different categories (e.g., small categories or large categories) are displayed on one product shelf, the one product shelf may be associated with multiple categories.

[0051] The size of the product shelves, the arrangement of the product shelves, the way products are displayed on the product shelves, etc. may differ depending on the store. Therefore, the information included in the product shelf position information may be set independently for each store.

[0052] For example, the number of major category names associated with a product shelf and the number of minor category names associated with a product shelf may be one or more. The number of product shelves associated with one major category name and one minor category name is determined according to the number of products belonging to the category.

[0053] Furthermore, when product shelves in a store are arranged on the same plane, the product shelf coordinates may indicate positions on a two-dimensional plane consisting of X and Y axes, rather than indicating positions in the height direction (for example, Z coordinate).

[0054] The category associated with the product shelf may be set in advance for each product shelf. In this case, after the category is set for each product shelf, products are displayed on each product shelf according to the preset category. For example, a store clerk determines a category for each product shelf, and the category is set for each product shelf by manual input or the like.

[0055] Alternatively, a category associated with a product shelf may be set based on product information of products displayed on the product shelf. In this case, before a category is set for each product shelf, the products are displayed on the product shelf, and after the products are displayed, a category associated with the product shelf is estimated based on the product information of the displayed products. For example, if it is detected based on the product information that the category of products displayed on a certain product shelf is commonly "carbonated drinks" at a certain ratio or more, the subcategory associated with the product shelf is estimated to be "carbonated drinks".

[0056] Based on the information in each of the above-mentioned DBs, the information processing device 20 sets a guidance position.

[0057] FIG. 4 is a diagram showing an example of a process for setting a guidance position when a user uses the device.

[0058] In the product guidance system 1 shown in FIG. 1, in the guidance position setting process, object recognition of the product is performed by referring to the product information DB. For object recognition of the target product, a camera photographs the target product from one or more directions. The information processing device 20 acquires an image of the target product photographed, compares the photographed image of the target product with an image (registered image) included in the product image information of each registered product, and calculates a recognition score indicating the similarity between the target product and the registered product. The recognition score for one target product is calculated between one target product and each registered product. One recognition score is associated with one registered product.

[0059] For example, the higher the similarity, the larger the recognition score. For example, the recognition score may take a value between 0% and 100%, with 100% indicating the highest similarity.

[0060] When one of the recognition scores calculated between one target product and each registered product is equal to or greater than a threshold (e.g., 60%), the registered product corresponding to the one recognition score is determined to be the target product. In this manner, the registered product corresponding to the target product is identified based on the recognition scores calculated between one target product and each registered product. In this case, as shown in FIG. 4, product image information and a JAN code included in the product information of the registered product determined to be the target product are acquired with reference to the product information DB. Then, as shown in FIG. 4, product position information associated with the acquired JAN code is acquired with reference to the product navigation DB. Then, the product position coordinates included in the acquired product position information are set as the guidance position of the target product. The threshold may be variable depending on the target product, etc.

[0061] If there are multiple recognition scores that are above a threshold (e.g., 90%) among the recognition scores calculated between a target product and each registered product, the category of the target product is estimated based on the information of the registered products corresponding to the multiple recognition scores.

[0062] For example, product information associated with JAN codes included in product image information of each registered product corresponding to a plurality of recognition scores is acquired by referring to a product information DB. Then, based on the major category and minor category included in the acquired product information, the category of the target product is estimated. Then, as shown in Fig. 4, in the product navigation DB, product shelf position information corresponding to the estimated category is referenced, and the position of the product shelf coordinates indicated by the product shelf position information corresponding to the estimated category is set to the position of the category of the target product.

[0063] Next, an example of a method for setting a guide position of a target product based on the recognition scores calculated between a certain target product and each registered product, and the product name, major category, and minor category of the registered product will be described.

[0064] <Example of setting recognition score and guidance position> An example of a recognition score for object recognition and an example of setting a guide position according to that example will be described below.

[0065] Fig. 5 is a diagram showing a first example of the result of object recognition. Fig. 5 shows registered products corresponding to the top four recognition scores for a certain target product. In the first example, there is one recognition score that is equal to or greater than a threshold value (e.g., 60%). In this example, the target product is identified as "Company A's cola," so the position of Company A's cola is set as the guidance position.

[0066] FIG. 6 is a diagram showing a second example of the result of object recognition. FIG. 6 shows registered products corresponding to the top four recognition scores for a certain target product. In the second example, there are two recognition scores that are equal to or higher than a threshold (e.g., 60%). In the second example, the subcategory of the two registered products whose recognition scores are equal to or higher than the threshold is commonly "carbonated drinks". In this case, among the registered products whose recognition scores are equal to or higher than the threshold, the subcategory of the registered products that make up a certain percentage (e.g., 3 / 4) or more of the registered products share the same subcategory, so it is estimated that the subcategory of the target product is "carbonated drinks". Then, a shelf whose subcategory is "carbonated drinks" is set as the guiding position.

[0067] FIG. 7 is a diagram showing a third example of the result of object recognition. FIG. 7 shows registered products corresponding to the top four recognition scores for a certain target product. In the third example, there are four recognition scores that are equal to or higher than a threshold (e.g., 60%). In the third example, among the four registered products whose recognition scores are equal to or higher than the threshold, there are three registered products whose subcategory is "carbonated drinks" in common. In this case, among the registered products whose recognition scores are equal to or higher than the threshold, the subcategory of the registered products is equal to or higher than a certain percentage (e.g., 3 / 4) and therefore it is estimated that the subcategory of the target product is "carbonated drinks". Then, a shelf whose subcategory is "carbonated drinks" is set as the guiding position.

[0068] FIG. 8 is a diagram showing a fourth example of the result of object recognition. FIG. 8 shows registered products corresponding to the top four recognition scores for a certain target product. In the fourth example, there are four recognition scores that are equal to or higher than a threshold (e.g., 60%). In the fourth example, among the four registered products whose recognition scores are equal to or higher than the threshold, there are two registered products whose small category is "carbonated drinks" in common. In this case, among the registered products whose recognition scores are equal to or higher than the threshold, the small category of the registered products is not common to a certain percentage (e.g., 3 / 4) or more of the registered products, so the small category of the target product is not estimated. In this case, the position of the small category is not guided. And, in the fourth example, among the four registered products whose recognition scores are equal to or higher than the threshold, there are three registered products whose large category is "soft drinks". In this case, among the registered products whose recognition scores are equal to or higher than the threshold, the large category of the registered products is common to a certain percentage (e.g., 3 / 4) or more of the registered products, so the large category of the target product is estimated to be "soft drinks". And, the shelf whose large category is "soft drinks" is set as the guidance position.

[0069] FIG. 9 is a diagram showing a fifth example of the result of object recognition. FIG. 9 shows registered products corresponding to the top four recognition scores for a certain target product. In the fifth example, there are three recognition scores that are equal to or higher than a threshold (e.g., 60%). In this example, among the three registered products whose recognition scores are equal to or higher than the threshold, there are two registered products whose small category is "carbonated drinks" in common. In this case, among the registered products whose recognition scores are equal to or higher than the threshold, there are no registered products whose small category is equal to or higher than a certain percentage (e.g., 3 / 4) of the registered products. Therefore, the small category of the target product is not estimated. In this case, the location of the small category is not announced. And, in the fifth example, among the three registered products whose recognition scores are equal to or higher than the threshold, there are two registered products whose large category is "soft drinks". In this case, among the registered products whose recognition scores are equal to or higher than the threshold, there are no registered products whose large category is equal to or higher than a certain percentage (e.g., 3 / 4) of the registered products. Therefore, the location of the large category is not announced.

[0070] FIG. 10 is a diagram showing a sixth example of the result of object recognition. FIG. 10 shows registered products corresponding to the top four recognition scores for a certain target product. In the sixth example, there is no recognition score that is equal to or greater than a threshold value (e.g., 60%). In this case, it may be determined that recognition has failed, and a user may be instructed to retake the image of the target product. When retaking the image of the target product, the user changes the direction in which the image of the target product is photographed and photographs the target product. However, if the image of the target product is not retaken, it may be determined that the product cannot be introduced. An example of a case in which the image of the target product is not retaken is when the target product is not the actual product, but a flat image of the target product printed on a paper medium or the like is photographed.

[0071] As in the first example described above, if there is one recognition score that is equal to or greater than the threshold, the registered product corresponding to that recognition score is identified as the target product, and the location of the identified registered product is announced.

[0072] As in the second to fifth examples described above, when there are multiple recognition scores equal to or greater than the threshold, the category of the target product is estimated based on the categories of multiple registered products corresponding to the multiple recognition scores. In estimating the category of the target product, a small category estimation process is performed first, and if the small category cannot be estimated, a large category estimation process is performed.

[0073] For example, in the process of estimating the small category, as shown in the second and third examples, if the proportion of the common small category among the small categories of the multiple registered products is equal to or greater than a certain proportion, the common small category is estimated to be the small category of the target product. On the other hand, as shown in the fourth and fifth examples, if the proportion of the common small category among the small categories of the multiple registered products is less than a certain proportion, the small category of the target product is not estimated. If the small category of the target product is not estimated, a process of estimating the large category of the target product is performed. Note that if there is no common small category among the small categories of the multiple registered products, that is, if the small categories of the multiple registered products are different from each other, this may be an example of a case where the proportion of the common small category among the small categories of the multiple registered products is less than a certain proportion.

[0074] For example, in the major category estimation process, as shown in the fourth example, if the proportion of the common major category among the categories of the multiple registered products is equal to or greater than a certain proportion, the common major category is estimated to be the major category of the target product. On the other hand, as shown in the fifth example, if the proportion of the common major category among the categories of the multiple registered products is less than a certain proportion, the major category of the target product is not estimated. Note that a case where there is no common major category among the major categories of the multiple registered products, that is, a case where the major categories of the multiple registered products are different from each other, may be an example of a case where the proportion of the common major category among the major categories of the multiple registered products is less than a certain proportion.

[0075] In addition, the "certain ratio" to be compared with the ratio of the common small category in the small category estimation process and the "certain ratio" to be compared with the ratio of the common large category in the large category estimation process may be set to the same value or may be set independently of each other. For example, these two ratios may be set for each product and / or each category.

[0076] In the above-described category estimation process, when there are multiple recognition scores equal to or greater than a threshold, the category is estimated based on whether or not the proportion of common categories among the categories (small or large categories) of the multiple registered products corresponding to the multiple recognition scores is equal to or greater than a certain proportion. However, the present disclosure is not limited to this. For example, the recognition score may be used in the category estimation process.

[0077] Below, as variations of the category estimation process, an example in which a cumulative total of recognition scores is used, an example in which a cumulative total of the number of products is used, and an example in which the maximum value of the recognition scores is used will be described.

[0078] Fig. 11 is a diagram showing another example of the category estimation process. Like Fig. 5 to Fig. 10, Fig. 11 shows examples of product names of multiple registered products, categories corresponding to the registered products, and recognition scores between a target product and the registered products.

[0079] In FIG. 11, the recognition score of "Company C's Soy Sauce" corresponding to the small category "soy sauce" is 0.9, and the recognition scores of "Company A's Cola", "Company B's Cola", and Company X's Grape Soda corresponding to the small category "carbonated drinks" are 0.6, 0.6, and 0.6, respectively. When the cumulative recognition scores are used in the category estimation process, the cumulative recognition scores of each small category are used as an evaluation value for evaluating the small category. For example, the evaluation value of the small category "soy sauce" may be calculated as 0.9 / (0.9+0.6+0.6+0.6)=0.33, and the evaluation value of the small category "carbonated drinks" may be calculated as (0.6+0.6+0.6) / (0.9+0.6+0.6+0.6)=0.67. Then, it may be estimated that the small category "carbonated drinks" with the larger evaluation value is the small category of the target product. This estimation process can reduce the influence of outliers in the recognition scores ("soy sauce" in the example of FIG. 11).

[0080] In FIG. 11, the number of products corresponding to the subcategory "soy sauce" is one, and the number of products corresponding to the subcategory "carbonated drinks" is three. When the cumulative number of products is used in the category estimation process, the cumulative number of products in each subcategory is used as the evaluation value for evaluating the subcategory. For example, the evaluation value of the subcategory "soy sauce" may be calculated as 1 / 4, and the evaluation value of the subcategory "carbonated drinks" may be calculated as 3 / 4. Then, it may be estimated that the subcategory "carbonated drinks" with the larger evaluation value is the subcategory of the target product. In this estimation process, if the characteristics of registered products in the same category as the target product are similar to each other, it is highly likely that the correct category can be estimated.

[0081] In FIG. 11, the recognition score of "Company C's Soy Sauce" corresponding to the minor category of "soy sauce" is 0.9, and the recognition scores of "Company A's Cola", "Company B's Cola", and Company X's Grape Soda corresponding to the minor category of "carbonated drinks" are 0.6, 0.6, and 0.6, respectively. When the maximum recognition score is used in the category estimation process, since the recognition score of "Company C's Soy Sauce" corresponding to the minor category of "soy sauce" is the maximum, it may be estimated that "soy sauce" is the minor category of the target product. In this estimation process, if there is no error in the recognition score, there is a high possibility that the correct category can be estimated.

[0082] As in the fifth example described above, when the proportion of common small categories among registered products with recognition scores equal to or greater than a threshold is not a certain percentage, and the proportion of common large categories is not a certain percentage, the user may be prompted to select a location to be guided to. For example, options for selecting either a common small category ("carbonated drinks" in the example of FIG. 9) or a common large category ("soft drinks" in the example of FIG. 9) may be displayed on the user's terminal 10 or the like, and the user may be requested to make a selection.

[0083] Fig. 12 is a diagram showing an example of a display screen that requests the user to select a location to be guided. Fig. 12 shows a display screen that is displayed on the display unit or the like of the user's terminal 10 when an object recognition result such as that of Fig. 9 is obtained. The user checks the options on the display unit and operates an operation unit (e.g., a touch panel) superimposed on the display unit to select a category. The terminal 10 transmits information on the selected category to the information processing device 20, and the information processing device 20 transmits information for guiding the location of the selected category to the terminal 10 based on the acquired information.

[0084] <Example of guidance display> Next, a display example for guidance will be described.

[0085] Fig. 13 is a diagram showing a first example of a display for providing guidance to a user. Fig. 13 shows an example in which a user uses a terminal 10 having a goggle-type display unit. Fig. 13 shows an example in which a frame indicating the position of a target product is displayed on the goggle-type display unit and superimposed on the user's field of vision.

[0086] In Fig. 13, since the location of the target product is included in the product shelf included in the user's field of vision, a frame indicating the location of the target product is displayed superimposed on the user's field of vision. The user can search for the location of the target product by checking this frame.

[0087] Fig. 14 is a diagram showing a second example of a display for providing guidance to a user. Fig. 14 shows an example in which a user uses a terminal 10 having a goggle-type display unit, similar to Fig. 13. Fig. 14 shows an example in which a frame indicating the location of a category of a target product, displayed on the goggle-type display unit, is superimposed on the user's field of vision.

[0088] In Fig. 14, since the location of the target product's category is included in the product shelf included in the user's field of vision, a frame indicating the location of the target product's category is superimposed on the user's field of vision. The user can search for the location of the target product's category by checking this frame. Then, the user can search for the location of the target product within the category location.

[0089] 13 and 14, the guidance position (the position of the target product or the position of the target product category) is included in the user's field of view, but if the guidance position is not included in the user's field of view (for example, if the guidance position is far from the user), information indicating the route to the guidance position may be displayed superimposed on the user's field of view. For example, an arrow indicating the route to the guidance position may be displayed superimposed at the user's feet.

[0090] Although FIG. 14 shows an example in which a frame indicating the position of the category of the target product is superimposed, a frame indicating the center position of the position of the category of the target product may be superimposed.

[0091] The method of guidance is not particularly limited. For example, the route to the destination (the location of the target product or the location of the category) may be guided on a two-dimensional plane, or the route to the destination may be guided in a three-dimensional space. The route guidance to the destination on a two-dimensional plane may be called 2D navigation, and the route guidance to the destination in a three-dimensional space may be called 3D navigation.

[0092] For example, 3D navigation using Mixed Reality (MR) may be performed on a terminal 10 in the form of a head-mounted display worn on the head. Alternatively, 3D navigation using Augmented Reality (AR) may be performed on a terminal 10 in the form of a smartphone. In these methods, an arrow indicating the guiding direction may be superimposed on a space photographed by the camera of the terminal 10 to display a route to the destination. In this case, the route to the destination includes information on the height direction. Furthermore, when product display locations are present across multiple floors, information on the floor on which the display locations are present may be included.

[0093] Also, for example, 2D navigation using AR may be performed on the terminal 10 in the form of a smartphone. Alternatively, 2D navigation using wireless technology may be performed on the terminal 10 in the form of a smartphone. For example, the terminal 10 may perform indoor positioning using wireless technology, and perform 2D navigation from the position of the terminal 10 to the position of the target product or the position of the category based on the position of the terminal 10 obtained by the indoor positioning. In these methods, the terminal 10 may display a map viewed from above (for example, an overhead view of the inside of the store) and superimpose an arrow indicating the guiding direction to display the route to the destination. In this case, the route to the destination does not include information on the height direction. Also, the route to the destination may be displayed on signage or electronic shelf labels in the store.

[0094] As described above, object recognition is performed for a target product for which a user wishes to be guided, and if a registered product corresponding to the target product is identified, the location of the registered product is guided to the user. Furthermore, if a registered product corresponding to the target product cannot be identified (for example, if the recognition accuracy is low), the category of the target product is estimated, and the user is guided to the location of the estimated category. With this configuration, the user does not need to operate the terminal to input information about the target product for which the user wishes to be guided, thereby reducing the user's effort required to guide the user to the display location of the product. Furthermore, with this configuration, even if the accuracy of image recognition is insufficient and a registered product corresponding to the target product cannot be identified, the location corresponding to the category of the target product is guided, so that even in such cases, guidance to the display location of the product is possible.

[0095] <Processing flow> Next, an example of the process flow of the information processing device 20 will be described. Note that the process flow described below is an example, and the present disclosure is not limited thereto. Some processes in the process flow described below may not be executed, processes not included in the process flow described below may be added, and the order of the processes may be different from the following description.

[0096] FIG. 15 is a flowchart showing an example of the flow of processing by the information processing device 20 according to the present embodiment.

[0097] The information processing device 20 acquires request information from the terminal 10, and acquires a product image of the target product A included in the request information (S101).

[0098] The information processing device 20 performs object recognition of the target product A that is the target of object recognition (S102). In the object recognition, a recognition score between the target product A and each of the registered products is calculated.

[0099] The information processing device 20 determines whether or not any of the recognition scores calculated for the target product A is equal to or greater than a threshold value (S103).

[0100] If there is no recognition score equal to or greater than the threshold (NO in S103), the flow returns to S101, and the information processing device 20 acquires a commodity image of the target commodity A again and performs object recognition.

[0101] If there is a recognition score equal to or greater than the threshold (YES in S103), the information processing device 20 determines whether there are a plurality of recognition scores equal to or greater than the threshold (S104).

[0102] If there are not multiple recognition scores equal to or above the threshold (NO in S104), that is, if there is only one recognition score equal to or above the threshold, the information processing device 20 identifies the registered product C corresponding to that one recognition score as the target product A (S105).

[0103] The information processing device 20 identifies the position of the registered product C that has been set in advance (S106).

[0104] The information processing device 20 sets the identified position of the registered product C as the guide position (S107).

[0105] If there are a plurality of recognition scores equal to or greater than the threshold (YES in S104), the server device 30 searches for categories of registered products corresponding to the recognition scores equal to or greater than the threshold (S108).

[0106] The server device 30 estimates the category B of the target product A based on the categories of a plurality of registered products (S109).

[0107] The information processing device 20 identifies the estimated location of category B of the target product A (S110).

[0108] The information processing device 20 sets the identified category B position as the guide position (S111).

[0109] The information processing device 20 transmits response information including the set guidance position (S112), and the guidance flow for the target product A ends.

[0110] <Example of device configuration> Next, an example of the device configuration of each of the terminal 10 and the information processing device 20 will be described. Note that the following device configuration is an example, and the present disclosure is not limited thereto. Some of the configurations of the device configurations described below may be omitted, and configurations not included in the device configurations described below may be added.

[0111] Fig. 16 is a diagram showing a configuration example of a terminal 10 according to the present embodiment. The terminal 10 shown in Fig. 16 includes an image detection unit 101, a request information generation unit 102, a transmission unit 103, a reception unit 104, a display control unit 105, and a display unit 106. One or more of the image detection unit 101, the request information generation unit 102, and the display control unit 105 may be collectively referred to as a control unit or a processing unit. Furthermore, the transmission unit 103 and the reception unit 104 may be collectively referred to as a communication unit. Furthermore, the terminal 10 may be referred to as an information processing device.

[0112] The image detection unit 101 acquires a photographed product image. The product image is acquired, for example, from a camera (omitted in FIG. 16) mounted on the terminal 10, or a camera (omitted in FIG. 16) connected to the terminal 10 by wire or wirelessly. The product photographed here is a target product for which a guide is requested. The target product is, for example, a product displayed by a store clerk who owns the terminal 10, or a product that a customer who uses the terminal 10 wishes to purchase.

[0113] The request information generating unit 102 generates request information including a product image of a target product for which guidance is requested and indicating the details of the guidance request.

[0114] The transmitting unit 103 transmits a signal including the request information to the information processing device 20 .

[0115] The receiving unit 104 receives a signal including response information indicating a response to the request information from the information processing device 20.

[0116] The display control unit 105 causes the display unit 106 to display information for providing guidance to a guidance position (the position of the target product or the position of the category of the target product) included in the response information. For example, the display control unit 105 causes the display unit 106 to display information indicating a route from the current position of the terminal 10 to the guidance position, based on the relationship between the current position of the terminal 10 and the guidance position.

[0117] The display unit 106 displays information to the user under the control of the display control unit 105 .

[0118] Fig. 17 is a diagram showing an example of the configuration of the information processing device 20 according to the present embodiment. The information processing device 20 shown in Fig. 17 includes a receiving unit 201, a product information DB 202, an object recognition unit 203, a guidance position setting unit 204, a product navigation DB 205, a response information generation unit 206, and a transmission unit 207. One or more of the object recognition unit 203, the guidance position setting unit 204, and the response information generation unit 206 may be collectively referred to as a control unit or a processing unit. Furthermore, the receiving unit 201 and the transmission unit 207 may be collectively referred to as a communication unit. The product information DB 202 and / or the product navigation DB 205 may be referred to as a storage unit.

[0119] The receiving unit 201 receives a signal transmitted by the terminal 10, and obtains request information contained in the signal.

[0120] As described above, the product information DB 202 stores product image information and product information.

[0121] The object recognition unit 203 performs object recognition based on the request information. For example, the object recognition unit 203 calculates a recognition score indicating a similarity between a product image of a target product included in the request information and a product image of a registered product stored in the product information DB 202.

[0122] If there is one recognition score that is equal to or greater than a threshold value among the calculated recognition scores, the object recognition unit 203 identifies the registered product that corresponds to that one recognition score as the target product.

[0123] When there are a plurality of recognition scores equal to or greater than the threshold among the calculated recognition scores, the object recognition unit 203 refers to the product information DB 202 and searches for the categories of the registered products corresponding to the plurality of recognition scores equal to or greater than the threshold. The object recognition unit 202 estimates the category of the target product based on the searched categories. For example, as described above, when the proportion of a common category among the categories of the registered products is equal to or greater than a predetermined proportion, the category is estimated to be the category of the target product.

[0124] The guidance position setting unit 204 sets a guidance position to guide the user based on the object recognition result. When a registered product corresponding to the target product is identified by the object recognition, the guidance position setting unit 204 refers to the product navigation DB 205 to identify the position of the registered product corresponding to the target product, and sets the identified position of the registered product as the guidance position. In addition, when a category of the target product is estimated by the object recognition, the guidance position setting unit 204 refers to the product navigation DB 205 to identify the position of the category of the target product, and sets the identified category position as the guidance position.

[0125] As described above, the product navigation DB 205 stores product position information, product shelf position information, and the like.

[0126] The response information generating unit 206 generates response information including the guidance position set by the guidance position setting unit 204. The response information may include information related to route guidance such as the route to the guidance position.

[0127] The transmitting unit 207 transmits a signal including the response information generated by the response information generating unit 206 to the terminal 10 .

[0128] 17 shows an example in which the information processing device 20 has both the product information DB 202 and the product navigation DB 205 and performs object recognition processing and setting of a guide position, but the present disclosure is not limited to this. Each of the components of the information processing device 20 shown in FIG. 17 may be included in any of two or more devices.

[0129] For example, an information processing device provided in a store or the like may have a product navigation DB, and a server device to which information processing devices of a plurality of stores are connected may have a product information DB. In this case, the information processing device in the store or the like may request the server device to process object recognition, and the server device may perform object recognition similarly to the object recognition unit 203. Then, the information processing device in the store or the like may obtain a result of the object recognition from the server device, and set a guidance position similarly to the guidance position setting unit 204 based on the result of the object recognition.

[0130] The information processing device described above includes a receiving unit that acquires an image of the target product (an example of a target item), a processing unit that, based on the acquired image, estimates the category of the target product based on the category to which each of the registered products similar to the target product belongs when there are multiple registered products (an example of a second registered item) similar to the target product among the registered products (an example of a first registered item) placed in a store (an example of a facility), and sets the location in the store corresponding to the estimated category of the target product as a guidance location to guide the user, and a transmitting unit that transmits information regarding the set guidance location to a terminal owned by the user.

[0131] With this configuration, the user does not need to operate the terminal to input information about the product he or she wishes to be guided to, reducing the user's effort required to guide the user to the display location of the product. Also, with this configuration, even if the accuracy of image recognition is insufficient and a registered product corresponding to the target product cannot be identified, the location corresponding to the target product's category is guided, so that guidance to the display location of the product can be provided even in such cases.

[0132] In the above-described embodiment, an example of guiding a user to the location of a product sold in a store has been shown, but the present disclosure is not limited thereto. For example, the present disclosure may be applied to a case where a user is guided to the location of an item stored in a warehouse or the like. In addition, when the present disclosure is applied to items stored in a warehouse or the like, the items do not need to be displayed so that the user can easily see them, and may be stored, for example, on a shelf. In other words, the present disclosure may be applied to various systems that guide a user to the location of an item placed in a facility where a plurality of items are placed, such as a store or warehouse.

[0133] In the above-described embodiment, the similarity between the target product and the registered product is determined based on the recognition score indicating the similarity between the product images. However, the similarity between the target product and the registered product may be evaluated based on other criteria. For example, the target product may be identified from the product image of the target product, and the registered product having the same or similar attribute as the identified target product may be evaluated as a registered product similar to the target product. In this case, the more identical or similar attributes there are, the higher the similarity between the target product and the registered product may be evaluated. In addition, the attributes used for evaluation include, for example, the color and size of the product. For example, a registered product having the same color and size as the target product is evaluated as having a relatively high similarity with the target product, and a registered product having a different color and size from the target product is evaluated as having a relatively low similarity with the target product. In the present disclosure, the condition for guiding a position corresponding to the category of the target product is that there are multiple registered products similar to the target product, so the criteria for evaluating the similarity between the target product and the registered product are not limited.

[0134] The operation of changing the category to be guided between large and small categories depending on the identification accuracy of the product image may be changed depending on the attributes and personality of the customer who is the user to be guided. For example, when a relatively large category is guided to a customer who complains about the guidance, it is possible to ask the customer to hold the target product to be guided up to the camera again without changing the category to be guided between large and small categories.

[0135] Although the above embodiment has been described with respect to the case where the present invention is applied to guidance on the display location of a product, the present invention can also be applied to other use cases. For example, the present invention can also be applied to a notification that guides the location of a product that has run out of stock on a display shelf.

[0136] It should be noted that the "... section" in the above embodiments may be a "... circuit", a "... device", a "... unit", or a "... module".

[0137] The present disclosure can be realized in software, hardware, or software in conjunction with hardware.

[0138] Each functional block used in the description of the above embodiments may be partially or entirely realized as an LSI, which is an integrated circuit, and each process described in the above embodiments may be partially or entirely controlled by one LSI or a combination of LSIs. The LSI may be composed of individual chips, or may be composed of one chip that includes some or all of the functional blocks. The LSI may have data input and output. Depending on the degree of integration, the LSI may be called an IC, a system LSI, a super LSI, or an ultra LSI.

[0139] The method of integration is not limited to LSI, and may be realized by a dedicated circuit, a general-purpose processor, or a dedicated processor. Also, after LSI manufacturing, a programmable FPGA (Field Programmable Gate Array) or a reconfigurable processor that can reconfigure the connections and settings of circuit cells inside the LSI may be used. The present disclosure may be realized as digital processing or analog processing.

[0140] Furthermore, if a new integrated circuit technology that can replace LSI appears due to the progress of semiconductor technology or a derivative technology, it is possible to integrate the functional blocks using that technology. The application of biotechnology is also a possibility.

[0141] The present disclosure may be implemented in any type of apparatus, device, or system with communications capabilities (collectively, communications apparatus), including, but not limited to, telephones (e.g., cell phones, smartphones, etc.), tablets, personal computers (PCs) (e.g., laptops, desktops, notebooks, etc.), cameras (e.g., digital still / video cameras), digital players (e.g., digital audio / video players), wearable devices (e.g., wearable cameras, smart watches, tracking devices, etc.), game consoles, digital book readers, telehealth / telemedicine devices, communications-enabled vehicles or mobile conveyances (e.g., cars, planes, ships, etc.), and combinations of the above-mentioned devices.

[0142] Communications Equipment includes, but is not limited to portable or mobile equipment, non-portable or fixed equipment, devices and systems of any kind, such as smart home devices (appliances, lighting equipment, smart meters or metering devices, control panels, etc.), vending machines and any other "Things" that may exist on an Internet of Things (IoT) network.

[0143] Communications include data communications via cellular systems, wireless LAN systems, communications satellite systems, etc., as well as data communications via combinations of these.

[0144] A communications apparatus also includes devices, such as controllers and sensors, connected or coupled to a communications device that performs the communications functions described in this disclosure, such as controllers and sensors that generate control and data signals used by the communications device to perform the communications functions of the communications apparatus.

[0145] The communication apparatus also includes infrastructure facilities, such as base stations, access points, and any other apparatus, device, or system that communicates with or controls the various apparatuses listed above, but are not limited to these.

[0146] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can come up with various modified or amended examples within the scope of the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure. In addition, the components in the above-mentioned embodiments may be arbitrarily combined within the scope of the disclosure.

[0147] Although specific examples of the present disclosure have been described in detail above, these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and alterations of the specific examples exemplified above. [Industrial Applicability]

[0148] The present disclosure is suitable for a product management system. [Explanation of symbols]

[0149] 10 Terminal 20 Information processing device 101 Image detection unit 102 Request information generation unit 103, 207 Transmitter 104, 201 Receiver 105 Display control unit 106 Display section 202 Product information DB 203 Object recognition section 204 Guidance position setting unit 205 Product Navigation DB 206 Response information generation unit

Claims

1. A receiving unit for acquiring an image of a target object; a processing unit that determines a second registered item similar to the target item from among first registered items placed in the facility based on the image, and when there are a plurality of second registered items, estimates a category of the target item based on a category to which each of the second registered items belongs, and sets a placement location in the facility corresponding to the estimated category of the target item as a guide location for guiding a user; a transmission unit that transmits information about the set guide location to a terminal owned by the user; An information processing device comprising:

2. When one of the second registered items is present, the processing unit sets a location of the second registered item in the facility as the guide location. The information processing device according to claim 1 .

3. The processing unit estimates that the first category is the category of the target item when a ratio of the first category common to the second registered items is equal to or greater than a predetermined ratio. The information processing device according to claim 1 .

4. The processing unit estimates a category of the target article based on a category to which each of the second registered articles belongs and a similarity between the target article and each of the second registered articles. The information processing device according to claim 1 .

5. the processing unit determines that, when a ratio of a first category common to the second registered articles is less than a predetermined ratio, the user is prompted to select one of a plurality of categories including the first category; The transmission unit transmits, to the terminal, information requesting selection from among the plurality of categories. The information processing device according to claim 1 .

6. The processing unit determines the placement location corresponding to the category based on a category of each of the first registered articles and an image showing a state in which the first registered articles are placed. The information processing device according to claim 1 .

7. the processing unit, when there are a plurality of placement locations corresponding to the estimated category of the target item, sets the guide location based on a position of the terminal. The information processing device according to claim 1 .

8. An information processing device, determining a second registered item that is similar to the target item among the first registered items located in the facility based on the image of the target item; When there are a plurality of the second registered articles, a category of the target article is estimated based on a category to which each of the second registered articles belongs; setting a location in the facility corresponding to the estimated category of the target item as a guide location to be guided to the user; Transmitting information about the set guide location to a terminal owned by the user; Information processing methods.

Images