Information processing device and information processing method

Abstract

To reduce the workload of store staff.SOLUTION: An information processing device comprises: an acquisition unit which acquires an image of a product being an identification object of image identification; a control unit which decides the highest similarity and the subsequent similarities in the similarities between the image of the product being the identification object and the image of a candidate of the identification object that has been registered, and decides whether to prompt for confirmation of the product on the basis of the relationship between the first candidate which corresponds to the highest similarity and the second candidate which corresponds to the subsequent similarities; and an output unit which outputs an instruction to prompt for confirmation of the product when deciding whether to prompt for confirmation of the product.SELECTED DRAWING: Figure 16

Description

[Technical field]

[0001] The present disclosure relates to an information processing device and an information processing method. [Background technology]

[0002] In stores such as supermarkets and convenience stores, many products are displayed and sold. There are many studies being done to reduce the workload of store staff required for displaying and selling products in such stores.

[0003] For example, Patent Document 1 describes a method for product picking operations in which the fact that the product has been stored in a storage location is registered, the product to be picked and the registered storage location of the product to be picked are output in sequence according to the picking operation, a determination is made as to whether the output product to be picked next is stored in the storage location, and if the product to be picked next is not stored in the storage location, the priority of the picking operation is changed so that the product to be picked next becomes the product to be picked last. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-131574 A Summary of the Invention [Problem to be solved by the invention]

[0005] There is room for consideration of ways to reduce the workload of store staff.

[0006] Non-limiting examples of the present disclosure contribute to providing an information processing device and an information processing method that can reduce the workload of store staff. [Means for solving the problem]

[0007] An information processing device according to one embodiment of the present disclosure includes an acquisition unit that acquires an image of a product to be identified in image recognition, a control unit that determines the highest similarity and the second or subsequent similarity between the image of the product to be identified and each of the registered candidate images of the product to be identified, and decides whether or not to prompt confirmation of the product based on the relationship between a first candidate corresponding to the highest similarity and a second candidate corresponding to the second or subsequent similarity, and an output unit that outputs an instruction to prompt confirmation of the product when it has been determined whether or not to prompt confirmation of the product.

[0008] An information processing method according to one embodiment of the present disclosure includes an information processing device that acquires an image of a product to be identified in image recognition, determines the highest similarity and the second or subsequent similarity among the similarities between the image of the product to be identified and each of the registered candidate images of the product to be identified, determines whether to prompt confirmation of the product based on the relationship between a first candidate corresponding to the highest similarity and a second candidate corresponding to the second or subsequent similarity, and outputs an instruction to prompt confirmation of the product if it has been determined whether to prompt confirmation of the product.

[0009] In addition, these comprehensive or specific aspects may be realized by a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium, or may be realized by any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. Effect of the Invention

[0010] According to an embodiment of the present disclosure, the workload of store staff can be reduced.

[0011] Further advantages and benefits of an embodiment of the present disclosure will become apparent from the specification and drawings, in which such advantages and / or benefits are provided by some of the embodiments and features described in the specification and drawings, respectively, but not necessarily all of which are provided to obtain one or more identical features. [Brief description of the drawings]

[0012] [Figure 1] A diagram showing the first case of image recognition processing [Diagram 2] A diagram showing the second case of image recognition processing. [Diagram 3] FIG. 13 is a diagram showing an example of a format of a notification signal to be notified. [Figure 4] FIG. 13 is a diagram showing an example of displaying the results of image classification processing. [Figure 5A] FIG. 1 shows a first example of an embodiment. [Figure 5B] FIG. 1 shows a variation of the first example in one embodiment. [Figure 6] FIG. 13 is a diagram showing a first example of a format of notification information notified in one embodiment. [Figure 7A] FIG. 1 is a diagram showing a first example of a display displayed on an information processing device according to an embodiment. [Figure 7B] FIG. 13 is a diagram showing a variation of a first example of a display displayed on an information processing device according to an embodiment. [Figure 8A] FIG. 2 shows a second example of an embodiment. [Figure 8B] FIG. 13 shows a variation of the second example in one embodiment. [Figure 9] FIG. 11 is a diagram showing a second example of the format of a notification signal sent in one embodiment. [Figure 10A] FIG. 11 is a diagram showing a second example of a display displayed on the information processing device in one embodiment. [Figure 10B] FIG. 11 is a diagram showing a variation of a second example of a display displayed on an information processing device according to an embodiment. [Figure 11A] FIG. 13 shows a third example of an embodiment. [Figure 11B] FIG. 11 shows a variation of the third example in one embodiment. [Figure 12] FIG. 11 is a diagram showing a third example of the format of a notification signal sent in one embodiment. [Figure 13A] FIG. 11 is a diagram showing a third example of a display displayed on the information processing device in one embodiment. [Figure 13B]FIG. 13 is a diagram showing a variation of a third example of a display displayed on an information processing device according to an embodiment. [Figure 14] A flowchart showing an example of a process flow of a terminal according to an embodiment. [Figure 15] A flowchart showing an example of a process flow of an information processing device according to an embodiment. [Figure 16] FIG. 1 is a diagram showing a configuration example of a terminal according to an embodiment; [Figure 17] FIG. 1 is a diagram showing an example of the configuration of an information processing device according to an embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functions are denoted by the same reference numerals, and redundant description will be omitted.

[0014] (One embodiment) In order to reduce the workload of store staff in displaying and selling products in stores such as supermarkets and convenience stores, the application of image recognition processing to identify (or identify) objects from images of the objects is being considered.

[0015] For example, when a customer visits a store and places items in a shopping basket or cart (hereinafter referred to as cart), it is being considered to automatically complete the payment when purchasing items in the cart by performing image recognition processing on the items being purchased.

[0016] Furthermore, studies are being conducted on detecting missing products and incorrect product display positions on product shelves using image recognition processing.

[0017] An example of image identification processing will be described. The image identification processing is, for example, processing for comparing an image of a product to be identified with images of each of pre-registered product candidates, and identifying which of the registered product candidates the product to be identified is.

[0018] Here, the product to be identified in the image identification process may be referred to as a “product to be identified.” In an automatic payment process for a purchased product, the product to be identified is a product that a shopper has placed in a cart to purchase.

[0019] In the image identification process, a camera provided on the cart captures an image of the product to be identified. The captured image includes the product to be identified.

[0020] In the image classification process, an image of the product to be classified is compared with each image of pre-registered product candidates, and a classification probability indicating the degree of similarity between the product to be classified and the product candidates is calculated. Here, the pre-registered product candidates are, for example, all products sold in a store. Hereinafter, the pre-registered product candidates may be referred to as "candidate products."

[0021] The classification probability is calculated in the range of 0% to 100%. The more similar the classification target product image and the candidate product image are, the higher the classification probability is.

[0022] In the image identification process, the number of images of each candidate product registered in advance may be one or more. For example, for one candidate product, multiple images of the candidate product taken from different angles may be registered. For example, the registered images may be updated by machine learning or the like.

[0023] In the image classification process, the highest classification probability among the calculated classification probabilities is identified, and if the highest classification probability is equal to or greater than a threshold, it is determined that the classification target product corresponds to the candidate product with the highest classification probability. If the highest classification probability is less than the threshold, a candidate product corresponding to the classification target product cannot be identified, and an error is output to the effect that classification is not possible.

[0024] As an example, a case will be described in which a certain identification target product X is identified by image identification processing in a store where products #1 to #N are sold. In this case, products #1 to #N correspond to candidate products. Images of products #1 to #N are registered in advance. When a shopper puts the identification target product X in a cart, a camera photographs the identification target product X. In the image identification processing, the photographed image of the identification target product X is compared with each of the images of products #1 to #N, which are the candidate products, and classification probabilities Px_1 to Px_N indicating the similarity between the identification target product X and each of products #1 to #N are determined. Note that Px_i represents the classification probability indicating the similarity between the identification target product X and product #i.

[0025] Fig. 1 is a diagram showing a first case of image classification processing. Fig. 1 shows an image of a classification target product X, images of products #1 to #N, and classification probabilities Px_1 to Px_N. The first case corresponds to a case in which the classification result is good.

[0026] In the example of Fig. 1, among Px_1 to Px_N, the largest classification probability is Px_2, which is equal to or greater than the first threshold. In this case, it is determined that the classification target product X corresponds to the classification probability Px_2. Note that the first threshold is a threshold that guarantees that the classification result in image classification is good (or sufficiently reliable).

[0027] Fig. 2 is a diagram showing a second case of the image classification process. Fig. 2 shows an image of a classification target product X, images of products #1 to #N, and classification probabilities Px_1 to Px_N, similar to Fig. 1. Note that the second case corresponds to a case where the image of the classification target product X is not clear and the classification result is not good.

[0028] 2, the highest classification probability among Px_1 to Px_N is Px_2, but Px_2 is less than the first threshold value. In this case, it is not determined that the classification target product X corresponds to the product #2 corresponding to the classification probability Px_2.

[0029] In the following, a candidate product corresponding to a certain identification probability Pz will be described as a candidate product with identification probability Pz or a product with identification probability Pz. Also, a identification probability corresponding to a certain candidate product Z may be described as a identification probability of candidate product Z or a identification probability of product Z.

[0030] As described above, in the image recognition process, if the highest recognition probability among the calculated recognition probabilities is less than the first threshold value, an error is output indicating that recognition is not possible because a candidate product corresponding to the product to be recognized cannot be identified.

[0031] An example of a case where an error occurs is when it is difficult to identify a product based on an image of the product to be identified. Examples of products that are difficult to identify based on an image include products that have similar appearances.

[0032] For example, products with similar appearances are products that exist in one category and have multiple types with similar packaging for each type. Here, category refers to a common characteristic defined in the classification of products. Here, type refers to different kinds of products among multiple products that have a common characteristic. For example, multiple products that share a common brand name (a common product name) but have different tastes, scents, colors, etc. are included in the same category because they share a common brand name. However, in this case, the types of each product are different because the tastes, scents, colors, etc. of each product are different.

[0033] For example, among air fresheners with a certain manufacturer's brand name, there are multiple types with different scents, such as an air freshener with a jasmine scent, an air freshener with a rose scent, and an air freshener with a citrus scent. These three air fresheners belong to the same category, but are different types from each other. If the packages of these air fresheners with different scents are similar, it is difficult to distinguish these air fresheners by images. Alternatively, among certain snack foods, there are multiple types with different flavors, such as salty, cheese, and curry flavors, and if the packages of these snack foods with different flavors are similar, it is difficult to distinguish these snack foods.

[0034] In addition, for example, products with similar appearances are products that happen to have similar appearances even though they are in different product categories, etc. For example, if the appearance of a bottle of shampoo happens to be similar to that of a bottle of beverage, it is difficult to distinguish the shampoo from the beverage by images.

[0035] An example of when an error occurs in the following use case is explained.

[0036] <Use case 1> In a case where payment for items in a cart is made automatically by image recognition performed by a terminal mounted on the cart, if the recognition result indicating the result of the image recognition process is not good (for example, if the recognition probability is less than a first threshold value), an alarm indicating that the recognition result is not good and / or a recognition probability that is not good is notified to an information processing device installed at a monitoring location. Note that a monitoring location is, for example, a location where store staff waits and is installed to monitor the situation within the store. Upon confirming the information notified to the information processing device at the monitoring location, the staff rushes to the location that caused the notification (for example, the location of the customer with the cart) and checks it visually or otherwise. The information notified to the information processing device at the monitoring location may be information for instructing the staff to respond.

[0037] In this type of use case 1, if there are many products with poor classification results, the staff will have a lot of work to do.

[0038] <Use case 2> In cases where a terminal carried by a staff member performs image recognition processing to detect that a product displayed on a product shelf is missing, or that the position of a product displayed on the product shelf is different from a specified position, or that a product is not displayed on the product shelf, if the recognition result indicating the result of the image recognition processing is not good, an alarm and / or a recognition probability is notified to an information processing device installed at the monitoring location. After checking the information notified to the information processing device at the monitoring location, the staff member rushes to the location that caused the notification (for example, the product shelf where the shortage occurred) and checks it visually, etc.

[0039] Even with this type of use case 2, if there are many products with poor identification results, it will result in a lot of work for staff.

[0040] Fig. 3 is a diagram showing an example of the format of the notification information to be notified. The format shown in Fig. 3 includes a "cart ID / location information" field, an "additional information" field, a "product ID" field, a "quantity information" field, and a "recognition probability" field.

[0041] The cart ID / location information field includes a cart ID (identification) that is identification information for identifying a cart, and location information of the cart. The location information of the cart is detected based on a communication method such as WiGig and / or Wi-Fi (registered trademark).

[0042] The additional information field includes information indicating whether or not other fields (e.g., a product ID field, a quantity information field) follow the field. For example, if there are no other fields following the additional information field, "0" is set, and if there are other fields following the additional information field, "1" is set.

[0043] The product ID field includes a product ID, which is identification information for identifying a product. In the example of Fig. 3, n types of products are detected. For example, the product ID field (#1) indicates that the product ID is #1.

[0044] The quantity information field is provided in association with the product ID field. The quantity information field contains information indicating the number of products with the product ID indicated by the corresponding product ID field. In the example of FIG. 3, the quantity information field (#i (i is an integer between 1 and n)) is provided in association with the product ID field (#i). The quantity information field (#i) contains information indicating the number of products with product ID #i.

[0045] The identification probability field is provided in association with the product ID field. The identification probability field contains information indicating the identification probability of a product having a product ID indicated by the corresponding product ID field. In the example of Fig. 3, the identification probability field (#i (i is an integer between 1 and n)) is provided in association with the product ID field (#i). The identification probability field (#i) contains information indicating the identification probability of a product having product ID #i.

[0046] For example, as described above, in a case where a certain identification target product X is identified as product #2 by image identification processing in a store selling products #1 to #N, the notification information includes one product ID field, one quantity information field, and one identification probability field. The one product ID field includes the product ID of product #2, the quantity information field includes information indicating that the quantity is one, and the identification probability field includes the identification probability Px_2 between the identification target product X and product #2.

[0047] The information processing device that receives the notification information having the format shown in FIG. 3 extracts the identification probability of each product ID, and determines whether or not there is a product ID whose identification probability is less than a first threshold value.

[0048] If there is a product ID whose identification probability is less than the first threshold, a staff member is notified with instructions to take action regarding the product with that product ID.

[0049] Fig. 4 is a diagram showing a display example of the result of the image identification process. Fig. 4 shows a display example displayed on an information processing device provided in a monitoring location where store staff wait. Fig. 4 is, for example, an example of displaying the result of the image identification process shown in Fig. 2.

[0050] As shown in Fig. 4, an image of the product X to be identified and an image of the product #2 corresponding to Px_2, which has the highest recognition probability calculated for the product X to be identified, are displayed. Also, as shown in Fig. 4, Px_2 and a message requesting confirmation from the staff because Px_2 is less than the first threshold are displayed.

[0051] However, for the two use cases described above, in the operation of the information processing device based on the signal having the format shown in Fig. 3, if there are many products for which the image recognition process results in poor recognition, the work of the staff will increase. In order to deal with this increase in the work of the staff, it is considered to prioritize the work.

[0052] For example, work that impacts revenue and / or customers purchasing products may be given a relatively high priority.

[0053] Therefore, in this embodiment, a method is described in which, for a product that has not been identified satisfactorily by image identification processing, information about multiple products that are candidates for the product and information showing the relationships between those multiple products is provided, thereby making it easier for staff to prioritize tasks and reducing the workload of the staff.

[0054] In the following, an example will be described in which the payment for products in a cart is automatically settled by image recognition performed by a terminal mounted on the cart. In this case, the terminal 10 generates notification information including the recognition result of the image recognition, transmits the notification information to an information processing device 20 installed at a monitoring location where store staff are waiting, and a display based on the notification information is performed on the information processing device 20.

[0055] In the following, a candidate product corresponding to the highest recognition probability among the recognition probabilities calculated for a certain recognition target product X is described as a "first candidate product" for the recognition target product X. In the following, a candidate product corresponding to the second or higher recognition probability among the recognition probabilities calculated for a certain recognition target product X is described as a "second candidate product" for the recognition target product X. The second candidate product may be a candidate product corresponding to a recognition probability equal to or higher than a second threshold among the recognition probabilities excluding the highest recognition probability. For example, the second candidate product corresponds to a candidate product whose recognition probability is equal to or lower than the recognition probability of the first candidate product but equal to or higher than the second threshold. The second threshold is a value less than the above-mentioned first threshold. The second threshold may be a lower limit value of the recognition probability of the second candidate product corresponding to the recognition target product X.

[0056] For example, when the highest classification probability among the classification probabilities calculated for a certain classification target product X is less than a first threshold, the determination that the classification target product X is the first candidate product is likely to be an erroneous determination or erroneous recognition. A product corresponding to a classification probability that is less than the first threshold and equal to or greater than the second threshold is included in the range of erroneous recognition when the classification target product X is erroneously recognized. The erroneous recognition of the classification target product X here refers to the fact that the classification target product X is not the first candidate product, but is recognized as the first candidate product based on the classification probability.

[0057] <First example> When the recognition probability of the first candidate product for the product to be identified is equal to or greater than a threshold, the recognition result for the product to be identified is good. When the recognition probability of the first candidate product for the product to be identified is less than a threshold, the recognition result for the product to be identified is not good. When the recognition result is not good, there is a higher possibility that the actual product to be identified is not the first candidate product than when the recognition result is good. Note that when the actual product to be identified is not the first candidate product, it is assumed that the actual product to be identified is one of the second candidate products.

[0058] For example, a case in which the product to be identified is actually the first candidate product is referred to as case 1, and a case in which the product to be identified is actually the second candidate product is referred to as case 2. When these two cases are taken into consideration, if the price of the first candidate product and the price of the second candidate product are the same, there will be no difference between the amount paid by the customer in case 1 and the amount paid by the customer in case 2.

[0059] Therefore, in the first example, the terminal 10 notifies the information processing device 20 of information indicating the relationship between the price of the first candidate product and the price of the second candidate product. Then, in a case where the price of the first candidate product and the price of the second candidate product are the same, the information processing device 20 displays that the prices are the same. Also, in a case where the price of the first candidate product and the price of the second candidate product are different, the information processing device 20 displays that the prices are different.

[0060] Fig. 5A is a diagram showing a first example in the present embodiment. Fig. 5A shows an image classification result similar to that in Fig. 2. In the example shown in Fig. 5A, the highest classification probability Px_2 is lower than the first threshold. And Px_2 and Px_3 are higher than the second threshold. Fig. 5A also shows the price of each candidate product.

[0061] In the example shown in Fig. 5A, the price of product #2 corresponding to Px_2 is the same as the price of product #3 corresponding to Px_3. In this way, if the price of product #2, which is the first candidate product, and the price of product #3, which is the second candidate product, are the same, even if product X to be identified is actually the second candidate product but is determined to be the first candidate product based on the identification probability, there will be no difference in the amount paid by the customer.

[0062] FIG. 5B is a diagram showing a variation of the first example in the present embodiment. FIG. 5B shows an image classification result similar to that in FIG. 2. In the example shown in FIG. 5B, the highest classification probability Px_2 is lower than the first threshold. Px_2, Px_3, and Px_1 are higher than the second threshold. FIG. 5B also shows the price of each candidate product.

[0063] In the example shown in Fig. 5B, the price of product #2 corresponding to Px_2 is the same as the price of product #3 corresponding to Px_3, but the price of product #2 corresponding to Px_2 is different from the price of product #1 corresponding to Px_1. In this case where the price of product #2, the first candidate product, is different from the price of product #1, the second candidate product, if product X to be identified is actually product #1, the second candidate product, but is determined to be the first candidate product based on the identification probability, there is a possibility that a difference will occur between the amount paid by the customer and the actual amount.

[0064] Therefore, in the first example, whether the price of the first candidate product and the price of the second candidate product are the same or not is notified to the information processing device 20. In the examples of Figures 5A and 5B, the first candidate product is product #2 corresponding to Px_2, and the second candidate product is product #3 corresponding to Px_3 in Figure 5A, and product #3 corresponding to Px_3 and product #1 corresponding to Px_1 in Figure 5B.

[0065] <Example of notification information for the first example> Fig. 6 is a diagram showing a first example of the format of the notification information notified in this embodiment. Fig. 6 shows a format having a plurality of fields. Note that in Fig. 6, the description of the same fields as in Fig. 3 is omitted.

[0066] In the format shown in Fig. 6, a field group having multiple fields is provided for one type of identification target product to be purchased. One field group is associated with one identification target product. For example, field group #k is associated with identification target product #k (k is an integer between 1 and n). Note that Fig. 6 shows an example in which there are n types of identification target products (n is an integer between 1 and n). Here, for example, when multiple units of the same identification target product are to be purchased, the multiple identification target products are regarded as one type of identification target product.

[0067] Below, the field group #1 will be described as an example, but the other field groups #2 to #n may be similar to the field group #1.

[0068] Field group #1 includes a "first candidate product field group," a "same price product field group," and a "different price product field group." Note that in Fig. 6, for the sake of convenience, the "first candidate product field group," the "same price product field group," and the "different price product field group" are shown, but in the notification information, these field groups are distinguished by the contents of the additional information.

[0069] Field group #1 includes an "additional information" field, a "product ID" field, a "quantity information" field, and a "recognition probability" field.

[0070] The first candidate product field group is provided at a specific position (for example, the leftmost position in FIG. 6) in the field group #1. The product ID field of the first candidate product field group contains the product ID of the first candidate product, which is the product with the highest identification probability. In the example of FIG. 5A and FIG. 5B, information about product #2 is included.

[0071] The quantity information field of the first candidate product field group includes information indicating the number of products with the product IDs indicated by the corresponding product ID fields (i.e., first candidate products). The identification probability field of the first candidate product field group includes information indicating the identification probability of the products with the product IDs indicated by the corresponding product ID fields (i.e., the identification probability of the first candidate products).

[0072] The additional information field contains, for example, 3 bits of information.

[0073] Of the three bits in the additional information field, the first bit indicates whether or not there is another field (e.g., a product ID field, a quantity information field) following the field. For example, if there is no other field following, the first bit is set to "0", and if there is another field following, the first bit is set to "1".

[0074] Of the three bits in the additional information field, the second bit indicates whether the product ID in the following product ID field is the first candidate product. For example, if the product ID in the following product ID field is the first candidate product, the second bit is set to "0," and if the product ID in the following product ID field is not the first candidate product, the second bit is set to "1."

[0075] Of the three bits in the additional information field, the third bit indicates whether the product ID in the subsequent product ID field is the same price as the first candidate product. For example, if the price of the product with the subsequent product ID is the same as the first candidate product, the third bit is set to "0," and if the price of the product with the subsequent product ID is not the same as the first candidate product, the third bit is set to "1."

[0076] The same price product field group includes information about the second candidate product that is the same price as the first candidate product. For example, in the example of Figures 5A and 5B, information about product #3 is included. The second candidate product that is the same price as the first candidate product may be simply referred to as the same price product below.

[0077] The product ID field of the same price product field group includes the product ID of the second candidate product that is the same price as the first candidate product. For example, in the example of Figures 5A and 5B, the product ID of product #3 is included.

[0078] The quantity information field of the same price product field group is provided in association with the product ID field of the same price product field group. The quantity information field contains information indicating the number of products of the product ID indicated by the corresponding product ID field.

[0079] The identification probability field of the same-price product field group is provided in correspondence with the product ID field of the same-price product field group. The identification probability field includes information indicating the identification probability of the product of the product ID indicated by the corresponding product ID field. For example, in the example of Figures 5A and 5B, the identification probability Px_3 of product #3 is included.

[0080] The different price product field group includes information about a second candidate product that is priced differently from the first candidate product. For example, in the example of FIG. 5B, information about product #1 is included. The second candidate product that is priced differently from the first candidate product may be simply referred to as a different price product below.

[0081] The product ID field of the different price product field group includes the product ID of the second candidate product that is priced differently from the first candidate product, for example, the product ID of product #1 in the example of FIG. 5B.

[0082] The quantity information field of the different price product field group is provided in association with the product ID field of the different price product field group. The quantity information field includes information indicating the number of products having the product ID indicated by the corresponding product ID field.

[0083] The identification probability field of the different price product field group is provided in correspondence with the product ID field of the different price product field group. The identification probability field includes information indicating the identification probability of the product of the product ID indicated by the corresponding product ID field. For example, in the example of FIG. 5B, the identification probability Px_1 of product #1 is included.

[0084] In addition, FIG. 6 shows an example in which there is one different price product field group, but the present disclosure is not limited to this. For example, when there are multiple different price products, the different price product field group is provided according to the number of different price products. In addition, the third bit of the additional information added before the different price product field group is "1". In addition, FIG. 6 shows an example in which the different price product field group is included in each of field group #1 and field group #n, but the present disclosure is not limited to this. For example, as in the example of FIG. 5A, when there are no different price products, the different price product field group may not exist in the field group. In addition, FIG. 6 shows an example in which a product ID field exists following the different price product field group in each field group, but the present disclosure is not limited to this.

[0085] Also, in FIG. 6, an example in which there is one same-price product field group is shown, but the present disclosure is not limited to this. For example, when there are multiple products with the same price, the same-price product field group is provided according to the number of products with the same price. Note that the third bit of the additional information added before the same-price product field group is "0". Also, in FIG. 6, an example in which the same-price product field group is included in each of field group #1 and field group #n is shown, but the present disclosure is not limited to this. There may be cases in which the same-price product field group does not exist in a field group. Also, in FIG. 6, an example in which a product ID field exists following the same-price product field group in each field group is shown, but the present disclosure is not limited to this.

[0086] If the second candidate product for a certain identified product is a product of the same price as the first candidate product, there is no difference in the amount paid by the customer whether the identified product is the first candidate product or the second candidate product. Therefore, information about products of the same price may be omitted, compressed, thinned out, etc.

[0087] For example, when there are multiple second candidate products that are the same price as the first candidate product, information on the multiple second candidate products that are the same price may be grouped together in one same-price product field group. When grouped together in one same-price product field group, the multiple same-price products may have different product IDs.

[0088] When multiple items are grouped into one same-price item field group, the item ID field of the same-price item field group includes a dedicated ID for indicating that multiple items of the same price are grouped together. This dedicated ID is different from the item ID of the same-price item. For example, this dedicated ID is different from the item ID of each of the candidate items.

[0089] When items are grouped into one same-price item field group, the quantity information field of the same-price item field group contains information indicating the number of items of the same price.

[0090] When the same-priced product fields are grouped into one same-priced product field group, the identification probability field of the same-priced product field group contains information indicating a probability based on the identification probability of each of the multiple same-priced products. The probability based on the identification probability of each of the multiple same-priced products may be, for example, a sum or average value of the identification probabilities of each of the multiple same-priced products, or a maximum or minimum value of the identification probabilities of each of the multiple same-priced products.

[0091] In this way, by consolidating information on a plurality of identically priced products into one identically priced product field group, the information on a plurality of identically priced products can be compressed, and the amount of information in the notification information can be reduced.

[0092] In the first example, as information indicating whether the price of a first candidate product and the price of a second candidate product for a certain identification target product are the same, information on the second candidate product that is the same price as the first candidate product (for example, information in the same priced product field group) and information on the second candidate product that is a different price from the first candidate product (for example, information in the different price information field group) are notified. By having the store staff check this notification, the store staff can confirm the priority of the work.

[0093] For example, in a case where there is a second candidate product for a certain identification target product X that is priced differently from the first candidate product, if the identification target product X is actually the second candidate product, but is determined to be the first candidate product based on the identification probability, a difference will occur between the amount paid by the customer and the actual price of the product. In such a case, by having the staff prioritize checking the identified product X visually or otherwise, it is possible to avoid a difference between the amount paid by the customer and the actual price of the product. As a result, it is possible to avoid cases where customers are inconvenienced by paying more than the purchase amount, or where store profits are reduced by customers paying less than the purchase amount.

[0094] For example, in a case where there is no second candidate product with a different price than the first candidate product for a certain identification target product X, but there is a second candidate product with the same price, if the identification target product X is actually the second candidate product, but is determined to be the first candidate product based on the identification probability, there will be no difference between the price paid by the customer and the actual price of the product. In such a case, the staff does not need to prioritize visual confirmation of the identification target product X, thereby reducing the burden on the staff.

[0095] <Display example of the first example> Fig. 7A is a diagram showing a first example of a display displayed on the information processing device 20 in this embodiment. Fig. 7A is, by way of example, a display example in a case where a classification result of the image classification process as shown in Fig. 5A is notified.

[0096] 7A shows an image of a product X to be classified, an image of a first candidate product #2, and an image of a second candidate product #3 that is the same price as the first candidate product, side by side. The classification probability of each candidate product is also shown.

[0097] In addition, when there are multiple second candidate products that are the same price as the first candidate product and multiple same-priced products are grouped into one same-priced product field group in the notification information, the images and identification probabilities of the second candidate products that are the same price as the first candidate product do not have to be displayed. In this case, the existence of the second candidate products that are the same price as the first candidate product may be displayed by a message or the like.

[0098] Fig. 7B is a diagram showing a variation of the first example of the display displayed on the information processing device 20 in this embodiment. Fig. 7B is, for example, a display example when the classification result of the image classification process as shown in Fig. 5B is notified.

[0099] In Fig. 7B, an image of product X to be identified, an image of product #2 which is a first candidate product, an image of product #3 which is a second candidate product that is the same price as the first candidate product, and an image of product #1 which is a second candidate product that is not the same price as the first candidate product are shown side by side. The identification probability of each product is also shown. In the case of Fig. 7B, the prices of product #2 and product #1 are not the same. In this case, a message is displayed to the store staff encouraging them to visually check.

[0100] In the first example, as shown in Figures 7A and 7B, whether the price of the second candidate product is the same as that of the first candidate product is displayed. If the prices are different, the price of the second candidate product is also displayed. Note that the display mode when the prices are different (e.g., the color and size of the characters, whether or not they blink, etc.) may be different from the other display modes. By changing the display mode, the difference in price is visually emphasized to a person checking the display.

[0101] Furthermore, if the recognition result satisfies a certain judgment condition, a message is displayed requesting that the store staff visually confirm the product to be purchased (i.e., the product to be recognized).

[0102] For example, if the classification result satisfies the condition that the classification probability of a second candidate product that is different in price from the first candidate product is equal to or greater than a second threshold, a message is displayed requesting the store staff to visually confirm the product to be purchased (i.e., the product to be classified). Note that the certain judgment condition is not limited to this example.

[0103] As described above, in the first example, the terminal 10 notifies the information processing device 20 of information indicating the relationship between the price of the first candidate product and the price of the second candidate product, and thereby the store staff can check, on the information processing device 20, whether or not there is a difference in the amount paid by the customer when the product to be identified is the first candidate product and when it is the second candidate product. Therefore, the store staff can omit the work required for checking when the identification result of the image identification process is not good, and the workload of the store staff can be reduced.

[0104] In a first example, the terminal 10 determines whether or not to prompt the user to confirm the identification target product based on the relationship between the price of the first candidate product and the price of the second candidate product. For example, if the price of the first candidate product and the price of the second candidate product are not the same, it is determined that the user should confirm the identification target product. Then, when the terminal 10 determines whether or not to prompt the user to confirm the identification target product, it transmits an instruction (for example, notification information) to prompt the user to confirm the identification target product.

[0105] Here, the instruction (e.g., notification information) to prompt the user to check the product to be identified may correspond to information (e.g., a group of different priced product fields) indicating that the price of the first candidate product and the price of the second candidate product are different. In the above example, the notification information is also sent when the price of the first candidate product and the price of the second candidate product are the same, but when the price of the first candidate product and the price of the second candidate product are the same, the notification information does not include the group of different priced product fields. Therefore, the display based on the notification information does not display a message to prompt the user to check the product to be identified because the prices are different. Therefore, the notification information when the price of the first candidate product and the price of the second candidate product are the same does not need to correspond to an instruction to prompt the user to check the product to be identified.

[0106] In the first example, the terminal 10 may determine not to prompt the user to confirm the identification target product when the difference between the price of the first candidate product and the price of the second candidate product is less than a predetermined value.

[0107] In the first example, the terminal 10 changes the notification information depending on whether the price of the first candidate product and the price of the second candidate product are the same or different from the price of the first candidate product and the price of the second candidate product.

[0108] <Second example> As in the first example, a case in which the product to be identified is actually the first candidate product is referred to as case 1, and a case in which the product to be identified is actually the second candidate product is referred to as case 2. When these two cases are considered, if the first candidate product and the second candidate product belong to the same category and the prices of different types of products in the category are the same, there will be no difference between the amount paid by the customer in case 1 and the amount paid by the customer in case 2.

[0109] Therefore, in the second example, the terminal 10 notifies the information processing device 20 of information indicating the relationship between the category of the first candidate product and the category of the second candidate product. Then, in a case where the category of the first candidate product and the category of the second candidate product are the same, the information processing device 20 displays that the categories are the same.

[0110] Fig. 8A is a diagram showing a second example in this embodiment. Fig. 8A shows an image classification result similar to that in Fig. 2. In the example shown in Fig. 8A, the highest classification probability Px_2 is lower than the first threshold. And Px_2 and Px_3 are higher than the second threshold. Fig. 8 also shows the category of each candidate product and the flavor (orange, apple, etc.) of each candidate product.

[0111] In the example shown in Fig. 8A, the category of product #2 corresponding to Px_2 is the same as the category of product #3 corresponding to Px_3. In this way, if the category of product #2, which is the first candidate product, is the same as the category of product #3, which is the second candidate product, there will be no difference in the amount paid by the customer even if product X to be identified is actually the second candidate product but is determined to be the first candidate product based on the identification probability.

[0112] Fig. 8B is a diagram showing a variation of the second example in this embodiment. Fig. 8B shows an image classification result similar to that in Fig. 2. In the example shown in Fig. 8B, the highest classification probability Px_2 is lower than the first threshold. Px_2, Px_3, and Px_1 are higher than the second threshold. Fig. 8B also shows the category of each candidate product and the flavor of each candidate product (orange, apple, etc.).

[0113] 8B, the category of product #2 corresponding to Px_2 is the same as the category of product #3 corresponding to Px_3, but the category of product #2 corresponding to Px_2 is different from the category of product #1 corresponding to Px_1. In this case where the category of product #2, which is the first candidate product, is different from the category of product #1, which is the second candidate product, if product X to be identified is actually the second candidate product but is determined to be the first candidate product based on the identification probability, there is a possibility that a difference will occur in the amount paid by the customer.

[0114] Therefore, in the second example, whether the category of the first candidate product is the same as the category of the second candidate product is notified to the information processing device 20. In the examples of Figures 8A and 8B, the first candidate product is product #2 corresponding to Px_2, and the second candidate product is product #3 corresponding to Px_3 in Figure 8A, and product #3 corresponding to Px_3 and product #1 corresponding to Px_1 in Figure 8B.

[0115] <Example of notification information for the second example> Fig. 9 is a diagram showing a second example of the format of the notification information notified in the present embodiment. Fig. 9 shows a format having a plurality of fields. Note that in Fig. 9, the description of the same fields as in Figs. 3 and 6 is omitted.

[0116] In the format shown in Fig. 9, a field group having a plurality of fields is provided for one type of identification target product to be purchased, similarly to Fig. 6. One field group is associated with one identification target product. For example, field group #k is associated with identification target product #k.

[0117] Below, the field group #1 will be described as an example, but the other field groups #2 to #n may be similar to the field group #1.

[0118] Field group #1 includes a “first candidate product field group,” a “same category product field group,” and a “different category product field group.” Note that in Fig. 9, for the sake of convenience, the “first candidate product field group,” the “same category product field group,” and the “different category product field group” are shown, but in the notification information, these field groups are distinguished by the contents of the additional information.

[0119] Field group #1 includes an "additional information" field, a "product ID" field, a "quantity information" field, and a "recognition probability" field.

[0120] The first candidate product field group includes information about the first candidate products, similar to the example of FIG.

[0121] The additional information field contains, for example, 3 bits of information.

[0122] Of the three bits in the additional information field, the first bit indicates whether or not there is another field (e.g., a product ID field, a quantity information field) following the field. For example, if there is no other field following, the first bit is set to "0", and if there is another field following, the first bit is set to "1".

[0123] Of the three bits in the additional information field, the second bit indicates whether the product ID in the following product ID field is the first candidate product. For example, if the product ID in the following product ID field is the first candidate product, the second bit is set to "0," and if the product ID in the following product ID field is not the first candidate product, the second bit is set to "1."

[0124] Of the three bits in the additional information field, the third bit indicates whether the product ID in the following product ID field is in the same category as the first candidate product. For example, if the category of the second candidate product in the following product ID is the same as the first candidate product, the third bit is set to "0," and if the category of the second candidate product in the following product ID is not the same as the first candidate product, the third bit is set to "1."

[0125] The same category product field group includes information about the second candidate product that is in the same category as the first candidate product. For example, in the example of Figures 8A and 8B, information about product #3 is included. The second candidate product that is in the same category as the first candidate product may be simply referred to as the same category product below.

[0126] The product ID field of the same category product field group includes the product ID of the second candidate product that is in the same category as the first candidate product. For example, in the example of Figures 8A and 8B, the product ID of product #3 is included.

[0127] The quantity information field of the same category product field group is provided in association with the product ID field of the same category product field group. This quantity information field contains information indicating the number of products of the product ID indicated by the corresponding product ID field.

[0128] The identification probability field of the same category product field group is provided in correspondence with the product ID field of the same category product field group. The identification probability field contains information indicating the identification probability of the product of the product ID indicated by the corresponding product ID field. For example, in the example of Figures 8A and 8B, the identification probability Px_3 of product #3 is included.

[0129] The different category product field group includes information about a second candidate product that is in a category different from that of the first candidate product. For example, in the example of FIG. 8B, information about product #1 is included. The second candidate product that is in a category different from that of the first candidate product may be simply referred to as a different category product below.

[0130] The product ID field of the different category product field group includes the product ID of the second candidate product, which is in a category different from the category of the first candidate product. For example, in the example of FIG. 8B, the product ID of product #1 is included.

[0131] The quantity information fields of the different category product field groups are provided in association with the product ID fields of the different category product field groups. The quantity information fields include information indicating the number of products having the product IDs indicated by the corresponding product ID fields.

[0132] The classification probability fields of different category product field groups are provided in correspondence with the product ID fields of different category product field groups. The classification probability fields contain information indicating the classification probability of the product of the product ID indicated by the corresponding product ID field. For example, in the example of FIG. 8B, the classification probability field contains the classification probability Px_1 of product #1.

[0133] In addition, FIG. 9 shows an example in which there is one different category product field group, but the present disclosure is not limited thereto. For example, when there are multiple different category products, the different category product field group is provided according to the number of different category products. In addition, the third bit of the additional information added before the different category product field group is "1". In addition, FIG. 9 shows an example in which the different category product field group is included in each of field group #1 and field group #n, but the present disclosure is not limited thereto. For example, as in the example of FIG. 8A, when there are no different category products, the field group may not include the different category product field group. In addition, FIG. 9 shows an example in which each field group includes a product ID field following the different category product field group, but the present disclosure is not limited thereto.

[0134] Also, in FIG. 9, an example in which there is one same-category product field group is shown, but the present disclosure is not limited to this. For example, when there are multiple products in the same category, the same-category product field group is provided according to the number of products in the same category. Note that the third bit of the additional information added before the same-category product field group is "0". Also, in FIG. 9, an example in which the same-category product field group is included in each of field group #1 and field group #n is shown, but the present disclosure is not limited to this. There may be cases in which the same-category product field group does not exist in a field group. Also, in FIG. 9, an example in which a product ID field exists following the same-category product field group in each field group is shown, but the present disclosure is not limited to this.

[0135] When a second candidate product for a certain identified product is a product in the same category as the first candidate product, there is no difference in the price paid by the customer whether the identified product is the first candidate product or the second candidate product. Therefore, information about products in the same category may be omitted, compressed, thinned out, etc.

[0136] For example, when there are multiple second candidate products in the same category as the first candidate product, information on the multiple second candidate products in the same category may be grouped into one same-category product field group. When grouped into one same-category product field group, the multiple same-category products may have different product IDs.

[0137] When multiple products are grouped into one same-category product field group, the product ID field of the same-category product field group includes a dedicated ID to indicate that multiple products of the same category are grouped together. This dedicated ID is different from the product IDs of the products of the same category. For example, this dedicated ID is different from the product IDs of each of the candidate products.

[0138] When the products are grouped into one same-category product field group, the quantity information field of the same-category product field group contains information indicating the number of products of the same category.

[0139] When the same-category product fields are grouped into one same-category product field group, the classification probability field of the same-category product field group contains information indicating a probability based on the classification probability of each of the multiple same-category products. The probability based on the classification probability of each of the multiple same-category products may be, for example, a sum or average value of the classification probabilities of each of the multiple same-category products, or a maximum or minimum value of the classification probabilities of each of the same-category products.

[0140] In this way, by consolidating information on a plurality of products of the same category into one same-category product field group, the information on a plurality of products of the same category can be compressed, and the amount of information in the notification information can be reduced.

[0141] In a second example, as information indicating whether a first candidate product and a second candidate product are in the same category for a certain identification target product, information on a second candidate product in the same category as the first candidate product (e.g., information on a same category product field group) and information on a second candidate product in a different category from the first candidate product (e.g., information on a different category information field group) are notified. By having the store staff check this notification, the store staff can confirm the priority of the work.

[0142] For example, in a case where there is a second candidate product in a category different from that of the first candidate product for a certain identification target product X, if the identification target product X is actually the second candidate product, but is determined to be the first candidate product based on the identification probability, a difference will occur between the amount paid by the customer and the actual price of the product. In such a case, by having the staff prioritize checking the identified product X by visual inspection or the like, it is possible to avoid a difference between the amount paid by the customer and the actual price of the product. As a result, it is possible to avoid cases where customers are inconvenienced by paying more than the purchase amount, or where store profits are reduced by customers paying less than the purchase amount.

[0143] For example, in a case where there is no second candidate product in a category different from that of the first candidate product for a certain identification target product X, but there is a second candidate product in the same category, if the identification target product X is actually the second candidate product, but is determined to be the first candidate product based on the identification probability, there will be no difference between the price paid by the customer and the actual price of the product. In such a case, the staff does not need to prioritize visual confirmation of the identification target product X, thereby reducing the burden on the staff.

[0144] <Display example of the second example> Fig. 10A is a diagram showing a second example of a display displayed on the information processing device 20 in this embodiment. Fig. 10A is, for example, a display example when a classification result of the image classification process as shown in Fig. 8A is notified.

[0145] 10A shows an image of a classification target product X, an image of a first candidate product #2, and an image of a second candidate product #3 in the same category as the first candidate product, side by side. The classification probability of each product is also shown.

[0146] In addition, when there are multiple second candidate products in the same category as the first candidate product, and multiple same category products are grouped into one same category product field group in the notification information, images and identification probabilities of the second candidate products in the same category as the first candidate product do not have to be displayed. In this case, the existence of the second candidate products in the same category as the first candidate product may be displayed by a message or the like.

[0147] Fig. 10B is a diagram showing a variation of the second example of the display displayed on the information processing device 20 in this embodiment. Fig. 10B is, for example, a display example when the classification result of the image classification process as shown in Fig. 8B is notified.

[0148] FIG. 10B shows an image of product X to be classified, an image of product #2 which is a first candidate product, an image of product #3 which is a second candidate product in the same category as the first candidate product, and an image of product #1 which is a second candidate product not in the same category as the first candidate product, all arranged side by side. The classification probability of each product is also shown. In the case of FIG. 10B, product #2 and product #1 are not in the same category. In this case, a message is displayed to the store staff encouraging them to visually check the product.

[0149] In the second example, as shown in Figs. 10A and 10B, whether the category of the second candidate product is the same as that of the first candidate product is displayed. If the categories are different, the category of the second candidate product is also displayed. Note that the display mode (e.g., color, size, and blinking of characters) when the price is different may be different from the other display modes. By changing the display mode, the fact that the categories are different is visually emphasized to a person checking the display.

[0150] Furthermore, if the recognition result satisfies a certain judgment condition, a message is displayed requesting that the store staff visually confirm the product to be purchased (i.e., the product to be recognized).

[0151] For example, if the classification result satisfies the condition that the classification probability of a second candidate product, which is in a different category from the first candidate product, is equal to or greater than a second threshold, a message is displayed requesting the store staff to visually confirm the product to be purchased (i.e., the product to be classified). Note that the certain judgment condition is not limited to this example.

[0152] As described above, in the second example, the terminal 10 notifies the information processing device 20 of information indicating the relationship between the category of the first candidate product and the category of the second candidate product, and thereby the store staff can check, on the information processing device 20, whether or not there is a difference in the amount paid by the customer when the product to be identified is the first candidate product and when it is the second candidate product. Therefore, the store staff can omit the work required for checking when the identification result of the image identification process is not good, and the workload of the store staff can be reduced.

[0153] That is, in the second example, if the first candidate product category and the second candidate product category belong to the same category, a message prompting the store staff to check is not displayed, and if the first candidate product category and the second candidate product category belong to different categories, a message prompting the store staff to check is displayed.

[0154] When the range of misrecognition of a product falls within the same category, the range of misrecognition is limited to differences in type, such as flavor, so the price, etc. is unlikely to change regardless of whether the first or second candidate product is the correct product to be identified. In other words, regardless of whether the recognition result is correct or not, there is little chance that the customer will be charged an incorrect amount even if confirmation by the store staff is omitted. Therefore, in the second example, when the misrecognition of a product falls within the range of the same category, the message prompting confirmation by the store staff is omitted.

[0155] On the other hand, if the misrecognition of a product spans multiple categories, there is a high possibility that products with different prices, etc. have been misrecognized. Therefore, there is a high possibility that the amount charged to the customer will change depending on whether the first or second candidate product was the correct product to be identified. Therefore, in the second example, if the misrecognition of a product spans multiple categories, a message is displayed urging the store staff to check.

[0156] In the second example, the categories for classifying products as the same category products may be limited to categories with the same price. For example, if the price varies depending on the type of taste, aroma, etc., even if the category is the same, the category may not be included in the categories for classifying products as the same category products.

[0157] In a second example, the terminal 10 determines whether or not to prompt the user to confirm the product to be identified based on the relationship between the category of the first candidate product and the category of the second candidate product. For example, if the category of the first candidate product and the category of the second candidate product are not the same, it is determined that the user should confirm the product to be identified. Then, when the terminal 10 determines whether or not to prompt the user to confirm the product to be identified, it transmits an instruction (for example, notification information) to prompt the user to confirm the product to be identified.

[0158] Here, the instruction (e.g., notification information) to prompt confirmation of the product to be identified may correspond to information (e.g., different category product field groups) indicating that the category of the first candidate product and the category of the second candidate product are different. In the above example, notification information is also transmitted when the category of the first candidate product and the category of the second candidate product are the same, but when the category of the first candidate product and the category of the second candidate product are the same, the notification information does not include different category product field groups. Therefore, the display based on the notification information does not display a message to prompt confirmation of the product to be identified because the categories are different. Therefore, the notification information when the category of the first candidate product and the category of the second candidate product are the same does not need to correspond to an instruction to prompt confirmation of the product to be identified.

[0159] In the second example, the terminal 10 may determine not to prompt the user to confirm the identification target product when the category of the first candidate product and the category of the second candidate product are the same category.

[0160] In the second example, the terminal 10 changes the notification information depending on whether the category of the first candidate product and the category of the second candidate product are the same or different from each other.

[0161] <Third Example> In the third example, a combination of the first and second examples will be described. For example, in the third example, information indicating the relationship between the price of the first candidate product and the price of the second candidate product, and information indicating the relationship between the category of the first candidate product and the category of the second candidate product are notified.

[0162] Fig. 11A is a diagram showing a third example in the present embodiment. In the example shown in Fig. 11A, the highest classification probability Px_2 is lower than the first threshold. Px_2, Px_3, and Px_1 are higher than the second threshold. Fig. 11A also shows the price, category, and flavor (orange, apple, etc.) of each candidate product.

[0163] In the example shown in FIG. 11A, the category of the product #2 corresponding to Px_2 is the same as the category of the product #3 corresponding to Px_3. Also, the price of the product #2 corresponding to Px_2 is the same as the price of the product #1 corresponding to Px_1. In this way, if the category of the product #2, which is the first candidate product, is the same as the category of the product #3, which is the second candidate product, even if the product X to be identified is actually the second candidate product but is determined to be the first candidate product based on the classification probability, there will be no difference in the amount paid by the customer. Also, in this way, if the price of the product #2, which is the first candidate product, is the same as the price of the product #1, which is the second candidate product, even if the product X to be identified is actually the second candidate product but is determined to be the first candidate product based on the classification probability, there will be no difference in the amount paid by the customer.

[0164] Fig. 11B is a diagram showing a variation of the third example in the present embodiment. In the example shown in Fig. 11B, the highest classification probability Px_2 is lower than the first threshold. Px_2, Px_3, Px_1, and Px_4 are higher than the second threshold. Fig. 11B also shows the category of each candidate product and the flavor of each candidate product (orange, apple, etc.).

[0165] 11B, the category and price of product #2 corresponding to Px_2 are different from the category and price of product #4 corresponding to Px_4. In this case where the category of product #2, which is the first candidate product, is different from the category of product #4, which is the second candidate product, and the price of product #2, which is the first candidate product, is different from the price of product #4, which is the second candidate product, there is a possibility that a difference will occur in the amount paid by the customer if product X to be identified is actually the second candidate product but is determined to be the first candidate product based on the identification probability.

[0166] Therefore, in the third example, whether or not the category of the first candidate product is the same as the category of the second candidate product, and whether or not the price of the first candidate product is the same as the price of the second candidate product are notified to the information processing device 20. The first candidate product is product #2 in the examples of Figures 11A and 11B, and the second candidate product is products #1 and #3 in the example of Figure 11A, and products #1, #3, and #4 in the example of Figure 11B.

[0167] <Example of notification information for the third example> Fig. 12 is a diagram showing a third example of the format of the notification information notified in the present embodiment. Fig. 12 shows a format having a plurality of fields. Note that in Fig. 12, the description of the same fields as in Fig. 6 and Fig. 9 is omitted.

[0168] In the format shown in Fig. 12, a field group having a plurality of fields is provided for one type of identification target product to be purchased, similarly to Fig. 6 and Fig. 9. One field group is associated with one identification target product. For example, field group #k is associated with identification target product #k.

[0169] Below, the field group #1 will be described as an example, but the other field groups #2 to #n may be similar to the field group #1.

[0170] Field group #1 includes a "first candidate product field group," a "same category product field group," a "same price product field group," and a "other product field group." Note that for the sake of convenience, Fig. 12 shows the "first candidate product field group," the "same price product field group," the "different price product field group," and the "other product field group," but in the notification information, these field groups are distinguished by the contents of the additional information.

[0171] Field group #1 includes an "additional information" field, a "product ID" field, a "quantity information" field, and a "recognition probability" field.

[0172] The first candidate product field group includes information about the first candidate product, similar to the examples of Figures 6 and 9. For example, in the examples of Figures 11A and 11B, information about product #2 is included.

[0173] The same category product field group includes information about the second candidate product that is in the same category as the first candidate product, as in the example of Fig. 9. For example, in the example of Fig. 11A and Fig. 11B, information about product #3 is included.

[0174] The same priced product field group includes information about the second candidate product that is the same price as the first candidate product, as in the example of Fig. 6. For example, in the example of Fig. 11A and Fig. 11B, information about product #1 is included.

[0175] The additional information field contains, for example, 4 bits of information.

[0176] Of the four bits in the additional information field, the first bit indicates whether or not there is another field (e.g., a product ID field, a quantity information field) following the field. For example, if there is no other field following, the first bit is set to "0", and if there is another field following, the first bit is set to "1".

[0177] Of the four bits in the additional information field, the second bit indicates whether the product ID in the following product ID field is the first candidate product. For example, if the product ID in the following product ID field is the first candidate product, the second bit is set to "0," and if the product ID in the following product ID field is not the first candidate product, the second bit is set to "1."

[0178] Of the four bits in the additional information field, the third bit indicates whether the product ID in the subsequent product ID field is the same price as the first candidate product. For example, if the price of the product with the subsequent product ID is the same as the first candidate product, the third bit is set to "0," and if the price of the product with the subsequent product ID is not the same as the first candidate product, the third bit is set to "1."

[0179] Of the four bits in the additional information field, the fourth bit indicates whether the product ID in the following product ID field is in the same category as the first candidate product. For example, if the category of the second candidate product in the following product ID is the same as the first candidate product, the third bit is set to "0," and if the category of the second candidate product in the following product ID is not the same as the first candidate product, the third bit is set to "1."

[0180] The other product field group in Fig. 12 includes information about second candidate products that are different in category and price from the first candidate product, for example, information about product #4 in the example of Fig. 11B.

[0181] The product ID field of the other product field group includes the product ID of the second candidate product that is in a category and price different from the category and price of the first candidate product. For example, in the example of FIG. 11, the product ID of product #4 is included.

[0182] The quantity information field of the other product field group is provided in association with the product ID field of the other product field group. The quantity information field includes information indicating the number of products having the product ID indicated by the corresponding product ID field.

[0183] The classification probability field of the other product field group is provided in association with the product ID field of the other product field group. The classification probability field includes information indicating the classification probability of the product of the product ID indicated by the corresponding product ID field. For example, in the example of FIG. 11, the classification probability field includes the classification probability Px_4 of product #4.

[0184] As in the cases of FIGS. 6 and 9, the number of each field group is not limited to the example shown in FIG.

[0185] <Display example of the third example> Fig. 13A is a diagram showing a third example of a display displayed on the information processing device 20 in this embodiment. Fig. 13A is, by way of example, a display example in the case where a classification result of the image classification process as shown in Fig. 11A is notified.

[0186] In Fig. 13A, an image of the product X to be classified, an image of the first candidate product #2, an image of the second candidate product #3 which is in the same category as the first candidate product, and an image of the second candidate product #1 which is the same price as the first candidate product are shown side by side. The classification probability of each product is also shown. Fig. 13A also shows the category and price of each product.

[0187] Fig. 13B is a diagram showing a variation of the third example of the display displayed on the information processing device 20 in this embodiment. Fig. 13B is, for example, a display example when the classification result of the image classification process as shown in Fig. 11B is notified.

[0188] FIG. 13B shows an image of product X to be identified, an image of product #2 which is a first candidate product, an image of product #3 which is a second candidate product in the same category as the first candidate product, an image of product #1 which is a second candidate product with the same price as the first candidate product, and an image of product #4 which is a second candidate product not in the same category or price as the first candidate product, all arranged side by side. The identification probability of each product is also shown. In the case of FIG. 13B, product #2 and product #4 are not in the same category or price. In this case, a message is displayed to encourage the store staff to visually check.

[0189] In the third example, as shown in FIG. 13A and FIG. 13B, whether the price of the second candidate product is the same as that of the first candidate product and whether the category of the second candidate product is the same as that of the first candidate product are displayed. If the category and price are different, the category and / or price of the second candidate product are also displayed. Note that the display mode (e.g., color, size, blinking, etc. of the characters) when the category and / or price are different may be different from the other display modes. By changing the display mode, the fact that the categories are different is visually emphasized to a person checking the display.

[0190] Furthermore, if the recognition result satisfies a certain judgment condition, a message is displayed requesting that the store staff visually confirm the product to be purchased (i.e., the product to be recognized).

[0191] For example, if the classification result satisfies a condition that the classification probability of a second candidate product that is different in category and / or price from the first candidate product is equal to or greater than a predetermined value, a message is displayed requesting the store staff to visually confirm the product to be purchased (i.e., the product to be classified). Note that the certain judgment condition is not limited to this example.

[0192] As described above, in the third example, the terminal 10 notifies the information processing device 20 of information indicating the relationship between the price of the first candidate product and the price of the second candidate product, and information indicating the relationship between the category of the first candidate product and the category of the second candidate product, thereby allowing the store staff to check, on the information processing device 20, whether or not there is a difference in the amount paid by the customer when the product to be identified is the first candidate product and when it is the second candidate product. Therefore, the store staff can omit the work required for checking when the identification result of the image identification process is not good, and the workload of the store staff can be reduced.

[0193] In a third example, the terminal 10 determines whether or not to prompt the user to confirm the identification target product based on the relationship between the first candidate product and the second candidate product. For example, the relationship between the first candidate product and the second candidate product may be the relationship between the price of the first candidate product and the price of the second candidate product as in the first example, or the relationship between the category of the first candidate product and the category of the second candidate product as in the second example. Then, when the terminal 10 determines whether or not to prompt the user to confirm the identification target product, it transmits an instruction (for example, notification information) to prompt the user to confirm the identification target product.

[0194] In the above-described embodiment, the terminal 10 may determine not to prompt confirmation of the recognition target product, regardless of the relationship between the first candidate product and the second candidate product, when the highest recognition probability is equal to or greater than the first threshold value.

[0195] Furthermore, in the above-described embodiment, the terminal 10 notifies the user of the notification information including the identification probability of the first candidate commodity and the identification probability of the second candidate commodity.

[0196] In the above-described embodiment, the terminal 10 transmits a signal including notification information to the information processing device 20 (an example of a display device).

[0197] In the above-described embodiment, the information processing device 20 performs the display, but the terminal 10 may perform the display. In this case, the terminal 10 outputs a signal including the notification information to the display unit of the terminal 10.

[0198] In the above-described embodiment, if the identification probability of the first candidate product is less than the first threshold, an alarm may be output to the store staff.

[0199] In the above-described embodiment, if the second candidate products include products with the same price as the first candidate products and products with a different price than the first candidate products, a message may be displayed to encourage the store staff to visually check the products. Also, if the second candidate products include products in the same category as the first candidate products and products in a different category than the first candidate products, a message may be displayed to encourage the store staff to visually check the products.

[0200] In the above-described embodiment, the message to the store staff may be changed according to the difference between the price of the second candidate product and the price of the first candidate product. For example, when the price difference is equal to or greater than a predetermined amount, a message to encourage visual confirmation may be displayed to the store staff, and when the price difference is less than the predetermined amount, the message to encourage visual confirmation may not be displayed. Alternatively, a message to encourage visual confirmation may be displayed both when the price difference is equal to or greater than the predetermined amount and when it is less than the predetermined amount. In this case, the content of the message may be changed depending on whether the price difference is equal to or greater than the predetermined amount or not. For example, when the price difference is less than the predetermined amount, a message to encourage visual confirmation may be displayed if the clerk has time and work schedules available. In addition, when the price difference is large, a message to encourage visual confirmation may be displayed regardless of the clerk's schedule, when the price difference is medium, the display of the message may be switched depending on the clerk's schedule, and when the price difference is small, the message may not be displayed.

[0201] In the above embodiment, the information processing device 20 displays a message to the store staff to prompt them to make a visual check, but the present disclosure is not limited to this. Any work may be instructed to the store staff as long as it is a necessary work to accurately settle the identification target product that is likely to be misrecognized. Specifically, it is possible to instruct the customer to make a manual payment. In addition, the person to whom the notification is sent is not limited to the store staff. For example, a message may be displayed on the display unit of the terminal 10 to the customer using the terminal 10 to prompt the customer to reprocess the image recognition process of the product. This allows the accuracy of automatic settlement to be improved without the store staff having to check. For example, the number of times the message is displayed to the customer may be set to a predetermined number of times that does not make the customer feel uncomfortable. Note that if the recognition probability does not improve even if the message is displayed to the customer a predetermined number of times, a message may be displayed to the store staff to prompt them to make a visual check. In this case, a message to prompt them to make a visual check may be displayed only when the recognition probability is not expected to improve. An example of a case where the recognition probability is not expected to improve is, for example, a case where the amount of improvement in the recognition probability does not exceed a predetermined threshold even if the message is displayed a predetermined number of times or more.

[0202] Furthermore, in the above-described embodiment, a customer and / or a store staff member may select to change from automatic payment using a cart to payment at the register, where the staff member makes the payment at the register. For example, when the identification probability of the first candidate product is less than the first threshold and an alarm is output, a customer and / or a store staff member who sees the alarm may select payment at the register. When payment is changed to payment at the register, a message prompting visual confirmation may not be displayed.

[0203] Furthermore, in the above-described embodiment, the presence or absence of a message urging a visual check may be changed depending on the congestion status of the store (e.g., the number of people around the cart in question). For example, when the store is crowded (e.g., when the density of people is above a predetermined level), it is difficult for staff to rush to the location of the target customer, so the message urging a visual check may not be displayed. In this case, the target customer may be moved to a predetermined location, and a staff member may rush to the predetermined location.

[0204] <Example of processing flow> Next, a description will be given of the process flow of the terminal 10 and the information processing device 20 in the present embodiment. Note that the process flow described below corresponds, for example, to the above-mentioned third example.

[0205] Fig. 14 is a flowchart showing an example of a process flow of the terminal 10 according to the present embodiment. The flow shown in Fig. 14 may be started, for example, when the terminal 10 is started up or when the customer starts using the terminal 10.

[0206] The terminal 10 judges whether it is time for payment (S101). The time for payment may be, for example, the time when the customer inputs an instruction to the terminal 10 to make a payment.

[0207] If it is not the time for payment (NO in S101), the flow returns to S101, and the terminal 10 waits for the time for payment.

[0208] If it is time for payment (YES in S101), the terminal 10 performs image recognition processing on the recognition target product (S102), thereby calculating the recognition probability of the candidate product for each recognition target product.

[0209] The terminal 10 starts classifying the classification results (S103). The terminal 10 compares the classification probability of each of the candidate products for one classification target product, and determines the first candidate product corresponding to the highest classification probability. If the highest classification probability is equal to or greater than the first threshold, the terminal 10 determines that the classification target product corresponds to the first candidate product. If the highest classification probability is not equal to or greater than the first threshold, the image classification unit 102 determines a classification probability that is equal to or greater than the second threshold among the second and subsequent highest classification probabilities, and determines a second candidate product corresponding to the determined classification probability. Then, the terminal 10 classifies the classification results for the classification target product based on the relationship between the first candidate product and the second candidate product. Below, the processes of S104 to S108 are classification processes for one classification target product. If there are multiple classification target products, the processes of S104 to S108 are performed for each of the classification target products. Furthermore, when there are a plurality of second candidate products for one recognition target product, the processes of S104 to S108 are performed for each of the second candidate products.

[0210] The terminal 10 determines whether the second candidate product is the same price as the first candidate product (S104).

[0211] If the second candidate product is the same price as the first candidate product (YES in S104), the terminal 10 determines whether the second candidate product is in the same category as the first candidate product (S105).

[0212] If the second candidate product is in the same category as the first candidate product (YES in S105), the terminal 10 classifies the second candidate product as being in the same category as the first candidate product (S106). Note that, if there are multiple second candidate products classified as being in the same category, the numbers and / or classification probabilities may be combined.

[0213] If the second candidate product is not in the same category as the first candidate product (NO in S105), the terminal 10 classifies the second candidate product as being of the same price (S107). Note that, if there are multiple second candidate products classified as being of the same price, the numbers and / or classification probabilities may be combined.

[0214] If the second candidate product is not the same price as the first candidate product (NO in S104), the terminal 10 classifies the second candidate product as a product with a different price from the first candidate product (S108).

[0215] The terminal 10 generates notification information based on the results of S106 to S108, and transmits the notification information (S109). Then, the flow in FIG. 14 ends.

[0216] Fig. 15 is a flowchart showing an example of the flow of processing of the information processing device 20 according to this embodiment. The flow shown in Fig. 15 may be started, for example, when the information processing device 20 is started.

[0217] The information processing device 20 determines whether or not notification information has been received from the terminal 10 (S201).

[0218] When no notification information is received from the terminal 10 (NO in S201), the flow returns to S201, and the information processing device 20 waits for notification information.

[0219] When notification information is received from the terminal 10 (YES in S201), the information processing device 20 analyzes the notification information (S202).

[0220] The information processing device 20 determines, as a result of the analysis, whether or not there is a second candidate product that is in the same category as the first candidate product (S203).

[0221] If there is a second candidate product that is in the same category as the first candidate product (YES in S203), the information processing device 20 performs display in a display manner indicating that there is a product in the same category (S204). Then, the flow in FIG. 15 ends.

[0222] If there is no second candidate product in the same category as the first candidate product (NO in S203), the information processing device 20 determines whether there is a second candidate product of the same price as the first candidate product (S205).

[0223] If there is a second candidate product that is the same price as the first candidate product (YES in S205), the information processing device 20 performs display in a display manner indicating that there is a product with the same price (S206). Then, the flow in FIG. 15 ends.

[0224] If there is no second candidate product that is the same price as the first candidate product (NO in S205), the information processing device 20 displays in a display manner indicating that there are no products of the same price or category (S207). Then, the flow in FIG. 15 ends.

[0225] <Example of device configuration> Fig. 16 is a diagram showing an example of the configuration of terminal 10 according to the present embodiment. Terminal 10 shown in Fig. 16 has storage unit 101, image identification unit 102, image identification result classification unit 103, notification information generation unit 104, and communication unit 105. One or more of image identification unit 102, image identification result classification unit 103, and notification information generation unit 104 may be collectively referred to as a control unit or a processing unit. Furthermore, terminal 10 may be referred to as an information processing device.

[0226] Information about candidate products is pre-stored in the storage unit 101. The information about candidate products includes images of the candidate products, product IDs of the candidate products, categories of the candidate products, types of the candidate products, and prices of the candidate products.

[0227] The image classification unit 102 acquires an image of a product to be classified. The image of the product to be classified is acquired, for example, from a camera (not shown in FIG. 16) mounted on the terminal 10, or a camera (not shown in FIG. 16) connected to the terminal 10 by wire or wirelessly. The image classification unit 102 calculates a classification probability indicating a similarity between the image of the product to be classified and the image of the candidate product based on the image of the product to be classified and the image of the candidate product stored in the storage unit 101. The image classification unit 102 compares the classification probabilities of the candidate products for one product to be classified, and determines a first candidate product corresponding to the highest classification probability. If the highest classification probability is equal to or greater than a first threshold, the image classification unit 102 determines that the product to be classified corresponds to the first candidate product. If the highest classification probability is not equal to or greater than the first threshold, the image classification unit 102 determines a classification probability equal to or greater than a second threshold among the second and subsequent highest classification probabilities, and determines a second candidate product corresponding to the determined classification probability.

[0228] The image recognition result classification unit 103 determines whether or not to prompt the user to confirm the recognition target product based on the relationship between the first candidate product and the second candidate product for the recognition target product. For example, the image recognition result classification unit 103 classifies the recognition result of the second candidate product for the first candidate product based on the relationship between the price of the first candidate product and the price of the second candidate product, and / or the relationship between the category of the first candidate product and the category of the second candidate product. Then, in the classification, if there is a second candidate product that satisfies a predetermined condition, the image recognition result classification unit 103 determines to prompt the user to confirm the recognition target product, and if there is no second candidate product that satisfies the predetermined condition, the image recognition result classification unit 103 determines not to prompt the user to confirm the recognition target product. For example, if there is a second candidate product with a price different from that of the first candidate product and / or if there is a second candidate product in a category different from that of the first candidate product, it may be determined to prompt the user to confirm the recognition target product. In addition, in cases where there are two or more second candidate products, if the price of the first candidate product differs from the price of at least one second candidate product and / or if the category of the first candidate product differs from the category of at least one second candidate product, it may be decided to prompt the user to confirm the product to be identified.

[0229] The notification information generating unit 104 generates notification information according to the classification result in accordance with the notification information format described above. For example, the notification information indicating the presence of a second candidate product that satisfies the above-mentioned predetermined condition corresponds to an instruction to prompt confirmation of the product to be identified.

[0230] The communication unit 105 transmits the notification information to the information processing device 20.

[0231] In the above-mentioned terminal 10, the image recognition unit 102 (an example of an acquisition unit) acquires an image of a product to be recognized in image recognition. Then, the image recognition unit 102 and the image recognition result classification unit 103 (an example of a control unit) determine the highest recognition probability and the second and subsequent recognition probabilities among the recognition probabilities (an example of similarity) between the image of the product to be recognized and each of the images of the registered candidate products, and determine whether or not to prompt confirmation of the product to be recognized based on the relationship between the first candidate product corresponding to the highest recognition probability and the second candidate product corresponding to the second and subsequent recognition probabilities. Then, the communication unit 105 (an example of an output unit) transmits an instruction to prompt confirmation of the product to be recognized.

[0232] 17 is a diagram showing an example of the configuration of an information processing device 20 according to this embodiment. The information processing device 20 has a communication unit 201, a discrimination result analysis unit 202, a display method selection unit 203, and a display unit 204. One or more of the discrimination result analysis unit 202 and the display method selection unit 203 may be collectively referred to as a control unit or a processing unit.

[0233] The communication unit 201 receives notification information from the terminal 10 .

[0234] The identification result analysis unit 202 analyzes the identification result included in the received notification information. For example, the identification result analysis unit 202 analyzes whether the price of the first candidate product is the same as the price of the second candidate product and / or whether the category of the first candidate product is the same as the category of the second candidate product.

[0235] The display method selection unit 203 selects a display method according to the analysis result of the image recognition result analysis unit 202. For example, the display method selection unit 203 selects information to be displayed (price, category, recognition probability) or sets the display mode of the information to be displayed according to whether or not the price of the first candidate product is the same as the price of the second candidate product and / or whether or not the category of the first candidate product is the same as the category of the second candidate product. For example, when the condition that the recognition probability of the second candidate product, which is different in category and / or price from the first candidate product, is equal to or higher than a predetermined value is satisfied, the display method selection unit 203 selects a display method to display a message requesting the store staff to visually confirm the product to be purchased (i.e., the product to be recognized).

[0236] The display unit 204 displays the information according to the display method selected (or set) by the display method selection unit 203 .

[0237] 16 and 17 show an example in which the terminal 10 performs the image recognition process, but the present disclosure is not limited thereto. For example, the information processing device 20 may perform the image recognition process. In this case, the terminal 10 acquires an image of the product to be recognized, and transmits the acquired image of the product to be recognized to the information processing device 20. In this case, the storage unit 101, the image recognition unit 102, and the image recognition result classification unit 103 may be included in the information processing device 20. In this case, since notification information is not necessary, the notification information generation unit 104 is omitted.

[0238] In the above embodiment, the information processing device 20 is provided at a monitoring location where the store staff waits and displays information for the store staff, but the present disclosure is not limited to this. For example, the information processing device 20 may transmit information for the store staff to a terminal owned by the store staff, and the terminal owned by the store staff may display the information. Alternatively, the information processing device 20 may be owned and carried by the store staff.

[0239] Furthermore, in the above embodiment, the notification given to the store staff or customers is not limited to a displayed message, but may be a sound or the like.

[0240] The above describes each embodiment.

[0241] It should be noted that the "... section" in the above embodiments may be a "... circuit", a "... device", a "... unit", or a "... module".

[0242] The present disclosure can be realized in software, hardware, or software in conjunction with hardware.

[0243] Each functional block used in the description of the above embodiments may be partially or entirely realized as an LSI, which is an integrated circuit, and each process described in the above embodiments may be partially or entirely controlled by one LSI or a combination of LSIs. The LSI may be composed of individual chips, or may be composed of one chip that includes some or all of the functional blocks. The LSI may have data input and output. Depending on the degree of integration, the LSI may be called an IC, a system LSI, a super LSI, or an ultra LSI.

[0244] The method of integration is not limited to LSI, and may be realized by a dedicated circuit, a general-purpose processor, or a dedicated processor. Also, after LSI manufacturing, a programmable FPGA (Field Programmable Gate Array) or a reconfigurable processor that can reconfigure the connections and settings of circuit cells inside the LSI may be used. The present disclosure may be realized as digital processing or analog processing.

[0245] Furthermore, if a new integrated circuit technology that can replace LSI appears due to the progress of semiconductor technology or a derivative technology, it is possible to integrate the functional blocks using that technology. The application of biotechnology is also a possibility.

[0246] The present disclosure may be implemented in any type of apparatus, device, or system with communications capabilities (collectively, communications apparatus), including, but not limited to, telephones (e.g., cell phones, smartphones, etc.), tablets, personal computers (PCs) (e.g., laptops, desktops, notebooks, etc.), cameras (e.g., digital still / video cameras), digital players (e.g., digital audio / video players), wearable devices (e.g., wearable cameras, smart watches, tracking devices, etc.), game consoles, digital book readers, telehealth / telemedicine devices, communications-enabled vehicles or mobile conveyances (e.g., cars, planes, ships, etc.), and combinations of the above-mentioned devices.

[0247] Communications Equipment includes, but is not limited to portable or mobile equipment, non-portable or fixed equipment, devices and systems of any kind, such as smart home devices (appliances, lighting equipment, smart meters or metering devices, control panels, etc.), vending machines and any other "Things" that may exist on an Internet of Things (IoT) network.

[0248] Communications include data communications via cellular systems, wireless LAN systems, communications satellite systems, etc., as well as data communications via combinations of these.

[0249] A communications apparatus also includes devices, such as controllers and sensors, connected or coupled to a communications device that performs the communications functions described in this disclosure, such as controllers and sensors that generate control and data signals used by the communications device to perform the communications functions of the communications apparatus.

[0250] The communication apparatus also includes infrastructure facilities, such as base stations, access points, and any other apparatus, device, or system that communicates with or controls the various apparatuses listed above, but are not limited to these.

[0251] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can come up with various modified or amended examples within the scope of the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure. In addition, the components in the above-mentioned embodiments may be arbitrarily combined within the scope of the disclosure.

[0252] Although specific examples of the present disclosure have been described in detail above, these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and alterations of the specific examples exemplified above. [Industrial Applicability]

[0253] The present disclosure is suitable for a wireless communication system. [Explanation of symbols]

[0254] 10 Terminal 20 Information processing device 101 Storage section 102 Image Identification Unit 103 Image Recognition Result Classification Unit 104 Notification information generation unit 105, 201 Communications Department 202 Image Recognition Result Analysis Unit 203 Display method selection section 204 Display section

Claims

1. An acquisition unit that acquires an image of a product to be identified by image recognition; a control unit that determines the highest similarity and the second and subsequent similarities among similarities between the image of the product to be identified and each of the registered candidate images of the product to be identified, and determines whether or not to prompt confirmation of the product based on a relationship between a first candidate corresponding to the highest similarity and a second candidate corresponding to the second and subsequent similarities; an output unit that outputs an instruction to prompt the user to confirm the product when it is determined whether or not to prompt the user to confirm the product; An information processing device comprising:

2. The control unit determines not to prompt the user to confirm the product when a difference between a price of the first candidate and a price of the second candidate is less than a predetermined value. The information processing device according to claim 1 .

3. the control unit changes an instruction to prompt the user to confirm the product depending on whether a price of the first candidate and a price of the second candidate are the same as a price of the first candidate and a price of the second candidate are different from a price of the first candidate. The information processing device according to claim 2 .

4. The control unit determines not to prompt confirmation of the product when a relationship between the first candidate and the second candidate is such that a category to which the first candidate belongs and a category to which the second candidate belongs are the same category. The information processing device according to claim 1 .

5. When the highest similarity is equal to or greater than a predetermined threshold, the control unit determines not to prompt confirmation of the product regardless of a relationship between the first candidate and the second candidate. The information processing device according to claim 1 .

6. the output unit outputs an instruction to notify the highest similarity and the second and subsequent similarities in addition to an instruction to prompt confirmation of the product. The information processing device according to claim 1 .

7. The output unit transmits a signal including an instruction to prompt the user to confirm the product to a display device that displays the instruction. The information processing device according to claim 1 .

8. A display unit is provided, The output unit outputs a signal including an instruction to prompt the user to confirm the product to the display unit. The information processing device according to claim 1 .

9. the control unit changes a message to be displayed depending on whether a relationship between the first candidate and the second candidate is such that a category of the first candidate is the same as a category of the second candidate or such that a category of the first candidate is the same as a category of the second candidate. The information processing device according to claim 1 .

10. An information processing device, Obtain an image of the product to be identified by image recognition; determining the highest similarity and the second and subsequent similarities between the image of the product to be classified and each of the registered candidate images of the product to be classified; determining whether or not to prompt the user to confirm the product based on a relationship between a first candidate corresponding to the highest similarity and a second candidate corresponding to the second or subsequent similarity; when it is determined whether or not to prompt the user to confirm the product, outputting an instruction to prompt the user to confirm the product; Information processing methods.

Images