Product candidate presentation device, electronic scale, product candidate presentation system, and product candidate presentation method

Abstract

To provide a commodity candidate presentation device, an electronic balance, a commodity candidate presentation system, and a commodity candidate presentation method which can reduce time and effort required for commodity selection.SOLUTION: A control unit 11 comprises: an acquisition unit 13 which acquires an image of a commodity on the basis of a captured image of the commodity; an estimation unit 14 which estimates commodity candidates from a commodity group on the basis of the image; a storage unit 12 in which commodities satisfying a prescribed condition in the commodity group are stored as specific commodities; and a presentation unit 15 which presents the commodity candidates estimated by the estimation unit 14 and the specific commodities stored in the storage unit 12.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to a product candidate presentation device, an electronic scale, a product candidate presentation system, and a product candidate presentation method.

Background Art

[0002] A product candidate presentation device includes an acquisition unit that acquires an image of a product, an estimation unit that estimates product candidates from a product group based on the image, and a presentation unit that presents the product candidates estimated by the estimation unit (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a product candidate presentation device, when a new product such as a new item (hereinafter referred to as a "new product") is registered, there may be no data corresponding to the new product in the learned model for estimating products, or the learning of the learned model may be insufficient. Therefore, it is difficult to estimate new products from the product group. In the product candidate presentation device, at least a part of the estimated product candidates is displayed in the order of the estimation results (the order in which the products are most likely). In this case, new products that are difficult to estimate from the product group are not displayed at the top in the estimation results. In the presentation unit, when a plurality of product selection screens are displayed or when the product selection screen is displayed by scrolling, the new product may be displayed on the last screen or below the scroll. Therefore, it has been troublesome to select new products.

[0005] An aspect of the present invention aims to provide a product candidate presentation device, an electronic scale, a product candidate presentation system, and a product candidate presentation method that can reduce the labor required for product selection. [Means for solving the problem]

[0006] A product candidate presentation device according to one aspect of the present invention is a product candidate presentation device that presents product candidates to a user for selection based on an image of a captured product, and comprises: an acquisition unit that acquires an image of a product; an estimation unit that estimates product candidates from a group of products based on the image; a storage unit that stores products in the group of products that satisfy predetermined conditions as specific products; and a presentation unit that presents the product candidates estimated by the estimation unit and the specific products stored in the storage unit.

[0007] In a product candidate presentation device according to one aspect of the present invention, a storage unit stores products that meet predetermined conditions in a group of products as specific products, and a presentation unit presents product candidates estimated by an estimation unit and specific products stored in the storage unit. In this configuration, for example, by storing a new product as a specific product in the storage unit, the specific product can be presented to the presentation unit in the same way as product candidates estimated by the estimation unit. Therefore, even if the product selection screen is displayed across multiple screens or the product selection screen is displayed by scrolling, it is possible to avoid displaying the new product on the last screen or at the bottom of the scroll. Thus, the product candidate presentation device can reduce the effort required to select a product.

[0008] In one embodiment, the estimation unit may estimate product candidates using a trained model. In a configuration where product candidates are estimated using a trained model, it is difficult for the estimation unit to estimate new products. Therefore, a configuration in which new products are stored as specific products and displayed on the display unit is particularly effective when product candidates are estimated using a trained model.

[0009] In one embodiment, the predetermined conditions may be at least one of the following: the product has been registered within a predetermined period of time; there is no pre-trained model corresponding to the product; and the product is designated as a specific product. New products fall under the cases of being registered within a predetermined period of time or not having a pre-trained model corresponding to the product. In addition, users may arbitrarily set a product as a specific product. Therefore, by setting the predetermined conditions to the above content, the product can be stored in the memory unit as a specific product.

[0010] In one embodiment, the storage unit may store the image acquired by the acquisition unit in association with the specific product when a specific product is selected in response to the presentation by the presentation unit. In this configuration, the data associating the image with the specific product can be used as training data for a trained model. This can improve the estimation accuracy of the specific product in the estimation unit.

[0011] In one embodiment, if the product group includes multiple specific products, the display unit may change the order in which the multiple specific products are displayed based on the estimation results from the estimation unit for each of the multiple specific products. In this configuration, multiple specific products can be displayed, for example, in the order in which they are most likely to be products.

[0012] In one embodiment, the display unit may display candidate products estimated by the estimation unit on the first screen, and the specific product stored in the storage unit may be displayed on a second screen different from the first screen. In this configuration, since the candidate products estimated by the estimation unit and the specific product are displayed on separate screens, the specific product can be confirmed at a glance.

[0013] In one embodiment, the display unit may display a specific product on the first screen if the likelihood of that product in the estimation result of the estimation unit falls within a predetermined rank in the product group. In this configuration, if there is a high probability that the product based on the image is the specific product, the specific product is displayed on the first screen instead of the second screen. This makes it easier to select the specific product.

[0014] An electronic scale according to one aspect of the present invention comprises the above-mentioned product candidate presentation device, a weighing unit for weighing the weight of a product, a selection unit for selecting a product from the candidates presented by the product candidate presentation device, and a calculation unit for calculating the price of a product based on the product selected by the selection unit and the weight of the product weighed by the weighing unit.

[0015] An electronic scale according to one aspect of the present invention is equipped with the above-mentioned product candidate presentation device. Therefore, the electronic scale can reduce the effort required for product selection.

[0016] A product candidate presentation system according to one aspect of the present invention is a product candidate presentation system that presents product candidates to a user for selection based on an image of a captured product, and comprises: an acquisition unit that acquires an image of a product; an estimation unit that estimates product candidates from a group of products based on the image; a storage unit that stores products in the group of products that satisfy predetermined conditions as specific products; and a presentation unit that presents the product candidates estimated by the estimation unit and the specific products stored in the storage unit.

[0017] In a product candidate presentation system according to one aspect of the present invention, a storage unit stores products that meet predetermined conditions in a group of products as specific products, and a presentation unit presents product candidates estimated by an estimation unit and specific products stored in the storage unit. In this configuration, for example, by storing a new product as a specific product in the storage unit, the specific product can be presented to the presentation unit in the same way as product candidates estimated by the estimation unit. Therefore, even if the product selection screen is displayed across multiple screens or the product selection screen is displayed by scrolling, it is possible to avoid displaying the new product on the last screen or at the bottom of the scroll. Thus, the product candidate presentation system can reduce the effort required for product selection.

[0018] A product candidate presentation method according to one aspect of the present invention is a product candidate presentation method for presenting product candidates to a user for selection based on an image of a captured product, including an acquisition step of acquiring an image of the product, an estimation step of estimating product candidates from a product group based on the image, a storage step of storing a product that satisfies a predetermined condition in the product group as a specific product, and a presentation step of presenting the product candidates estimated in the estimation step and the specific product stored in the storage step.

[0019] In the product candidate presentation method according to one aspect of the present invention, in the storage step, a product that satisfies a predetermined condition in the product group is stored as a specific product, and in the presentation step, the product candidates estimated by the estimation unit and the specific product stored in the storage step are presented. In this method, for example, by storing a new product as a specific product in the storage step, the specific product can be presented equivalently to the product candidates estimated in the estimation step. Therefore, even when a plurality of product selection screens are displayed or when the product selection screen is displayed by scrolling, it is possible to avoid the new product being displayed on the last screen or below the scroll. Therefore, in the product candidate presentation method, the labor required for product selection can be reduced.

Effect of the Invention

[0020] According to one aspect of the present invention, the labor required for product selection can be reduced.

Brief Description of the Drawings

[0021] [Figure 1] FIG. 1 is a schematic diagram showing an accounting processing system according to an embodiment. [Figure 2] FIG. 2 is a block diagram of a price determination device included in the accounting processing system. [Figure 3] FIG. 3 is an example of product candidates displayed on the display unit. [Figure 4] FIG. 4 is an example of product candidates displayed on the display unit. [Figure 5]FIG. 5 is an example of the price of a product displayed on the display unit. [Figure 6] FIG. 6 is a block diagram of the product candidate presentation system. [Figure 7] FIG. 7 is a diagram conceptually showing the neural network of the algorithm of the learning model. [Figure 8] FIG. 8 is a flowchart of the accounting process of the accounting system.

MODE FOR CARRYING OUT THE INVENTION

[0022] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or corresponding elements are denoted by the same reference numerals, and duplicate descriptions are omitted.

[0023] (1) Overall Outline The outlines of the product candidate presentation system 10 and the accounting system 40 will be described with reference to FIG. 1. FIG. 1 is a diagram schematically showing an accounting system 40 including a product candidate presentation system 10.

[0024] As shown in FIG. 1, the accounting system 40 includes a product candidate presentation system 10 and a price determination device 20. The product candidate presentation system 10 presents candidates for products to allow the user to select based on the captured image of the product. The accounting system 40 determines the price of the product selected by the user with the price determination device 20 from among the candidates presented by the product candidate presentation system 10.

[0025] The product candidate presentation system 10 includes a control unit (product candidate presentation device) 11. The control unit 11 is communicated with the price determination device 20 via a network NW. The network NW may be a LAN or a WAN such as the Internet. In another embodiment, some or all of the functions of the control unit 11 may be incorporated into the price determination device 20. In yet another embodiment, some or all of the functions of the control unit 11 may be incorporated into a cloud server. That is, some or all of the functional parts of the control unit 11 may be included in the cloud server. There may also be multiple cloud servers. Therefore, the functional parts of the control unit 11 may be distributed and included in multiple cloud servers.

[0026] The control unit 11 may also be connected to the store computer 100 via a network NW. The store computer 100 is a computer that manages various information about the product group sold or provided at the store where the accounting system 40 is used. The various information about the product group includes a product master in which product-related information is recorded for each product. The product master is, for example, a table to which product names, product numbers, unit prices, etc., are associated.

[0027] The accounting system 40 is not limited to any particular use, but it can be used, for example, in stores such as supermarkets. The products that are accounted for by the accounting system 40 are, for example, various types of products 200 such as prepared foods. The products that are accounted for by the accounting system 40 (the candidate products presented by the product candidate suggestion system 10) may be food products such as bread and vegetables, or non-food products.

[0028] (2) Price determination device The price determination device 20, which constitutes the accounting processing system 40, will be described with reference to Figures 1 to 5. Figure 2 is a block diagram of the price determination device 20. Figures 3 and 4 are examples of the display of product candidates presented by the product candidate suggestion system 10 on the display unit 26. Figure 5 is an example of the display of product prices on the display unit 26.

[0029] The price determination device 20 is installed in the sales area of ​​the product. The price determination device 20 is communicated via a network NW to the control unit 11 and the store computer 100 of the product candidate presentation system 10. The price determination device 20 receives candidate products 200 placed on the weighing platform 24a from the control unit 11 via the network NW. The price determination device 20 (weighing unit 24) has the function of weighing the weight of the product 200 placed on the weighing platform 24a. The price determination device 20 determines the price of the product 200 based on the candidate products transmitted from the control unit 11, the unit price of the products obtained from the store computer 100, and the weight of the product 200 measured by the weighing unit 24.

[0030] As shown in Figure 1, the price determination device 20 comprises a housing 21, an imaging unit 22, a light source 23, a weighing unit 24, a fixed key 25, a display unit 26, and a control unit 27. As shown in Figure 2, the control unit 27 includes a control unit 28, a selection unit 29, a storage unit 30, and a calculation unit 31. The display unit 26 includes the selection unit 29.

[0031] (2-1) Cabinet As shown in Figure 1, the housing 21 includes a housing section 21a, a holding section 21b, and a connecting section 21c. The housing section 21a houses the weighing section 24. The holding section 21b is positioned on the housing section 21a. The connecting section 21c connects the housing section 21a and the holding section 21b. The connecting section 21c extends in the vertical direction.

[0032] (2-2) Imaging Department As shown in Figures 1 and 2, the imaging unit 22 images the product 200. The imaging unit 22 is located in the holding unit 21b. When the product 200 is placed on the weighing platform 24a, the imaging unit 22 is controlled by the control unit 28 of the control unit 27 (described later) to image the product 200 and acquire image I (see Figures 3 and 4). The imaging unit 22 is, for example, a CCD image sensor or a CMOS image sensor that acquires a color image. The imaging unit 22 may also include a stereo camera or an infrared camera that acquires a temperature image of the product 200. The imaging unit 22 transmits the acquired image I from the price determination device 20 to the control unit 11 of the product candidate presentation system 10 via the network NW.

[0033] The imaging unit 22 may be a device independent of the price determination device 20, and the image I captured by the imaging unit 22 may be transmitted to the control unit 11 using the communication device of the imaging unit 22 or the gateway to which the imaging unit 22 is connected.

[0034] (2-3) Light source The light source 23 illuminates the product 200 being imaged by the imaging unit 22. The light source 23 is provided in the holding unit 21b. One or more light sources 23 are arranged in the vicinity of the imaging unit 22.

[0035] (2-4)Measuring part The weighing unit 24 weighs the product. The weighing unit 24 is located in the storage unit 21a. The weighing unit 24 includes a weighing platform 24a, a load cell (not shown), a signal processing circuit, and a transmission module. The product 200 is placed on the weighing platform 24a. A load cell is located below the weighing platform 24a. The load cell converts the mechanical strain generated when the product 200 is placed on the weighing platform 24a into an electrical signal. The signal processing circuit amplifies the signal output by the load cell and converts it into a digital signal, and the transmission module transmits the digital signal to the control unit 27.

[0036] (2-5) Sticky keys The fixed key 25 has various keys necessary for operating the price determination device 20. The fixed key is provided in the housing section 21a.

[0037] (2-6)Display section The display unit 26 displays various types of information. In this embodiment, the display unit 26 is a touch panel type display. The display unit 26 may have one screen or multiple screens. The display unit 26 is provided in the housing unit 21a.

[0038] As shown in Figures 3 and 4, the display unit 26 displays an image I of the product 200 captured by the imaging unit 22, and multiple product candidates presented by the product candidate presentation system 10 based on this image I. In the example shown in Figures 3 and 4, image I is displayed on the left and product candidates are displayed on the right.

[0039] As shown in Figure 5, the display unit 26 displays the product name selected by the user (by touching) from the product candidates by the selection unit 29, the weight value of the product 200 weighed by the weighing unit 24, the unit price of the product obtained from the store computer 100, and the price of the product 200 calculated by the calculation unit 31.

[0040] (2-7) Control Unit As shown in Figure 1, the control unit 27 is provided in the housing section 21a. The control unit 27 is implemented by a computer. The control unit 27 comprises a control arithmetic unit and a memory device. A processor such as a CPU or GPU can be used for the control arithmetic unit. The control arithmetic unit reads a program stored in the memory device and performs predetermined image processing and arithmetic processing according to this program. Furthermore, the control arithmetic unit can write the calculation results to the memory device and read information stored in the memory device according to the program.

[0041] The control unit 27 is connected to the control unit 11 and the store computer 100 via a network NW, enabling communication. As shown in Figure 2, the control unit 27 includes a control unit 28, a selection unit 29, a storage unit 30, a calculation unit 31, and an output unit.

[0042] (2-7-1) Control Unit When the control unit 28 detects that a product 200 has been placed on the weighing platform 24a of the weighing unit 24 based on the weighing value of the weighing unit 24, it controls the imaging unit 22 to capture an image of the product 200 placed on the weighing platform 24a. Alternatively, the control unit 28 may also capture an image of the product 200 placed on the weighing platform 24a when it detects a difference between the base image of the weighing platform 24a (an image taken without any products in it) which is pre-stored in the storage unit 30 and the image currently being captured. Furthermore, instead of automatically controlling the imaging unit 22, the control unit 28 may capture an image of the product 200 based on an operation input from a fixed key 25 or the like. The control unit 28 also controls the display of the display unit 26.

[0043] (2-7-2) Selection Section The selection unit 29 selects a product from the candidates presented by the product candidate presentation system 10. As shown in Figures 3 and 4, the selection unit 29 is included in the display unit 26. The user operates the selection unit 29 on the display unit 26, which is a touch panel display, to select the correct product from the displayed product candidates.

[0044] In Figure 3, the selection section 29 consists of selection buttons (each area displayed as a partition) labeled with products A to G. In Figure 4, the selection buttons are labeled with products H to L. If the correct product is among the multiple candidates products A to G and products H to L displayed on the display section 26, the user can select the correct product by pressing the selection button labeled with the correct product in the selection section 29. The product selected in the selection section 29 is accepted as the input for the product corresponding to image I. Therefore, information about the product selected in the selection section 29 is transmitted to the calculation section. The user's selection result is transmitted from the price determination device 20 to the store computer 100 via the network NW.

[0045] The control unit 27 outputs a pair of images of the product 200 captured by the imaging unit 22 and information about the product selected by the selection unit 29 (the correct product name determined by a human) as training data. The outputted training data is input to the control unit 11 and stored in a training data storage area (not shown) of the storage unit 12, which will be described later. When the control unit 11 generates (constructs) a new trained model, it trains the pre-generation trained model with at least a portion of the originally existing training data and at least a portion of the newly stored training data. In other words, the trained model is updated by appropriately training with the added training data. Such training is performed separately from accounting processing, for example, when the store closes.

[0046] (2-7-3) Storage section The storage unit 30 stores the product master and information necessary for product identification. The storage unit 30 retrieves the product master from the store computer 100. The product master includes a data table that associates at least the product number (ID) with the product name, unit price, fixed price, etc. The product master is updatable (modifiable).

[0047] (2-7-4) Calculation section The calculation unit 31 calculates the price of a product based on the product selected by the selection unit 29 and the weight measured by the weighing unit 24. Specifically, when the calculation unit 31 receives information about the product selected by the selection unit 29, it reads information about that product from the product master. Here, the calculation unit 31 obtains the unit price of the product corresponding to the selected product from the product master. The calculation unit 31 obtains the weight value measured by the weighing unit 24. Then, the calculation unit 31 calculates the price of the product based on the obtained weight value and the unit price of the product. The calculation unit 31 transmits the calculated price to the display unit 26.

[0048] (3) Product candidate suggestion system The product candidate suggestion system 10 will be explained with reference to Figures 1 to 7. Figure 6 is a block diagram of the product candidate suggestion system 10. Figure 7 is a conceptual diagram of the neural network of the trained model algorithm in the memory unit 12 of the control unit 11.

[0049] As shown in Figure 6, the product candidate presentation system 10 comprises the control unit 11 and the display unit 26 described above. The control unit 11 controls the display of the display unit 26. In this embodiment, the display unit 26 is incorporated into the price determination device 20. The control unit 11 and the display unit 26 are separate components, but they may be an integrated tablet terminal or the like.

[0050] The control unit 11 is implemented by a computer. The control unit 11 comprises a control arithmetic unit and a memory device. The control arithmetic unit can use a processor such as a CPU or a GPU. The control arithmetic unit reads a program stored in the memory device and performs predetermined image processing and arithmetic processing according to this program. Furthermore, the control arithmetic unit can write the calculation results to the memory device or read information stored in the memory device according to the program.

[0051] The control unit 11 includes a storage unit 12, an acquisition unit 13, an estimation unit 14, and a presentation unit 15.

[0052] (3-1) Storage section The memory unit 12 has a product master memory area 12a, a trained model memory area 12b, and a specific information memory area 12c as memory areas for storing various types of information.

[0053] The product master storage area 12a stores the product master. The product master is the same as the product master stored in the storage unit 30 of the price determination device 20. However, since the product master stored in the storage unit 30 is used for the accounting process described above, it is desirable that it be constantly updated with the latest product master obtained from the store computer 100. On the other hand, since the product master stored in the product master storage area 12a is used for the product estimation results by the trained model described later, it may be the product master at the time the trained model stored in the trained model storage area 12b was trained. In other words, there may be discrepancies in the contents of the product master between the storage unit 30 and the product master storage area 12a due to the addition or discontinuation of products for sale.

[0054] The pre-trained model memory area 12b stores the pre-trained model. The pre-trained model is a model generated by machine learning. The pre-trained model is acquired via a storage medium, network, etc.

[0055] The specific information storage area 12c stores specific information relating to a specific product. A specific product is a product that meets predetermined conditions within a product group. A product group is a collection of products sold or offered at a store, etc. The predetermined conditions are at least one of the following: the product has been registered within a predetermined period, there is no pre-trained model corresponding to the product, and it is designated as a specific product.

[0056] In this embodiment, the specific information is information relating to a product when, under predetermined conditions, the product has been registered within a predetermined period of time. When the control unit 11 stores the latest product master obtained from the store computer 100 in the product master storage area 12a, it compares the latest product master (hereinafter referred to as the "new product master") with the product master stored in the product master storage area 12a (hereinafter referred to as the "old product master"). Based on the comparison between the new product master and the old product master, the control unit 11 extracts the differences and sets the products with differences as specific information for a certain period of time (for example, 7 days). The specific information storage area 12c stores this specific information. In this embodiment, one example will be described in which five pieces of specific information are stored in the specific information storage area 12c (there are five specific products). When a certain period of time has elapsed for a specific product, the control unit 11 deletes the specific information relating to that product from the specific information storage area 12c.

[0057] (3-2) Acquisition part The acquisition unit 13 acquires an image I of the product 200. The image I of the product 200 may be an image captured by the imaging unit 22, or it may be an image of the product extracted from an image captured by the imaging unit 22. In addition, the image I of the product 200 may be an image captured by the imaging unit 22 to which image processing has been applied (for example, correction to match the brightness value of the captured image to the brightness value of an image learned by a trained model). Here, the acquisition unit 13 acquires the image I transmitted from the price determination device 20 via the network NW.

[0058] (3-3) Estimation part The estimation unit 14 estimates candidate products from the product group based on image I. The estimation unit 14 acquires image I from the acquisition unit 13 to be used for estimating product 200.

[0059] In this embodiment, the estimation unit 14 estimates product candidates using a trained model. The trained model predicts and outputs the product indicated by the image based on the image data using machine learning. The trained model includes a neural network. The trained model may include a convolutional neural network. Furthermore, the trained model may include neural networks of multiple layers (e.g., 8 or more layers). That is, the trained model may be generated by deep learning.

[0060] As shown in Figure 7, a neural network consists of, for example, an input layer, multiple hidden layers, and an output layer. The input layer outputs an input value x (=x0, x1, x2, ... xp) with p parameters as elements, directly to the hidden layer. Each hidden layer uses an activation function to convert the total input into an output and passes that output to the next layer. The output layer uses an activation function to convert the total input into a neural network output vector y (=y0, y1, ..., yq) with q parameters as elements. Each yi is a numerical representation of the probability that it is product Si, and is output for each product Si. i is an identification value assigned to each product, and is associated with one of the values ​​from 1 to q, based on the number of parameter elements determined by the total number of products (types).

[0061] Here, the neural network takes the pixel value of each pixel in an image as input and outputs information (output vector y) indicating the estimated product. The input layer of the neural network has as many neurons as there are pixels in the image. The output layer of the neural network has neurons that output information related to the estimated product. Based on the output values ​​of the neurons in the output layer, the product can be estimated. The output values ​​of the neurons are, for example, between 0 and 1. In this case, the larger the neuron value (the closer the value is to 1), the higher the probability that it is the product in the image, and the smaller the neuron value (the closer the value is to 0), the lower the probability that it is the product in the image. That is, if the neuron value of product Si (yi, or a value calculated based on yi) is large, the probability that the product in the image is product Si is high, and if the neuron value of product Si is small, the probability that it is product Si is low.

[0062] The estimation unit 14 inputs image I into the trained model. The estimation unit 14 may normalize the input image. Image normalization can be performed, for example, by reducing, enlarging, or cropping the image. The estimation unit 14 may also perform various processing on the input image, such as adjusting the contrast, changing the color, or changing the format. In response to inputting the image into the trained model's neural network, the estimation unit 14 obtains an estimation result that includes the output value output from the neural network. The estimation result includes all types of products registered in the product master.

[0063] The estimation unit 14 ranks the candidate products based on the estimation results. Specifically, the estimation unit 14 calculates the likelihood of each product being estimated from image I (the probability of each product being estimated based on the neuron value described above) for each possible candidate product. In particular, the estimation unit 14 ranks the products in descending order of neuron value. The estimation unit 14 generates estimation information for all products, associating the product number with the rank. The estimation unit 14 transmits the image information and estimation information used in the estimation process to the presentation unit 15.

[0064] (3-4) Presentation section The presentation unit 15 selects product candidates estimated by the estimation unit 14 (products that fall within a predetermined rank when sorted from those with the highest likelihood among the products estimated by the estimation unit 14) as the first candidate group, and also selects product candidates as the second candidate group based on specific information, and presents the first and second candidate groups. The first and second candidate groups consist of products included in the product master and do not include the same product twice. Each of the first and second candidate groups consists of one or more products. The product candidates presented by the presentation unit 15 are displayed on the display unit 26.

[0065] The first candidate group in this embodiment is determined based on the ranking of candidate products, which is determined by the results (likelihood) estimated from image I by the estimation unit 14 (trained model). The first candidate group is formed by arranging products up to a predetermined rank in descending order of rank from the product ranks estimated by the estimation unit 14. Specifically, the first candidate group is extracted by selecting products up to the kth rank (item number), which is the predetermined rank, in order of the likelihood of the estimation output by the trained model. In detail, products Si from the 1st to the kth rank (item number) are extracted from products Si that have been ranked as 1st (item number), 2nd (item number), ..., qth (item number), starting with those with the largest neuron values ​​yi corresponding to each product Si output by the trained model.

[0066] In this embodiment, the second candidate group is determined based on specific information, excluding the products selected in the first candidate group. The second candidate group is a selection of specific products arranged in descending order of their estimated rank by the estimation unit 14. Note that, for specific products, if the estimation unit 14 determines that a particular product falls within a predetermined rank, it will be included in the first candidate group, even if it is a specific product.

[0067] Here, the presentation unit 15 presents the second group of candidates on a different screen than the first group of candidates. In this case, the display unit 26 displays a predetermined number of products on the first screen (first screen) G1 in order of decreasing likelihood based on the estimation unit 14, and then displays specific products on the next screen (second screen) G2 in order of decreasing likelihood based on the estimation unit 14. Here, since there is a limit to the number of products that can be displayed on the display unit 26 at the same time, the presentation unit 15 limits the first and second group of candidates that can be displayed on the display unit 26 screen, as will be described later. The second group of candidates that could not be displayed on screen G2 of the display unit 26 will be displayed on subsequent screens, which can be accessed by selecting the page-turning button B displayed on the display unit 26 (by touching it).

[0068] Furthermore, the presentation unit 15 presents at least a portion of the first candidate group on the first screen G1 of the display unit 26, and at least a portion of the second candidate group on screen G2 of the display unit 26. In other words, the presentation unit 15 presents the first and second candidate groups so that the display unit 26 can display them on different screens. Specifically, as described above, the presentation unit 15 designates the first candidate group to the 1st to kth (where k is an arbitrary integer) of the product ranks estimated by the estimation unit 14. The presentation unit 15 also reads specific information from the specific information storage area 12c and designates the second candidate group as the specific information that has been ranked excluding the products selected in the first candidate group.

[0069] The presentation unit 15 may also present the products of the first candidate group and the products of the second candidate group by displaying them in separate sections on the display unit 26, rather than on a single screen. Alternatively, the presentation unit 15 may present the products of the first candidate group and the products of the second candidate group by displaying them on the display unit 26 in a visually distinct manner.

[0070] When a trained model performs estimation on images of products that can be estimated relatively accurately (images of products that the user does not dislike), it has been found that the products estimated by the trained model are likely to be correct up to a certain rank, starting from the products with the highest likelihood, but the likelihood of being correct for products below that rank is low. For example, if we add the top 1, 2, ... up to the 4th product from a group of products as candidates, the statistical probability that at least one of them is correct improves with each additional product added, reaching about 80%, but even if we add the 5th and 6th products as candidates, the statistical probability that at least one of them is correct may not improve much from 80% (it may saturate).

[0071] In this case, it is preferable to set k to 4. When k is 4, on screen G1 of the display unit 26 in Figure 3, products A to D (four products) are candidates 1 to 4 of the products estimated by the estimation unit 14, and products E to G (determined by the number of products that can be displayed simultaneously, in this case three products) are randomly selected products. In this case, the presentation unit 15 may also set a boundary line between product D and product E, or it may present products A to D and products E to G in different colors. The value of k can vary depending on the group of products being estimated, and is not limited to 4 depending on the group of products.

[0072] (3-5)Display section The display unit 26 has a screen that displays the first and second candidate groups presented by the presentation unit 15. Here, the display unit 26 is the display of the price determination device 20. Therefore, the product candidates corresponding to the image I presented by the presentation unit 15 are transmitted to the display unit 26 of the price determination device 20 via the network NW.

[0073] The display unit 26 has a screen. As described above, based on the output (presentation) of the presentation unit 15, the display unit 26 displays candidate products in order from the first screen to the next, in descending order of the probability that they are the correct product. In Figures 3 and 4, each screen displays the product image I, as well as the product candidates presented by the presentation unit 15. As shown in Figure 3, the screen G1 of the display unit 26 displays a predetermined number (for example, 7) of product candidates corresponding to image I, arranged vertically with the products with the highest probability at the top. As shown in Figure 4, the screen G2 of the display unit 26 displays a predetermined number (for example, 5) of candidate products for a specific product, arranged vertically with the products with the highest probability at the top.

[0074] (4) Method of presenting product candidates and accounting treatment The method for presenting product candidates and the accounting treatment method will be explained with reference to Figure 8. Figure 8 is a flowchart of the pricing method. Note that the flowchart shown in Figure 8 is merely one example of the product price determination process and may be modified as appropriate within the bounds of consistency. For example, the flowchart in Figure 8 does not limit the order of each step, and the order of each step may be changed as appropriate within the bounds of consistency.

[0075] The method for presenting product candidates is the product candidate presentation system 10 described above. The method for accounting is the accounting processing system 40 described above.

[0076] First, the acquisition unit 13 acquires an image I of the product 200 (step S1: acquisition step). Step S1 is carried out, for example, as follows: When the product 200 is placed on the weighing platform 24a, the control unit 28 controls the imaging unit 22 to cause the imaging unit 22 to image the product 200. Next, the control unit 27 of the price determination device 20 transmits the image I to the control unit 11 of the product candidate presentation system 10 via the network NW. The acquisition unit 13 acquires the transmitted image I.

[0077] Next, the estimation unit 14 estimates candidate products from the product group based on image I (step S2: estimation step). Here, the estimation unit 14 uses a pre-trained machine learning model to rank the candidate products based on the results estimated from image I.

[0078] Next, the presentation unit 15 selects the product candidates estimated by the estimation unit 14 as the first candidate group, and also selects product candidates as the second candidate group based on specific information, and presents the first and second candidate groups (Step S3: Presentation Step). Step S3 is carried out, for example, as follows.

[0079] The estimation unit 14 selects the first to k (where k is an arbitrary integer) of the estimated product ranks as the first candidate group. The presentation unit 15 reads the stored (storage step) specific information from the storage unit 12's specific information storage area 12c and selects the ranked specific information, excluding the products selected in the first candidate group, as the second candidate group.

[0080] Next, the first and second candidate groups presented by the presentation unit 15 are displayed on the display unit 26 (step S4). In step S4, the control unit 11 transmits the first and second candidate groups presented by the presentation unit 15 to the control unit 27 of the price determination device 20. The control unit 27 displays the product candidates on the display unit 26 for the user to select.

[0081] Next, the selection unit 29 selects a product from the candidates presented by the product candidate presentation system 10 (step S5). Here, the user operates the display unit 26, which is a touch panel display, and presses the selection unit 29 that lists one product from among the multiple product candidates displayed on the display unit 26. Steps S4 and S5 above are carried out, for example, as follows.

[0082] On the first screen of the display unit 26, as shown in Figure 3, seven candidate products corresponding to image I are displayed, arranged vertically with the most likely product at the top. On the next screen of the display unit 26, as shown in Figure 4, five candidate products for a specific product are displayed, arranged vertically with the most likely product at the top. The user, upon viewing this, operates the selection unit 29 of the display unit 26 to select the correct product from products A-G and products H-L. For example, the user selects the correct product by pressing the selection unit 29 that displays the correct product.

[0083] Furthermore, the display unit 26 is configured to allow the user to select the correct product if all of the estimations by the estimation unit 14 are incorrect. In this example, the first candidate group is displayed on the first screen G1, and the second candidate group is displayed on the next screen G2. If there are no correct products among products A-G and H-L, the next screen displays the other products. In this way, the user can sequentially navigate through the screens until they find a screen containing the correct product. In this example, the display unit 26 displays seven products on screen G1 and five products on screen G2, but the number of products displayed on a screen is not limited. Also, the number of products displayed on each screen may or may not be constant.

[0084] The user's product selection result is transmitted from the price determination device 20 to the control unit 11 via the network NW. The weighing unit 24 of the price determination device 20 weighs the product 200 placed on the weighing platform 24a (step S6). In step S6, the control unit 27 obtains the weight value of the product 200.

[0085] Next, the price of product 200 is calculated based on the product selected by the selection unit 29 and the weight measured by the weighing unit 24 (step S7). In step S7, the calculation unit 31 calculates by multiplying the unit price of the product selected by the user in step S5 (the unit price of the product transmitted from the store computer 100), which is read from the storage unit 30, by the weight value of product 200 obtained in step S6, and determines the calculated value as the price of product 200. Finally, the control unit 27 displays the price of product 200 determined in step S7, along with the weight value and unit price of product 200, on the display unit 26, as shown in Figure 5.

[0086] (Effects and Benefits) As described above, in the product candidate presentation system 10 according to this embodiment, the storage unit 12 stores products that meet predetermined conditions in a group of products as specific products, and the presentation unit 15 presents product candidates estimated by the estimation unit 14 and specific products stored in the storage unit 12. In this configuration, for example, by storing a new product as a specific product in the storage unit 12, the specific product can be presented to the presentation unit 15 in the same way as product candidates estimated by the estimation unit 14. Therefore, even if the product selection screen is displayed across multiple pages in the presentation unit 15, or if the product selection screen is displayed by scrolling, it is possible to avoid displaying the new product on the last page or at the bottom of the scroll. Thus, the product candidate presentation system 10 can reduce the effort required for product selection.

[0087] In the product candidate presentation system 10 according to this embodiment, the estimation unit 14 estimates product candidates using a trained model. In a configuration where product candidates are estimated using a trained model, it is difficult for the estimation unit to estimate new products. Therefore, a configuration in which new products are stored as specific products and displayed on the presentation unit 15 is particularly effective when product candidates are estimated using a trained model.

[0088] In the product candidate suggestion system 10 according to this embodiment, the predetermined conditions may be at least one of the following: the product has been registered within a predetermined period of time; there is no pre-trained model corresponding to the product; and the product has been designated as a specific product. New products fall under the cases where the product has been registered within a predetermined period of time or where there is no pre-trained model corresponding to the product. In addition, users may want to arbitrarily set a product as a specific product. Therefore, by setting the predetermined conditions to the above content, the product can be stored in the storage unit 12 as a specific product.

[0089] In the product candidate presentation system 10 according to this embodiment, the storage unit 12 stores the image acquired by the acquisition unit 13 in association with the specific product when a specific product is selected in response to the presentation by the presentation unit 15. In this configuration, the data associating the image with the specific product can be used as training data for a trained model. This makes it possible to improve the estimation accuracy of the specific product in the estimation unit 14.

[0090] In the product candidate presentation system 10 according to this embodiment, the presentation unit 15 changes the order in which it presents multiple specific products based on the estimation results from the estimation unit 14 for each of the multiple specific products when the product group includes multiple specific products. With this configuration, multiple specific products can be displayed, for example, in the order in which they are most likely to be products.

[0091] In the product candidate presentation system 10 according to this embodiment, the presentation unit 15 presents product candidates estimated by the estimation unit 14 on the first screen, and also displays a specific product stored in the storage unit 12 on a different screen G2 (see Figure 4) from screen G1 (see Figure 3). In this configuration, since the product candidates estimated by the estimation unit 14 and the specific product are displayed on separate screens, the specific product can be confirmed at a glance.

[0092] In the product candidate presentation system 10 according to this embodiment, the presentation unit 15 presents a specific product on screen G1 if the likelihood of the specific product in the estimation result of the estimation unit 14 is within a predetermined rank in the product group. In this configuration, if there is a high probability that the product based on the image is the specific product, the specific product is displayed on screen G1 instead of screen G2. Therefore, it becomes easier to select the specific product.

[0093] While embodiments of the present invention have been described above, the present invention is not necessarily limited to the embodiments described above, and various modifications are possible without departing from the spirit of the invention.

[0094] In the above embodiment, a configuration comprising a control unit 11 and a price determination device 20 was described as an example. However, the control unit 11 and the price determination device 20 may be configured as an electronic scale.

[0095] In the above embodiment, the specific information stored in the storage unit 12 was described as, as an example, information relating to a product when, under predetermined conditions, a predetermined period has passed since the product was registered. However, the predetermined conditions for the specific information may also include cases where there is no pre-trained model corresponding to the product, and / or when the product is designated as a specific product. When a product is designated as a specific product, it is stored as specific information in the specific information storage area 12c of the storage unit 12 through a predetermined operation by the user.

[0096] In the above embodiment, the presentation unit 15 was described as an example in which it displays a second group of candidates on screen G2 of the display unit 26. Since five products are stored in the storage unit 12 as specific information, in the example shown in Figure 4, products H to L are displayed on screen G2 of the display unit 26. However, if the number of products displayed on screen G2 is less than or equal to a predetermined number (for example, 7 products), randomly selected products may be displayed on screen G2. For example, in addition to products H to L, products M and N may also be displayed.

[0097] In the above embodiment, the first candidate group consists of the 1st to kth (where k is any integer) products among the product ranks estimated by the estimation unit 14, but it is not necessary to separate them by number. In this configuration, the first candidate group consists of products whose neuron value is above a threshold. Therefore, the number of products in the first candidate group will differ depending on the neuron value in the estimation unit.

[0098] In the above embodiment, the estimation unit 14 estimates the product corresponding to image I using a trained model, but is not limited to this. The estimation unit 14 may also estimate the product from image I using a rule-based method without using a trained model. For example, in the control unit 11, the relationship between images and products is described in advance by a program, and the estimation unit 14 may estimate the product corresponding to image I based on this program.

[0099] In the above embodiment, the product candidate presentation system 10 is configured to communicate with the store computer 100 via a network NW, but the store computer 100 may be omitted.

[0100] In the above embodiment, the estimation unit 14 estimates product candidates from all product groups registered in the product master based on the captured product images. The product groups include products that are not handled for reasons such as being out of stock. In this modified example, the estimation unit estimates product candidates from a product group that does not include products that are not handled (a product group consisting of products that are handled) based on the captured product images I. Specifically, the product master storage area 12a stores a product master containing products that are handled. Products that are handled mean products that are sold or provided at the store where the product candidate presentation system 10 is used when the product candidate presentation system 10 is in operation, or products that are in stock. More specifically, products that are sold or provided are products that are managed by the store as being sold or provided. More specifically, products that are in stock are products that are managed as being in stock.

[0101] For example, the store computer 100 is configured to transmit information regarding the availability of each product (hereinafter also referred to as "product information") to the control unit 11 via the network NW. Product information is, in short, information about products that are available and products that are not. The store computer 100 transmits the product information at predetermined timings. The store computer 100 may also transmit the product information in response to a transmission request from the control unit 11. Based on this information transmitted from the price determination device 20, the control unit 11 rewrites the product master stored in the product master storage area 12a of the storage unit 12.

[0102] Note that product information may be transmitted from the price determination device 20 to the control unit 11, rather than to the store computer 100. For example, store employees at a supermarket or other store input the products to be handled (sold / provided) on that day into the price determination device 20 using an input device such as the display unit 26, which is a touch panel display. Also, for example, store employees input products that are out of stock into the price determination device 20 as appropriate using an input device such as the display unit 26. The price determination device 20 transmits this input information to the control unit 11. Based on this information transmitted from the price determination device 20, the control unit 11 rewrites the product master stored in the product master storage area 12a of the storage unit 12.

[0103] Furthermore, if there is a discrepancy between the contents of the product master in the storage unit 30 and the product master storage area 12a due to the addition or discontinuation of sales products, the estimation unit 14 may be configured to acquire a product master (the difference from the product master in the product master storage area 12a) which is a collection of products that are handled and stored in the storage unit 30. According to the product candidate presentation system and accounting processing system of this modified example, the estimation unit 14 reduces the possibility that a product that is not handled will be estimated as the product corresponding to image I. Therefore, since the estimation unit can estimate products from image I with high accuracy, the possibility that the correct product will be included in the first candidate group can be improved.

[0104] In the above embodiment, the accounting system 40 weighs the product 200 and determines the price of the product 200 by multiplying the weight of the product 200 by the unit price of the product 200, but is not limited to this. The price determination device of the accounting system does not need to have a function to weigh the product 200. In this modified version of the accounting system, the price of the product 200 is determined based on the selection result of the product 200 and the price information of the product obtained from the store computer 100. [Explanation of Symbols]

[0105] 10...Product candidate presentation system, 11...Control unit (product candidate presentation device), 12...Storage unit, 13...Acquisition unit, 14...Estimation unit, 15...Presentation unit, 24...Weighing unit, 29...Selection unit, 31...Calculation unit, G1...Screen (first screen), G2...Screen (second screen), I...Image.

Claims

1. A product candidate presentation device that presents product candidates to the user for selection based on an image of the captured product, An acquisition unit for acquiring an image of the aforementioned product, An estimation unit that estimates candidate products from a group of products based on the aforementioned image, A storage unit that stores products that meet predetermined conditions in the aforementioned product group as specific products, The system comprises a presentation unit that presents candidate products estimated by the estimation unit and specific products stored in the storage unit, The estimation unit estimates candidate products using the trained model, A product candidate presentation device, wherein the predetermined conditions are at least one of the following: the product is registered within a predetermined period of time, and there is no pre-trained model corresponding to the product.

2. The product candidate presentation device according to claim 1, wherein the storage unit stores the image acquired by the acquisition unit in association with the specific product when the specific product is selected in response to the presentation by the presentation unit.

3. The product candidate presentation device according to claim 1 or 2, wherein, when the product group includes multiple specified products, the presentation unit changes the order in which the multiple specified products are presented based on the estimation results of the estimation unit for each of the multiple specified products.

4. The product candidate presentation device according to any one of claims 1 to 3, wherein the presentation unit presents the product candidates estimated by the estimation unit on a first screen, and displays the specific product stored in the storage unit on a second screen different from the first screen.

5. The product candidate presentation device according to claim 4, wherein the presentation unit presents the specific product on the first screen if, in the estimation results of the estimation unit, the likelihood of the specific product is within a predetermined rank in the product group.

6. A product candidate presentation device according to any one of claims 1 to 5, A weighing unit for weighing the aforementioned product, A selection unit that selects the product from among the candidates presented by the product candidate presentation device, An electronic scale comprising: a calculation unit that calculates the price of the product based on the product selected in the selection unit and the weight of the product measured in the weighing unit.

7. A product candidate presentation system that presents candidates for the product to the user for selection based on an image of the product captured, An acquisition unit for acquiring an image of the aforementioned product, An estimation unit that estimates candidate products from a group of products based on the aforementioned image, A storage unit that stores products that meet predetermined conditions in the aforementioned product group as specific products, The system comprises a presentation unit that presents candidate products estimated by the estimation unit and specific products stored in the storage unit, The estimation unit estimates candidate products using the trained model, A product candidate suggestion system in which the predetermined conditions are at least one of the following: the product is registered within a predetermined period of time, and there is no pre-trained model corresponding to the product.

8. A method for presenting product candidates to a user for selection based on an image of the captured product, The acquisition step involves obtaining an image of the aforementioned product, An estimation step in which candidates for the product are estimated from the product group based on the aforementioned image, A storage step in which products that meet predetermined conditions in the aforementioned product group are stored as specific products, The presenting step includes presenting the candidate products estimated in the estimation step and the specific products stored in the storage step, In the estimation step described above, candidate products are estimated using the trained model, A method for suggesting product candidates, wherein the predetermined conditions are at least one of the following: the product has been registered within a predetermined period of time, and there is no pre-trained model corresponding to the product.

Images