Information processing device, information processing system, and computer program

Abstract

This information processing device comprises: an imaging means that captures an image of a product; an extracting means that uses the image of the product captured by the imaging means as a basis for extracting candidate products for which a training image is to be added by referring to a product information storage means storing product information; a designation accepting means that accepts a designation of a product for which the training image is to be added and registered from among the candidate products extracted by the extracting means; and a registering means that registers the image of the product captured by the imaging means as the training image in the product information storage means, in association with the product designated in the designation accepting means.

Description

Information processing device, information processing system, and computer program

[0001] The present invention relates to an information processing device, an information processing system, and a computer program.

[0002] 2. Description of the Related Art A so-called object scanning technique is known in which an image of an object is captured and compared with a learning image serving as a reference image for image recognition to identify the object.

[0003] For example, Patent Document 1 discloses an item identification device that includes a tray on which items are placed, a backlight source that illuminates the tray from the bottom side with backlight light, a digital camera that captures images of the items on the tray, and an identification computer that recognizes the images from the digital camera and identifies the items. This item identification device extracts the items on the tray from a differential image between an image taken when the backlight source is on and an image taken when the backlight source is off.

[0004] Furthermore, for example, Patent Document 2 discloses a product recognition device that includes a storage unit that pre-stores an image database in which at least learning image information and product names are associated for a plurality of products; a learning unit that performs machine learning using the learning image information and product names stored in the image database to generate a trained identification engine; and an identification unit that uses the identification engine to identify the product name of a target product based on image information of the target product.

[0005] JP 2017-162217 A

[0006] Patent No. 7125179

[0007] In this regard, image learning for object scanning requires capturing an image of each target product, specifying the product each time, and registering the image, which is extremely tedious.

[0008] Furthermore, when performing an object scan of products to be purchased in a store such as a supermarket, if all products registered in a predetermined storage device are subject to the object scan, there is a possibility that products that are not currently on sale will be extracted. Therefore, it is necessary to search for the target products in the predetermined storage device and set whether or not to make them available for sale. However, searching for and setting each target product is cumbersome and troublesome.

[0009] Therefore, one of the objects of the present invention is to provide a device that can easily register learning images to be used in object scanning.

[0010] Another object of the present invention is to provide a device that can easily set product sales targets.

[0011] In order to achieve the above-mentioned object, the information processing device of the present invention comprises an imaging means for imaging a product, an extraction means for extracting candidate products for which a learning image is to be added based on the image of the product imaged by the imaging means and by referring to a product information storage means for storing product information, a designation acceptance means for accepting designation of a product for which the learning image is to be added and registered from the candidate products extracted by the extraction means, and a registration means for registering the image of the product imaged by the imaging means as the learning image in the product information storage means in association with the product designated by the designation acceptance means.

[0012] In addition, in order to achieve the above-mentioned object, the information processing device of the present invention comprises an imaging means for imaging a product, an extraction means for extracting product candidates corresponding to the product image captured by the imaging means in a predetermined operation mode in which an operation for purchasing the product is performed, based on the product image captured by the imaging means, by referring to a product information storage means that stores product information, and a setting means for setting the product extracted by the extraction means in the predetermined operation mode as a product for sale or a product not for sale.

[0013] FIG. 1 is a diagram showing the configuration of a system including a weighing device according to an embodiment of the present invention.

[0014] FIG. 2 is a functional block diagram showing the hardware configuration of a weighing device according to an embodiment of the present invention.

[0015] FIG. 3 is a functional block diagram showing the software configuration of the weighing device according to the embodiment of the present invention.

[0016] FIG. 4 is an example of a screen displayed on a weighing device according to an embodiment of the present invention, showing a standby screen in the first operation mode.

[0017] FIG. 5 shows an example of a screen displayed on a weighing device according to an embodiment of the present invention, which is a screen for specifying a product in the first operation mode.

[0018] FIG. 6 shows an example of a screen displayed on the weighing device according to the embodiment of the present invention, which is a screen for specifying a product in the second operation mode.

[0019] FIG. 7 shows an example of a screen displayed on a weighing device according to an embodiment of the present invention, which is a screen for specifying a product in the third operation mode.

[0020] FIG. 8 shows an example of a screen displayed on a weighing device according to an embodiment of the present invention, which is a screen for specifying a product in the fourth operation mode.

[0021] FIG. 9 is a processing flow diagram showing an example of processing by the weighing device according to an embodiment of the present invention, and showing the overall processing in the first operation mode.

[0022] FIG. 10 is a process flow diagram showing an example of processing by the weighing device according to an embodiment of the present invention, specifically, processing for extracting product items in the first operation mode.

[0023] FIG. 11 is a processing flow diagram showing an example of processing by the weighing device according to an embodiment of the present invention, and showing the entire processing in the second operation mode.

[0024] FIG. 12 is a process flow diagram showing an example of processing by the weighing device according to an embodiment of the present invention, and showing the overall processing in the third operation mode.

[0025] FIG. 13 is a processing flow diagram showing an example of processing by the weighing device according to an embodiment of the present invention, and showing the overall processing in the fourth operation mode.

[0026] System Configuration An information processing device according to an embodiment of the present invention will be described below with reference to the drawings. In the following description, unless otherwise specified, functional units, processes, etc. that are assigned the same reference numerals as those assigned to already-described functional units, processes, etc. will be assumed to be configured in the same way as the already-described functional units, processes, etc., and description thereof will be omitted.

[0027] FIG. 1 shows an example of the configuration of a system including a weighing device 1 according to this embodiment, and FIG. 2 shows the hardware configuration of the weighing device 1. The weighing device 1 according to this embodiment is an embodiment of an information processing device according to the present invention. The weighing device 1 is installed in a supermarket that sells fresh foods such as vegetables and fruits. In the fresh food section of a supermarket, products such as vegetables and fruits are sold individually and displayed in their respective display areas. Because each product is sold individually, no label bearing a price or barcode is attached. A customer or a store clerk weighs the required number of products by type and receives labels for the number of products purchased from the weighing device 1. The customer or store clerk then attaches the labels to the products. A symbol code containing information such as price is printed on the label. The symbol code can be read by a POS terminal 3, allowing the product to be registered as a purchase item in the POS terminal 3.

[0028] The weighing device 1 displays a list of selectable products on the screen of the display touch panel 104 and accepts the selection of products to be purchased from an operator such as a customer or a store clerk. The weighing device 1 also weighs the weight of the products placed on the weighing unit 109, retrieves product data, such as the unit price of the product selected by the operator on the display touch panel 104, from a storage unit such as RAM 103, calculates the price of the product, and issues a label. In particular, the weighing device 1 according to this embodiment performs an object scan of the product selected by the customer or store clerk and presents the customer or store clerk with candidate products so that they can be selected. Note that "object scan" here refers to the process of capturing an image of the product, performing image recognition of the product, and identifying the product by comparing it with a learning image. This process is executed by the image capture control unit 124 and extraction unit 125 (see FIG. 3 ), which will be described later.

[0029] Note that the product data, and part or all of the product file 121 (see FIG. 3 ), which will be described later, may be stored not only in a storage unit provided in the weighing device 1 itself, such as the RAM 103, but also in an appropriate device connected by wire or wirelessly. In particular, the learning images and learning data for object scanning, which will be described later, do not need to be stored in the weighing device 1 itself, but may be stored in, for example, the upper store controller 2 or cloud server 4, and the weighing device 1 may refer to them.

[0030] The store controller 2 is a computer that manages various data within the supermarket, and is connected via a LAN (Local Area Network) to the weighing device 1 and the POS terminal 3. The various data include, for example, a product file that stores information such as product selling prices stored by the POS terminal 3, a product file 121 (see FIG. 3 ) stored by the weighing device 1, and a performance file sent from the POS terminal 3.

[0031] The POS terminal 3 is a so-called POS register, and performs operations such as registering and settling the purchases of products (including fresh foods weighed by the weighing device 1 as well as products with pre-labeled items) purchased by customers. Product labels include product identification numbers such as product codes, and, in the case of labels issued by the weighing device 1, price information based on weight, printed in the form of barcodes or two-dimensional codes. The POS terminal 3 is equipped with a reader that reads these barcodes or two-dimensional codes. The POS terminal 3 then performs product registration processing based on the barcode or two-dimensional code information read by the reader and information such as the product selling price stored in memory. Although not shown in the figure, multiple POS terminals 3 may be installed within a store. The POS terminal 3 may include a terminal operated by a store clerk as well as a self-service terminal operated by customers themselves.

[0032] The cloud server 4 is a device that executes processes performed by in-store devices, including the weighing device 1, and stores data necessary for those processes, and is provided as appropriate. In this embodiment, object scanning of products is performed by the weighing device 1, but the weighing device 1 may only capture images of the products, with the cloud server 4 performing processes related to image recognition and analysis. Furthermore, the cloud server 4 may be configured to generate matching data (learning images) using deep learning using AI (artificial intelligence), thereby improving the accuracy of object scanning.

[0033] The store controller 2 and the cloud server 4 constitute a higher-level management device than the weighing device 1 and the POS terminal 3. Regardless of the present embodiment, the higher-level device may be constituted by a single terminal or device, or may be constituted by a terminal or device different from the cloud server 4. The higher-level device may be a terminal or device within the store that is connected to a network such as a LAN, or may be a terminal or device outside the store that is connected to a network such as the Internet.

[0034] Hardware Configuration of Weighing Device 1 Figure 2 shows the hardware configuration of the weighing device 1, and Figure 3 shows the software configuration of the weighing device 1. As shown in Figure 2, the hardware configuration of the weighing device 1 is made up of a CPU 101, ROM 102, RAM 103, a display touch panel 104, a key operation unit 105, a clock unit 106, a voice input unit 107, a communication unit 108, a weighing unit 109, a printing unit 110, an imaging unit 111, and an external storage control unit 112.

[0035] The CPU 101 is a central processing unit that controls the operation of the weighing device 1, and realizes the operation of the weighing device 1 in accordance with the program by reading and executing a program from the ROM 102. The ROM 102 is a read-only memory in which the program is stored. The RAM 103 is a write / read memory that can be written to or read from at any time, and stores a product file 121 (described later), an icon table, and other temporary data.

[0036] The display touch panel 104 is a user input / output interface between the weighing device 1 and an operator, such as a customer, who operates the weighing device 1. It displays preset keys G221 (see FIG. 5, etc.) for specifying a product to be weighed and various other information on the screen, and also accepts input from the operator using the displayed preset keys G221. Examples of screen displays on the display touch panel 104 will be described later. Furthermore, data on the product corresponding to the preset key G221 specified by the operator on the screen is read from the product file 121 in the RAM 103, and the price of that product is calculated.

[0037] The key operation unit 105 is used to perform input other than that performed on the display touch panel 104 .

[0038] The clock unit 106 outputs time information, which is used to control the display mode of the preset keys G221 (see FIG. 5, etc.) displayed on the display touch panel 104.

[0039] The voice input unit 107 is for performing voice input, and allows the operator to input product information by voice.

[0040] The communication unit 108 is a LAN interface and controls communication with the store controller 2. A product file 121 is acquired from the store controller 2 via this communication unit 108. A program may be transferred from the store controller 2 to the weighing device 1 via the communication unit 108. The transferred program may be stored in the RAM 103, for example.

[0041] The weighing unit 109 measures the weight of the product placed on a weighing pan provided on the top of the weighing device 1 main body using a mass detection method using a load cell or the like, and outputs the measured value. In detail, the weighing unit 109 is composed of a mass detection unit and an A / D converter, and the A / D converter converts the analog signal input from the mass detection unit into a digital signal.

[0042] The printing unit 110 constitutes an issuing unit that issues labels on which the price, barcode, etc. are printed for the products weighed by the weighing unit 109. This label is printed based on the weight value output as the weighing result from the weighing unit 109 and product information (such as unit price data) related to the product in the product file stored in RAM 103. In this embodiment, the label is a sticker, and can be affixed to the product or a bag containing the product. Furthermore, in this embodiment, information necessary for product payment is printed on a label and issued, but this is not limited to this, and the information may also be printed on the bag containing the product.

[0043] The imaging unit 111 is a device that captures images of the products placed on the weighing unit 109, and is configured with an image sensor such as a color CCD image sensor or a color CMOS image sensor. The number of imaging units 111 is arbitrary, and multiple imaging units 111 may be provided. For example, the imaging units 111 may be arranged facing at different angles so that images of the items placed on the weighing unit 109 can be captured from multiple angles.

[0044] A portable storage medium 113 such as a DVD-ROM or USB memory is connected to the external storage control unit 112. Information including programs is stored in the portable storage medium 113, and the external storage control unit 112 reads and writes the information.

[0045] 3, the software configuration of the weighing device 1 is configured using the above-mentioned arithmetic unit such as the CPU 101 and storage unit such as the RAM 103 to form the following functional blocks: a product file 121, a weighing control unit 122, a change detection unit 123, an imaging control unit 124, an extraction unit 125, a setting unit 126, a registration unit 127, a grouping unit 128, a calculation unit 129, a printing control unit 130, a designation reception unit 131, and a display control unit 132. Note that some or all of the functional units may be implemented in a higher-level device such as a store controller 2 connected by wire or wirelessly, or in a cloud server 4 connected via the Internet or the like.

[0046] Here, the weighing device 1 is mainly switched between first to fourth operation modes and performs functions corresponding to each operation mode. The first operation mode is an operation mode used when providing services to customers, while the second to fourth operation modes are so-called maintenance modes in which product information is set, and the screens displayed in these maintenance modes can be said to constitute maintenance screens.

[0047] The first operation mode is an operation mode in which a customer performs operations to purchase or trade goods. In this embodiment, this first operation mode is an operation mode in which goods are weighed and priced, and goods placed on the weighing unit 109 as goods to be purchased are weighed and a label displaying the price is issued. In other embodiments, if the weighing device 1 has the functionality of a POS terminal 3 or the information processing device according to the present invention is configured as a POS terminal 3, in this first operation mode, goods are weighed and goods are registered as goods to be purchased. Furthermore, although the weighing device 1 according to this embodiment performs weighing and pricing, it may also be configured as a device that only performs weighing, with pricing being performed by another linked device.

[0048] The second operation mode is a mode for registering a learning image of a product. In this second operation mode, an image of the product captured by the image capturing unit 111 is registered as a learning image of the product.

[0049] The third operation mode is a mode for setting products as saleable or non-saleable. In this third operation mode, it is possible to set whether the store will handle products registered in the product file 121 as saleable or non-saleable. This setting is necessary depending on the seasonal products and the availability of inventory, as will be described later.

[0050] The fourth operation mode is a mode for grouping products. In this fourth operation mode, multiple types of products that are commonly associated with a predetermined learning image are registered as one group. In the following description of each functional unit, functions that are not specifically mentioned in a predetermined mode are performed in all modes.

[0051] <Product File 121> The product file 121 is a file that stores product information for each product to be weighed by the weighing device 1, including product identification information such as product code, product name, category, preset key name, preset key image, weight (mass), unit price, product color, learning image, group information, and sales target setting. This file constitutes a product information storage unit. The product file 121 can also store other information, such as barcode classification (in-store barcode system), cost, unit weight, and tare weight. The product file 121 may also be created by the store controller 2 or the cloud server 4 and transmitted to the weighing device 1 and stored in RAM 103. However, this does not prevent the weighing device 1 from creating the product file 121 in other embodiments. The selling price may be updated at least once a day or at a predetermined interval. The barcode classification includes PLU code classifications such as JAN (EAN) codes and barcode systems (types) for in-store barcodes used for bulk sales.

[0052] The product code is product identification information and is also called the product number. The product name is the name of the product and is set for search purposes. The classification is a category (item) according to the characteristics of each product. By classifying and registering each product into a predetermined classification, products belonging to the predetermined classification are narrowed down when a predetermined classification tab is selected. The preset key name is the product name or the like displayed on the preset key G221 (see Figure 5, etc.), and if no name is entered, the general name of the product is set. The preset key image is an image used when displaying the preset key G221 to accept product specification. The preset key image displays a pattern, photographic image, text information, etc. that allows the product to be identified.

[0053] The unit weight data is a value set as the average value per product and is used to calculate the number of products from the weight measured by the weighing device 1. This weight data can also be used to determine whether multiple products are placed on the weighing unit 109. Therefore, this data is set only for products sold by the number, such as apples. For products not sold by the number, such as meat sold by the gram, the price is calculated using only the unit price data (price per gram). The unit price data is set as the price per product for products sold by the number, and as the price per 100 grams for products sold by the gram. To allow products such as apples to be sold by the gram, the sales category may be set to "sold by gram" or "sold by unit." When sold by unit, the transaction amount is calculated based on the unit weight setting. The unit weight may be registered by inputting the number of items placed on the weighing pan or by the measured value. When multiple products are weighed, the number of items placed on the weighing pan may be input, and the average weight calculated by dividing the number may be stored. The color of the product is designated as a color associated with the product. One or more colors can be arbitrarily registered as the color of the product so as to accommodate various perceptions of the color of the product.

[0054] The training image is an image of a product, and serves as a reference image to be referenced when recognizing and identifying the product in the captured image through object scanning and extracting product candidates from the product file 121. Training images can be registered by capturing an image of an actual product using the imaging unit 111, or by loading pre-captured image data. For example, a product code can be specified, and training images of the target product can be captured in order and registered. Multiple training images can be registered for one product. Furthermore, new training images can be automatically generated from previously registered training images using AI deep learning and registered.

[0055] The group information is information that specifies a group of products to which product images captured by the imaging unit 111 are associated as learning images. That is, for products set in a predetermined group, the learning image of one product belonging to the predetermined group is also associated with other products belonging to the predetermined group. For example, if there are "apples" that are the same fruit variety but are sold in different regions or brands, when these are grouped, the "apples" from each region or brand are treated as belonging to the same group, regardless of their differences in region or brand. Then, as described below, when a learning image is associated with one product belonging to the same group, the same learning image is also associated with products other than the one product belonging to the same group.

[0056] The sales target setting is information indicating whether the store will treat an item as a sales target or a non-sale target, and is set by the setting unit 126, which will be described later. The products handled by the weighing device 1, i.e., the products registered in the product file 121, include seasonal items and items that are not available year-round. Therefore, it is not appropriate to display products that are not currently on sale on the screen during weighing, and excluding them from the object scan improves the accuracy of the object scan. This sales target setting can be performed by calling up a specific product from a list of products by calling up its product number, but in this embodiment, the setting unit 126, which will be described later, can be used to select a specific product from the candidate products extracted by the extraction processing unit.

[0057] Note that the sales target setting is not limited to an example in which an operator designates a product and sets it as a sales target or non-sale target. It may also be possible to set a product to be automatically treated as a sales target or non-sale target according to a preset period, etc. For example, similar to a sales period set for a product in a special sale promotion, a sales period, such as from one predetermined date to another, may be set for each product, and the product may be treated as a sales target or non-sale target depending on whether the date and time of use of the weighing device 1 falls within the sales period. That is, the sales period for each product is set in the product file 121, and the weighing device 1 acquires calendar information stored on the weighing device itself or an external terminal when powered on. The weighing device 1 refers to the calendar information to determine whether the current date and time for a specific product falls within the sales period. If the current date and time falls within the sales period, the specific product is treated as a sales target or non-sale target. If the current date and time falls within the sales period, the specific product is treated as a sales target or non-sale target.

[0058] The sales period can also be set according to the season. For example, the product file 121 can set the season in which a product is to be sold, and a separate seasonal master that associates dates with seasons can be created. The weighing device 1 can then refer to the seasonal master to determine the season of the day and set the product as a sale item or a non-sale item.

[0059] Here, the season set for a product does not have to be one of spring, summer, autumn, or winter; overlapping seasons, such as spring, summer, and autumn and winter, are also possible. In this case, the product may be treated as a sales item as long as it falls into one of the overlapping seasons. Alternatively, a pop-up screen may be displayed, allowing the operator to select which of the set seasons the product will be treated as a sales item or not. The seasonal master may also associate the season with the location information of the store in which the weighing device 1 is installed. This may allow seasonal settings to be reflected depending on the location, such as spring in Hokkaido and summer in Okinawa, even on the same date and time. The seasonal sales setting may be switched on / off by any operation, such as a predetermined button, allowing the user to set whether or not the seasonal sales setting should be reflected on the product on that day. The seasonal master may also refer to a category set in the product file 121.

[0060] The sales period can also be set on a monthly basis. For example, a flag indicating whether a product is to be treated as a sale item in a specific month can be set in the product file 121. The weighing device 1 determines the month of the current day based on the date of the current day, and sets the product as a sale item or a non-sale item according to the flag.

[0061] It should be noted that the sales period may be set by both the season and the month, so that the operator can selectively set either one.

[0062] <Weighing Control Unit 122 > The weighing control unit 122 controls the weighing unit 109 to weigh the weight of the product placed on the weighing pan of the weighing unit 109 .

[0063] <Change Detection Unit 123> The change detection unit 123 is a functional unit that detects a change in the weight on the weighing pan being measured by the weighing unit 109. The change detection unit 123 determines that the weight has changed, for example, by detecting a change in weight of a predetermined value or more within a predetermined time period. The change detection unit 123 may also calculate the amount of change in weight of the product being measured by the weighing unit 109 and determine that the weight has changed if the amount of change is greater than or equal to a predetermined value. The change detection unit 123 may also calculate the amount of change in weight as one indication of a weight change and detect weight fluctuations. The change detection unit 123 can also detect whether the weight being measured is stable. A stable weight state refers to, for example, a state in which the weight value of the product measured by the weighing unit 109 is within a predetermined threshold value for a predetermined short period of time. When this change detection unit 123 detects a change in weight of the weighing unit 109, it is assumed that a product has been placed on the weighing unit 109, and processing is initiated from capturing an image of the product by the imaging unit 111 to extracting the product by the extraction unit 125.

[0064] <Image Capture Control Unit 124> The image capture control unit 124 controls the image capture unit 111 to capture images of products placed on the weighing unit 109. The image capture control unit 124 may start image capture for object scanning when the change detection unit 123 detects a weight change in the weighing unit 109, or may start image capture for object scanning when the weight of the product weighed by the weighing control unit 122 falls within the transaction weighing range. For example, a value of 20 or more divisions from the minimum scale is treated as the transaction weight. The operation for object scanning may also be started in response to an optional operation by the operator.

[0065] <Extraction Unit 125> The extraction unit 125 is a functional unit that extracts product information from the product file 121. For example, the extraction unit 125 extracts product candidates corresponding to product images captured by the imaging unit 111 from the product file 121, or extracts product information from the product file 121 in response to a search such as a product number search. The extraction unit 125 recognizes products in the image captured by the imaging unit 111 through so-called object scanning using appropriate image analysis technology or general object recognition technology. Note that general object recognition may be performed by dividing the image into regions for each object when recognizing the products. The number or type of products placed on the weighing unit 109 may also be identified.

[0066] The extraction unit 125 uses a learning image associated with each product in the product file 121 as a reference image, compares the reference image with the product image in the captured image recognized by object scanning, and calculates the similarity. The similarity indicates the degree to which all or part of the product images are similar, for example, when the learning image of each product stored in the product file 121 is 100% (i.e., "similarity: 1.0"). The extraction unit 125 then extracts multiple products from the product file 121 as product candidates corresponding to the captured product image, for example, in order of similarity. Information related to the extracted product candidates is passed to the display control unit 132. Because multiple products are extracted as candidates, the likelihood of an appropriate product being extracted increases. As a result, the operator is not forced to re-execute the extraction process because an appropriate product candidate was not extracted. Extracting multiple product candidates is particularly useful when the operator is a customer, because redoing the process can be confusing and the increased operational effort can be inconvenient.

[0067] The extraction unit 125 may calculate the similarity between the captured product image and the learning images of each product registered in the product file 121 as an absolute evaluation or a relative evaluation. Regarding the extraction of multiple products in order of similarity, various extraction methods may be set. For example, if the similarity of the product with the highest similarity is lower than a predetermined value, three products may be extracted as candidates in order of similarity. Furthermore, if the similarity of the product with the highest similarity is 70% and multiple products are within a similarity range of -5% (65%) from that, all products within that range may be extracted as candidates. Furthermore, it is also possible to set the extraction unit 125 to extract non-sale products as candidates if the non-sale products have a higher similarity than the product with the highest similarity, or if the non-sale products are within a ±3% range of the product with the highest similarity.

[0068] Furthermore, the extraction unit 125 may start the extraction process in response to a change in weight without waiting for the weight of the weighed product to stabilize, or may start the extraction process when a change in weight is detected. Alternatively, the extraction unit 125 may perform the product extraction process when the weight measured by the weighing control unit 122 is stable. Furthermore, the extraction unit 125 may identify the color of the product from the captured image of the product during object scanning and compare it with the registered product colors to identify products with similar colors. The extraction unit 125 may use color as one of the similarity indices and select captured products with similar colors as candidates. Note that the product color may be identified from the image, or may be registered in advance in the product file 121. The color similarity may be calculated using, for example, elements such as hue, lightness, or saturation, or a combination thereof, or any other technique, such as calculation by projecting onto an appropriate coordinate system, may be used.

[0069] Furthermore, the extraction unit 125 may function as follows, particularly in each operation mode. In the first operation mode, the extraction unit 125 refers to the product file 121 based on the product image captured by the imaging unit 111 and extracts product candidates captured by the imaging unit 111, i.e., product candidates placed on the weighing unit 109 as products eligible for purchase. In this case, the extraction unit 125 may first extract product candidates corresponding to the captured product image from products set as products eligible for sale. Since products not eligible for sale are not normally displayed, the accuracy of extracting appropriate candidates can be increased by extracting product candidates only from products eligible for sale. On the other hand, if a product cannot be extracted from products set as products eligible for sale, the extraction unit 125 may further extract product candidates corresponding to the captured product image from products set as products not eligible for sale.

[0070] In the second operation mode, the extraction unit 125 extracts candidate products to which learning images are to be added by referring to the product file 121 based on product images captured by the imaging unit 111. In this case, the extraction unit 125 may extract candidate products corresponding to the images captured by the imaging unit 111 from both products set as sale items and products set as non-sale items, i.e., from all products regardless of sale item settings. Furthermore, if candidate products to which learning images are to be added cannot be extracted based on the images captured by the imaging unit 111, the extraction unit 125 may extract candidates for products to which learning images are to be newly registered by registering learning images from products that do not have learning images registered in the product file 121, i.e., products that do not have learning images associated with them. Furthermore, if candidate products cannot be extracted because no learning images are registered in association with the product or the amount of learning is low, i.e., there are few learning images associated with the product, a message indicating "unlearned" may be displayed on the screen. Furthermore, the extraction unit 125 may determine the current state and notify the operator of the state, such as by displaying a message on the screen indicating that no learning images have been registered or that there are few learning images registered.

[0071] In the third operation mode, product designation can be accepted from a state in which product candidates have been extracted in the first operation mode or the second operation mode, but regardless of this, product candidates for which sales target setting is to be performed may be extracted by referring to the product file 121 based on product images captured by the imaging unit 111, or product candidates may be extracted based on a product number search, etc. In this third operation mode, to perform sales target setting of products, the extraction unit 125 extracts product candidates from all products regardless of the sales target setting.

[0072] In the fourth operation mode, any multiple products are extracted by searching for product numbers or the like from all products registered in the product file 121. However, when specifying each of the multiple products, product candidates corresponding to the product image captured by the imaging unit 111 may be extracted, as in the other operation modes.

[0073] In addition, in any operation mode, in the extraction process by the extraction unit 125, of all products including products for sale and products not for sale, products that do not have associated learning images for object scanning, i.e., products for which learning images are not registered in the product file 121, may be excluded from the targets of object scanning, i.e., extraction targets.

[0074] <Setting Unit 126> The setting unit 126 sets whether a product registered in the product file 121 is a sale product or a non-sale product. In the description of this embodiment, the setting by the setting unit 126 to set a product as a sale product or a non-sale product may be referred to as a "sale target setting."

[0075] Here, the setting process by the setting unit 126 is executed in the third operation mode, but can also be executed during execution of the first operation mode. That is, during execution of the first operation mode in which a product is registered as a purchase target product, the extraction unit 125 extracts product candidates. When the user selects a change to the third operation mode while the product candidates are displayed, the first operation mode is temporarily suspended and the operation mode transitions to the third operation mode. In this third operation mode, the products extracted by the extraction unit 125 can be set as sales target products or non-sale target products. While operations are being executed in the third operation mode, operations performed in the first operation mode before the operation mode was changed to the third operation mode remain valid. When the sales target setting is completed in the third operation mode and the operation mode returns to the first operation mode, the operations that were valid can be continued. Note that, as described below, the operation mode may be transitioned from the second operation mode or the fourth operation mode to the third operation mode in the same manner as the transition from the first operation mode to the third operation mode.

[0076] As described above, the operations in the first operation mode that remain valid when switching to the third operation mode include, for example, product weights and operations and information related to product candidates. However, as a result of a change in the sales target setting in the third operation mode transitioned from the first operation mode, the products to be extracted as candidates may change depending on the product similarity. Therefore, when returning from the third operation mode to the first operation mode, the extraction process by the extraction unit 125 may be executed again. When the extraction process is executed again in this manner, if a new product candidate to be extracted based on the similarity arises due to a change in the sales target setting, this new product candidate may be extracted and displayed in addition to the product candidates previously extracted in the first operation mode, or a predetermined number of product candidates may be displayed in order of similarity from the previously extracted products and this new product candidate. In either case, in the first operation mode after switching back to the first operation mode, the results of the new extraction process are reflected in the results of the extraction process in the previous first operation mode, and the first operation mode can be said to be maintained valid. However, regardless of this, when switching from the first operation mode to the third operation mode and then returning to the first operation mode again, the processing results of the previous first operation mode can be erased, and the extraction processing by the extraction unit 125 or the series of processing starting from image capture can be executed from the beginning.

[0077] Furthermore, in the first operation mode, when the extraction unit 125 extracts product candidates from the product file 121 based on product images captured by the imaging unit 111, if the product candidates are extracted from products set as products eligible for sale and no product candidates are extracted from products not eligible for sale, the sales target setting by the setting unit 126 can also be said to be a setting (extraction target setting) in the first operation mode as to whether or not the extraction unit 125 will extract candidates for each product.

[0078] <Registration Unit 127> The registration unit 127 registers a training image of a product in the product file 121. Specifically, in the second operation mode, the registration unit 127 registers an image of a product captured by the imaging unit 111 as a training image in the product file 121 in association with a product designated by the designation receiving unit 131 (described later). Here, if a training image has not yet been registered for the product to be associated, a new registration is performed, whereas if a training image has already been registered, an additional registration is performed. Note that the registration destination for the training image may be a file provided separately from the product file 121, such as a file (storage area) for registering training images, as long as the training image is associated with the product.

[0079] In addition, if the product for which designation has been accepted is one of the products grouped by the grouping unit 128, the registration unit 127 registers the image of the product captured by the imaging unit 111 as a learning image in the product file 121 in association with all products grouped with the product designated by the designation accepting unit 131.

[0080] In the initial stage of registering learning images, no learning images are registered. At this stage, a product is retrieved, for example, by product number search, and the product image is registered as a learning image. As the registration of learning images progresses from the initial stage to a certain extent, product candidates can be extracted by object scanning. Simply selecting a specific product from the product candidates displayed corresponding to the captured product image associates the learning image with the product, and learning progresses. In this regard, regardless of the progress of learning, if a product candidate completely different from the captured product is extracted as a result of object scanning, the captured product image is determined to be unsuitable as a learning image. Therefore, a cancel button may be provided to "not register" or "skip" the captured product image as a learning image, thereby canceling the registration. In this case, instead of or in addition to providing a cancel button, the process of registering the captured product image as a learning image may be canceled if the product is removed from the weighing unit 109 without performing an operation to select the product. This avoids the situation where a learning image that will later be subject to deletion as an irregular image is registered, unlike when learning images are registered based on product numbers. In the first place, as the number of registered learning images increases and accuracy improves, only products similar to the imaged product will be extracted as candidates, so images of products with completely different characteristics from the imaged product will no longer be associated with the wrong product.

[0081] <Grouping Unit 128> In the fourth operation mode, the grouping unit 128 groups predetermined products and registers the setting in the product file 121. In the description of this embodiment, the setting for grouping products by the grouping unit 128 may be referred to as a "group setting."

[0082] <Calculation unit 129> The calculation unit 129 calculates a price based on the weight and unit price of the product by referring to the product file 121. Note that the calculation unit 129 performs calculations necessary for selling the product, such as calculating the tax-inclusive price based on the tax rate of consumption tax or the like, as appropriate.

[0083] <Printing Control Unit 130> The printing control unit 130 issues labels reflecting information such as the weight measured by the weighing unit 109 and the price calculated by the calculation unit 129. That is, the printing control unit 130 controls the printing unit 110 to print information necessary for product payment on labels, etc., based on the product information about the product in the product file 121, the weighing value output as the weighing result from the weighing unit 109, and the price calculated by the calculation unit 129. Note that instead of or in addition to printing the information necessary for product payment on labels, the printing control unit 130 may print the information on bags containing the products. The printing control unit 130 may present the information to be printed to the operator and wait for confirmation that the presented information is correct before issuing a label, or it may issue a label immediately upon generation of the information to be printed without waiting for confirmation. The information necessary for product payment may also be associated with a recording medium capable of identifying a member or a transaction, such as an IC tag or barcode (membership card). Before starting measurement or immediately before issuing a label, an input screen for identification information of the recording medium may be displayed to prompt the operator to input the information.

[0084] <Designation Accepting Unit 131> The designation accepting unit 131 is a means for displaying product candidates extracted from the product file 121 as product candidates corresponding to product images captured by the imaging unit 111 and accepting designations from the operator. This designation accepting unit 131 is displayed on a screen operated by the operator, for example, and is realized by an operator provided for each product. For example, as described below, this operator may be displayed as a preset key G221 (see FIG. 5, etc.) displaying product information. This allows the operator to easily identify a product and specify a product with a simple operation, even if multiple product candidates are extracted.

[0085] In a first operation mode, the designation accepting unit 131 accepts designation of a product for which a label is to be issued from the product candidates extracted by the extraction unit 125. In a second operation mode, the designation accepting unit 131 accepts designation of a product for which a learning image is to be registered from the product candidates extracted by the extraction unit 125. Note that if a learning image is not yet associated with the designated product, a new registration is performed, and if a learning image is already associated with the designated product, an additional registration is performed. In a third operation mode, the designation accepting unit 131 accepts designation of a product for which a setting of a sale product or a non-sale product is to be switched from the product candidates extracted by the extraction unit 125. In this third mode, the product setting may be switched to a sale product or a non-sale product in response to receiving an operation to designate the product. In a fourth operation mode, the designation accepting unit 131 accepts designation of products to be grouped from the product candidates extracted by the extraction unit 125. Note that in the fourth operation mode, group registration may be requested by specifying multiple products from the product candidates and then executing an operation to request registration. In this case, the product may be selected or deselected depending on the product designation until the operation requesting registration is executed. The operation requesting registration may be performed in various ways, such as by pressing a separate button on the screen or a completion key on a mechanical key.

[0086] In addition, when products are grouped, when accepting the designation of a target for registering a learning image from among the candidate products, the designation may be accepted based on the product code of the product, or a predetermined product among the grouped products may be displayed as a representative product and the designation may be accepted. The representative predetermined product may be, for example, the product corresponding to the product with the smallest product code number when the products are grouped, or the product selected first when the products are grouped, and is not particularly limited. Furthermore, when registering a learning image, only one product among the grouped products may be displayed as a candidate product, and in this case, the one product may be the aforementioned representative predetermined product.

[0087] <Display control unit 132> The display control unit 132 is a functional unit that displays various information, such as information about products extracted by the extraction unit 125, on the display touch panel 104. For each product extracted as a candidate product corresponding to the product image captured by the imaging unit 111, the display control unit 132 displays an operator representing the product on a specification screen (described later). This operator is used to receive designation from the user for each extracted product and to set sales targets using the setting unit 126, and is displayed, for example, as a preset key G221 (see FIG. 5, etc.) on which product information is displayed.

[0088] When the display control unit 132 displays on the selection screen the controls for multiple products extracted by the extraction unit 125 as candidate products for the captured image, the product candidates indicated as controls may be displayed, for example, in the order of product numbers stored in the product file 121. The product candidates may also be displayed in the order of similarity calculated by the extraction unit 125. A predetermined number of candidate products with high similarity may also be displayed in the order of product numbers. Highly similar product candidates may be displayed in a different manner from other candidate products. This can guide the user to select a highly similar product and allows the user to select a product without performing any special operations even if the accuracy of extraction by the extraction unit 125 decreases. Note that if multiple highly similar product candidates have the same similarity, the display manner may not be different even if other products with low similarity are displayed. This is to guide the user to make a more proactive decision, since the accuracy of extraction is considered low when there are candidate products with the same similarity.

[0089] When displaying product candidates indicated by operators in order of similarity, the order does not need to be determined strictly by similarity. For example, product candidates with a similarity of 90% or more may be displayed without distinguishing display modes as product candidates with the same similarity. Furthermore, if the extraction unit 125 extracts only products with a similarity of less than 90% and 70% or more, the operators for all extracted products may be displayed without narrowing down the candidates in order of similarity. This increases the likelihood that at least one of the operators displayed on the selection screen is appropriate, eliminating the need for the operator to specify a product using a special operation.

[0090] Furthermore, if only products with a similarity below a predetermined level, for example, less than 70%, are extracted as product candidates, the product controls may not be displayed, and a store clerk call screen or a message requesting that a different item of the same product be placed may be displayed. This is because there is a risk that a product that is not on display may have been placed on the weighing unit 109, or there may be a malfunction in the object scanning function, such as dust on the lens of the imaging unit 111 or a malfunction in the imaging unit, and therefore inspection is required before continued use.

[0091] Furthermore, if the extraction unit 125 is unable to extract product candidates, the display control unit 132 may display a product search screen. From the search screen, products registered in the product file 121 can be searched for, for example, by product name, product number, department, or classification. Product names and product numbers may be searched for by either an exact match or a partial match. The operator can specify any product from the search results. Furthermore, if the extraction unit 125 is unable to extract product candidates, all products may be displayed on the specification screen to allow the operator to specify a product.

[0092] This configuration allows the operator to easily specify a product even if a product training image is not stored in the product file 121 or if the extraction unit 125 is unable to extract the product due to reasons such as inability to recognize the product using image analysis technology or general object recognition technology. The display control unit 132 may display all product controls by any classification, such as department. The classification may be by color, or by type, such as fruits or vegetables. The display of controls corresponding to each classification may be performed using a display area or display page that can be switched by selecting a tab. Furthermore, when a product is object-scanned during an operation to purchase the product, the product image obtained by the object scan may be registered in the product file 121 as a training image. Whether or not such a setting is performed may be determined during setup as an optional process.

[0093] <Screen Example (First Operation Mode)> Here, an example of a screen displayed by the display control unit 132 will be described. Fig. 4 shows a screen G10 (standby screen) in the first operation mode, in which no product has been placed on the weighing unit 109 and the weighing unit 109 is waiting to perform weighing and pricing. In the center of this screen G10, the words "Place the product in the center of the scale and start" and an illustration showing this state are displayed, thereby prompting the operator to place the product on the weighing unit 109.

[0094] A product information display area G21 is provided at the top of the screen G10. The product information display area G21 displays the product's tare weight, weight (or number of items), unit price, and price. The weight field reflects the weighing value measured by the weighing unit 109. To display the number of items, the number can be calculated from the weight and unit weight. The tare weight and unit price are read from the product file 121 when product candidates are extracted by the extraction unit 125 and an operator specifies a product. When an operator specifies a product, the price is calculated and displayed by the calculation unit 129 based on the tare weight, weight (or number of items), and unit price. In the illustrated example, no product is placed on the weighing unit 109, so all values ​​are zero.

[0095] 5 shows a screen G11 in the first operation mode in which a product is placed on the weighing unit 109 and product candidates extracted by the extraction unit 125 are displayed. In addition to the product information display area G21 described above, the screen G11 is provided with a preset key display area G22, tabs G23, a character input area G24, a virtual keyboard G25, a numeric input area G26, and a numeric keypad G27.

[0096] In the product information display area G21, the weight of the product placed on the weighing unit 109 is displayed as the weight.

[0097] In the preset key display area G22, a preset key G221 serving as an operator for accepting product specification is displayed in a pressable manner for each product candidate extracted by the extraction unit 125 in accordance with the product image captured by the imaging unit 111. In the first operation mode, specification of a product for which a label is to be issued is accepted in response to pressing of this preset key G221.

[0098] In the first operation mode, the extraction unit 125 extracts only products that are set as sale items. Therefore, in the preset key display area G22, only preset keys G221 corresponding to products that are set as sale items are displayed, and those corresponding to products that are set as non-sale items are not displayed. Furthermore, if the products extracted by the extraction unit 125 include products that are set as part of a group, the extraction unit 125 may also extract other products that are set as part of the group, and display the preset keys G221 of the other extracted products in the preset key display area G22. However, even in this case, it is preferable not to display products that are set as non-sale items.

[0099] Furthermore, in the preset key display area G22, multiple product candidates extracted by the extraction unit 125 are displayed using preset keys G221. However, if candidate products differ in similarity (range of similarity) with the learning image, the preset keys G221 may be displayed in a display mode that distinguishes the similarity. For example, if there is one product with a similarity of 90% or more and two products with a similarity of less than 90%, the product with a similarity of 90% or more may be displayed in a different mode from the other products to guide the operator's selection. On the other hand, if there are two or more products with a similarity of 90% or more, the display mode may not be changed. Furthermore, if there are only candidate products with a similarity of less than 90%, the display mode of the preset keys G221 may be similarly left unchanged, or all preset keys G221 may be displayed without narrowing down the candidates by similarity.

[0100] The display (arrangement) of the preset keys G221 may be in ascending or descending order of product code, etc., or in order of similarity. Furthermore, if there is one product with a similarity of 90% or more as described above, the display columns of the preset keys G221 for that product and other products may be arranged in two rows. That is, if there is one product with a similarity of 90% or more, the preset key G221 for that product may be displayed in the first row, and the others may be displayed in the second row, or may be spaced apart by the length of one preset key G221. Furthermore, if only products with a similarity below a certain level are extracted as candidates, or if no object scan results are obtained, it is possible that an off-display product may have been placed on the weighing unit 109 due to vandalism or other reasons. Therefore, the operator may be notified to place a different product of the same type on the weighing unit 109, or the weighing unit may be stopped from operating, or a store clerk may be called. The notification may be provided in various ways, such as by displaying a message on the screen, illuminating a sign light, or outputting a sound or buzzer.

[0101] Furthermore, while the illustrated preset key G221 displays the name of the product, other product information registered in the product file 121, such as unit price, may also be displayed as appropriate. Furthermore, a button or operation for transitioning to a screen for editing product information may be associated with the preset key G221 so that the product information displayed on the preset key G221 can be edited. In this case, it is preferable that the screen be configured to return to the original product designation screen once editing of the product information is complete. Such buttons and operations may also be provided in other operation modes.

[0102] The tab G23 consists of a tab G231 that accepts an operation mode switching operation and a tab G232 that allows a classification for each product to be specified. The tab G231 accepts an operation mode switching operation, but the switching operation is not enabled when a product is being weighed and priced, and the switching operation is enabled when, for example, a store clerk or the like performs authorization authentication as described below. When the switching operation is enabled, the operation mode can be switched from the first operation mode to another operation mode, such as the third operation mode, by pressing the tab G231.

[0103] When a predetermined tab G232 is selected, the preset keys G221 of the products classified in the predetermined tab G232 are displayed in the preset key display area G22, or the preset keys G221 displayed in the preset key display area G22 are narrowed down to those related to the products classified in the predetermined tab G232. Note that, in the drawing, only one tab G232 is provided that can specify the classification for each product, but multiple tabs may be provided depending on the classification.

[0104] The character input area G24 is an area for inputting character strings of product information using a virtual keyboard G25 that displays the Japanese syllabary. When a product name is input into the character input area G24, a search process or a narrowing-down process is executed by referring to the product file 121, and a preset key G221 corresponding to the input product name is displayed in the preset key display area G22, or preset keys G221 already displayed in the preset key display area G22 are narrowed down to those corresponding to the input product name. Note that the character input area G24 may be configured to allow the input of character strings of the product color as product information. For example, by inputting "green" into the character input area G24, a preset key G221 of a product associated with the color green is displayed in the preset key display area G22.

[0105] The numeric input area G26 is an area for inputting numeric values ​​related to product information, such as a product code, from the numeric keypad G27. When a product code is input into the numeric input area G26, a preset key G221 corresponding to the input product code is displayed in the preset key display area G22, or the preset keys G221 already displayed in the preset key display area G22 are narrowed down to those corresponding to the input product code.

[0106] In this screen configuration, the character input area G24 and the virtual keyboard G25, and the numeric input area G26 and the numeric keypad G27 all constitute a search means for searching for a desired product and a narrowing-down means for narrowing down the search results to products that meet predetermined conditions. This makes it possible to call up products that are not displayed as preset keys G221 in the preset key display area G22.

[0107] In this first operation mode, when a predetermined preset key G221 is selected from the preset key display area G22, the contents are confirmed and the screen transitions to a label issuance confirmation screen for printing a label. On the label issuance confirmation screen, for example, the product's tare weight, weight, unit price, and price are specifically displayed in the product information display area G21, which is provided in the same manner as screen G11. After confirming the contents, if the operator wishes to print a label, he or she presses a button requesting label issuance. The printing control unit 130 then controls the printing unit 110 to print the label, and the label is then printed. If the operator wishes to cancel label issuance, the operator may press a cancel button or a button to return to the product selection screen. Label issuance may also be canceled by removing the product from the weighing unit 109. Regardless of this example, instead of or in addition to printing a label, information about the specified product may be sent to an external device as registration data. When the label printing is completed and the product is removed from the weighing unit 109, the weighing value becomes zero and the screen returns to screen G10, which is on standby so that the next product label printing can be executed.

[0108] Furthermore, if the change detection unit 123 detects a change in the weight measured by the weighing unit 109 during processing before the designation of the preset key G221 is accepted, the product extraction process and, depending on the settings and circumstances, the process may be executed again from the image capture, and the product candidates may be redisplayed and updated. Furthermore, if the preset key G221 is designated at a timing before the product candidates are redisplayed in this way, it may be treated as if a product was designated from the product candidates extracted by the process executed before the reprocessing.

[0109] <Screen Example (Second Operation Mode)> Figure 6 shows screen G12 when registering a captured product image as a learning image in the second operation mode. This screen G12 is an example of a screen displayed when a product is placed on the weighing unit 109 and object-scanned in a situation where the learning image has been registered to the extent that it can be object-scanned. Screen G12 displays "Validation Mode" (the system name corresponding to the second operation mode), allowing the operator to understand that the operation mode is the second operation mode. Note that the difference between the operation mode and other operation modes may be recognized by the background color, the display mode of the preset key G221, or other text. However, by making the screen configuration similar to that of the first operation mode, the operator can perform operations with roughly the same feel as in the first operation mode.

[0110] In the second operation mode, the designation of a product to which a learning image is to be added is accepted in response to pressing of a preset key G221 displayed in the preset key display area G22. When the designation of a product is accepted by the preset key G221, the learning image is associated with the designated product and registered in the product file 121.

[0111] Furthermore, if the products extracted by the extraction unit 125 are grouped, only the preset key G221 of a representative product among the grouped products may be displayed in the preset key display area G22. Even if only the preset key G221 of the representative product is displayed, when the designation of the preset key G221 of one product is accepted, learning images are also associated with the other grouped products, so that learning images can be associated with a group of products that are indistinguishable from one another. Note that the preset keys G221 displayed in the preset key display area G22 in the second operation mode are not limited to those corresponding to products set as products for sale, but also include those set as products not for sale. Furthermore, the preset keys G221 displayed in the preset key display area G22 may be displayed in order of product similarity or in ascending or descending order of product code, as in the first operation mode described above.

[0112] In the second operation mode, pressing the tab G231 switches the operation mode from the second operation mode to the first operation mode, and the tab G231 displays an image that enables the user to recognize that the operation mode is being switched.

[0113] Note that when a product is object scanned at an early stage before a learning image of the product is registered, there is no learning image to compare with the product image in the captured image. Therefore, the screen may display, for example, preset keys G221 for all products registered in the product file 121. In response, the operator designates the preset key G221 for the product for which the captured product image is to be registered as a learning image. This associates the captured product image with the product designated as the learning image and registers it in the product file 121. In this case, the screen displaying the preset keys G221 for all products registered in the product file 121 may be configured with multiple pages that can be switched using tabs. In this case, the operator can browse products categorized by a specific type by switching between tabs and select a specific product from among them. Furthermore, the screen may be configured to allow for narrowing down the search by product name search or by selecting a department or classification (category). Instead of displaying the preset keys G221 for all products registered in the product file 121 on the screen, product candidates may be displayed as preset keys G221 in response to a search such as a product name search.

[0114] Furthermore, when a product is imaged by the imaging unit 111 and product candidates are extracted by the extraction unit 125, even if the product for which a learning image is to be registered is not included in the candidates, all products registered in the product file 121 may be displayed as preset keys G221. Then, when a specific preset key G221 is designated, a learning image is associated with the corresponding product and registered in the product file 121. Note that even in this case, as in the case of registering learning images in the initial stage as described above, the preset key display area G22 may be configured to allow the user to narrow down the products by switching tabs, etc., and tabs may be switched depending on whether or not a learning image has been registered, in addition to departments and classifications (categories). This allows a learning image to be registered by specifying a product, regardless of whether or not a learning image has been registered.

[0115] In this second operation mode, when a predetermined preset key G221 is designated from the preset key display area G22, the screen does not transition to the label issuance confirmation screen as in the first operation mode, but instead displays a message indicating that the learning image has been registered. The message may be displayed on the screen or output as audio. It may also be transmitted to another higher-level terminal such as the store controller 2. After the message has been displayed for a predetermined period of time, or after an operation such as pressing a button to accept the message, the screen returns to the screen waiting for a product to be placed on the weighing unit 109 so that the learning image can be registered.

[0116] <Screen Example (Third Operation Mode)> Figure 7 shows screen G13 when setting product sales targets in the third operation mode. Screen G13 displays "Status Mode" (the system name corresponding to the third operation mode), allowing the operator to understand that the operation mode is the third operation mode. Note that in this example, as with the screen G12 described above, the difference between this operation mode and the other operation modes may be indicated by the background color or the display mode of the preset key G221.

[0117] In the third operation mode, the designation of a product is accepted by pressing a preset key G221 displayed in the preset key display area G22, and the sales target setting of the product is switched. That is, in the third operation mode, the preset key G221 also serves as an operator for setting the sales target by the setting unit 126, and each time the preset key G221 is pressed, the designated product is set as a sales target product or a non-sale target product, and the setting is registered in the product file 121.

[0118] Here, the display mode of the preset key G221 may differ depending on whether the corresponding product is set as a sale product or a non-sale product. That is, in the example shown, the preset key G221-1 indicating "kiwi," which is set as a sale product, is displayed in blue, and the preset key G221-2 indicating "lemon," which is set as a non-sale product, is displayed in yellow. When the preset key G221 is pressed, a change in the sale target setting is registered in response to the press, and the color of the preset key G221 also changes to the corresponding color.

[0119] Note that this color scheme is merely an example, and any color scheme may be used as long as it allows for the distinction between on-sale products and off-sale products. Furthermore, the distinction is not limited to color, and various methods can be used as long as it allows for the distinction between on-sale products and off-sale products. For example, one of the products may be made to blink, or a design or wording that distinguishes the two may be added to either or both of them. Furthermore, on-sale products and off-sale products may be displayed in groups, which makes it easier to understand when there are many product candidates.

[0120] Furthermore, the display mode of the preset keys G221 displayed in the third operation mode may be different from that of the preset keys G221 displayed in the first operation mode. For example, in the third operation mode, the preset keys G221 for products set as products for sale are displayed in blue, and those not for sale are displayed in yellow, while the preset keys G221 displayed in the first operation mode are displayed in white. This makes it easy to understand in which operation mode the operation is being performed.

[0121] In the preset key display area G22, all preset keys G221 of the extracted products are displayed regardless of whether the products corresponding to the preset keys G221 are grouped or not, and even grouped products can be individually set. Furthermore, if grouping is set for the extracted products, all preset keys G221 of the grouped products may be displayed in the preset key display area G22.

[0122] The third operation mode may be switched to in response to an arbitrary operation by an operator such as a store clerk while the first operation mode is being executed. For example, if a product is imaged in the first operation mode and product candidates are displayed using the preset keys G221, and a preset key G221 for a specific product that is not actually for sale is displayed, the first operation mode is temporarily suspended and the system transitions to the third operation mode. In the preset key display area G22 of the transitioned third operation mode, the preset keys G221 displayed in the first operation mode are displayed in a manner that makes it possible to distinguish whether they are set as a product for sale or a product that is not for sale. The operator can change the setting of the specific product from a product for sale to a product that is not for sale and return to the first operation mode. As a result, the preset key G221 for the specific product that was previously displayed as a product for sale is no longer displayed. However, this does not prevent the third operation mode from being executed separately from the first operation mode. For example, the third operation mode can be executed by selecting the third operation mode from a menu screen independent of each operation mode.

[0123] Furthermore, in the third operation mode, pressing the tab G231 switches the operation mode from the third operation mode to the first operation mode, and the tab G231 displays an image that allows the operator to recognize that the operation mode has been switched. In this way, the target product for which sales target setting is to be performed can be specified from the candidate products extracted by object scanning, which reduces the amount of work required compared to searching for the target product based on its product number from the list of products registered in the product file 121 and setting it as a sales target. Furthermore, since sales target setting can be performed with an operation similar to that for registering learning images in the second operation mode described above, the operator will find the operation easy to use.

[0124] When a new product is displayed in a store and the product has not been displayed before, the store clerk needs to confirm whether the object scan is performed correctly after the product is displayed. This confirmation can be performed, for example, by checking whether the correct product candidate is displayed as a preset key G221 on the screen G11 after placing the product on the weighing unit 109 in the first operation mode. If several other items of the same species are placed on the weighing unit 109 and appropriate product candidates are extracted as expected, preparation is complete. On the other hand, if appropriate product candidates are not extracted, the sales target settings of the product must be confirmed. In this case, it is convenient for the store clerk or other operator to be able to transition to the third operation mode by performing authorization authentication on the weighing and pricing screen G11. The authorization may be authenticated, for example, by scanning the store clerk's barcode using the imaging unit 111, or by entering a password, and is not particularly limited. Furthermore, transition to the third operation mode is not limited to pressing the tab G231 described above, but may also be triggered by authorization authentication during the first operation mode. When the operator transitions to the third operation mode, as described above, the operator can check the sales target setting for each product candidate on screen G13, indicating whether the product is for sale or not, and can switch the sales target setting.

[0125] <Screen Example (Fourth Operation Mode)> Fig. 8 shows a screen G14 when grouping products in the fourth operation mode. "Grouping Mode" (the system name corresponding to the fourth operation mode) is displayed on the screen G14, so that the operator can understand that the operation mode is the fourth operation mode.

[0126] In the fourth operation mode, when multiple products are designated by pressing preset keys G221 displayed in the preset key display area G22 and an operation to request registration is performed, the grouping unit 128 groups the designated multiple products and registers the group setting in the product file 121. The number of designated preset keys G221 for products is not particularly limited, and the group setting is validly accepted as long as there are two or more. Furthermore, until the operation to request registration is performed, products can be selected and deselected according to the designation of the preset keys G221. The operation to request registration can be performed in various ways, such as by pressing a separate button on the screen or a completion key on a mechanical key.

[0127] Processing Flow Next, the flow of processing executed by the weighing device 1 according to this embodiment will be explained for each operation mode. When the weighing device 1 is powered on, it displays a menu screen from which an operation mode can be selected. When a predetermined operation mode is selected from the menu screen, a standby screen for that operation mode is displayed. It may be possible to return to the menu screen from the standby screen for each operation mode by performing a predetermined operation. However, a switching means such as an operation mode switching button may be provided on the screen for each operation mode, and it may be possible to switch from a predetermined operation mode to another operation mode by operating the switching means.

[0128] <Process Flow 1 (First Operation Mode)> Figure 9 shows the process flow in the first operation mode. When a product is placed on the weighing unit 109 on the standby screen shown in the example of screen G10 described above, the change detection unit 123 detects a change in weight on the weighing unit 109 (S101), and the above-mentioned screen G11 is displayed. Note that the weighing control unit 122 constantly performs weighing using the weighing unit 109, and the weighed value is displayed on screen G11. Furthermore, if a product is removed from the weighing unit 109 in the subsequent processing and the weight becomes zero, the screen returns to the standby screen G10.

[0129] The imaging unit 111 captures an image of the product placed on the weighing unit 109 in response to a change in weight (S102). When the weight measured by the weighing unit 109 stabilizes (S103), the extraction unit 125 extracts product candidates corresponding to the imaged product from the product file 121 (S104). Details of the extraction process will be described later.

[0130] The display control unit 132 displays the extracted product candidates as operators (preset keys G221) on the screen (S105). When a product is designated by pressing any of the operators, and the designation acceptance unit 131 accepts the designation (S106), the calculation unit 129 calculates a price according to the product weight (or the number calculated from the unit weight) (S107), and the price is displayed on the screen.

[0131] When the content of the label to be printed is finalized and an operation indicating that the content has been finalized is received from the operator (S108), the print control unit 130 controls the print unit 110 to print the label (S109).

[0132] 10 shows a detailed flow of the extraction process in S104 described above. First, the extraction unit 125 attempts to extract candidate products from the products set as products for sale by referring to the product file 121 based on the product image captured by the imaging unit 111 (S111). As described above, the extraction of candidate products involves calculating the similarity between the product image and the registered learning images. As a result, if a candidate product corresponding to the captured product can be extracted from the products set as products for sale, the candidate product is extracted as a candidate product (S112).

[0133] On the other hand, if a candidate product cannot be extracted from the products set as the sale products, the extraction unit 125 attempts to extract a candidate product from the products set as the non-sale products (S113).As a result, if a candidate product corresponding to the imaged product can be extracted from the products set as the non-sale products, the extraction unit 125 extracts the candidate product as a candidate (S112).

[0134] If a candidate product cannot be extracted from the products set as non-sale products (S113), error processing is executed (S114). The error processing may, for example, display a store clerk call screen to prompt the user to call a store clerk, or display a message requesting the user to place a different product of the same type. When the store clerk is called, the store clerk may search for the corresponding product by product number search, or if the product is placed on the weighing unit 109 again in response to a message requesting the user to place a different product of the same type, the above-described series of processes from S101 onward are executed again. Instead of such error processing, a preset key may be displayed together with a message such as "Please select from the preset key" to prompt the user to specify a product.

[0135] When the customer receives the label, they use it to pay at the POS terminal 3. The label is printed with a barcode and product code that encodes payment information including, for example, the product name, price, content volume corresponding to the weight, unit price, store name, price, etc., and the payment can be made based on the information printed on the label.

[0136] In addition, if a product is extracted from the products set as non-sale products in S113, this fact may be displayed on the screen, or the display mode of the preset key G221 may be changed. Alternatively, the product may not be displayed on the screen as a preset key G221, but may be displayed as a mark that only the store clerk can understand. In addition to or instead of a screen display, audio output, lighting or blinking of a sign lamp, or notification to another terminal may be output. In the illustrated example, if a product candidate cannot be extracted from the sale products, an attempt is made to extract a product candidate from the non-sale products. However, in other embodiments, if a product candidate cannot be extracted from the sale products, error processing (S114) may be performed without attempting to extract a product candidate from the non-sale products. In this case, if a product candidate can be extracted by targeting the non-sale products, a mark to this effect may be displayed on the screen in a manner that only the store clerk can understand. The notification method may be the same as that described above, or may be other methods.

[0137] Furthermore, if the error processing step of S114 is reached, it is possible that, in extracting candidate products corresponding to the captured image, there are products associated with learning images that are at least a certain level of similarity, but the similarity is not sufficient to display them as candidates, or no candidates could be extracted from the products associated with the learning images. Therefore, the situation may be notified to store clerks or other personnel in a manner that allows them to recognize each situation, such as by displaying a message or a mark. In this way, if it is determined that a learning image has not been registered, the corresponding product can be immediately retrieved by searching for a product number, without having to try to understand the product's status from a maintenance screen or other screen.

[0138] Step S104 may be performed simultaneously with step S103 or before the weight is stabilized. However, if step S104 is performed after the conditions of step S103 are satisfied, the orientation of the product is stabilized and the similarity can be calculated in a stationary state, ensuring higher identification accuracy.

[0139] <Process Flow 2 (Second Operation Mode)> Figure 11 shows the process flow in the second operation mode. Similar to the process (S101-S103) in the first operation mode described above with reference to Figure 9, when an operator, such as a store clerk, places a product on the weighing unit 109 for which the weighing device 1 is to learn, the change detection unit 123 detects a change in weight on the weighing unit 109 (S201), and the above-mentioned screen G12 is displayed. Then, the image capture unit 111 captures an image of the product placed on the weighing unit 109 in response to the weight change (S202), and the weighing unit 109 waits for the weight to stabilize (S203). Note that even in this second operation mode, the weighing control unit 122 continues weighing the product using the weighing unit 109, and the weighed value is displayed on screen G12. Furthermore, when the product is removed from the weighing unit 109 and the weight is reset to zero, the system returns to the standby screen.

[0140] When the weight on the weighing unit 109 stabilizes, the extraction unit 125 attempts to extract candidate products to which learning images are to be added based on the captured product image and by referring to the product file 121 (S204). As a result, if a product is identified that has an associated learning image that matches the captured product image at a similarity level equal to or higher than a predetermined level, the extraction unit 125 extracts the identified product as a candidate product to which learning images are to be added (S205).

[0141] On the other hand, if a product associated with a learning image that matches the captured product image at a similarity level equal to or higher than the predetermined level cannot be identified, and therefore a candidate product for which a learning image is to be added cannot be extracted, it is assumed that the captured product has not yet had a learning image registered, or that there are not enough learning images registered to enable effective object scanning, and products for which no learning image is associated are extracted as candidates for products for which a new learning image is to be added (S206). Note that in the processes of S205 and S206, the extraction unit 125 may attempt to extract candidate products from both products set as available products and products set as non-available products.

[0142] The display control unit 132 displays the extracted product candidates as operators (preset keys G221) on the screen (S207). When a product is designated by pressing any of the operators and the designation receiving unit 131 receives the designation (S208), the registration unit 127 acquires the captured image of the product as a learning image and registers it in the product file 121 in association with the designated product (S209).

[0143] <Processing Flow 3 (Third Operation Mode)> Figure 12 shows the processing flow in the third operation mode. Here, we will explain the flow of processing by switching the operation mode to the third operation mode while the first operation mode is being executed. When a switch of the operation mode is accepted (S301) in a state in which product candidates are displayed as preset keys G221 in step S105 according to the steps described with reference to Figure 9, the sales target setting of the product is read from the product file 121 (S302), and the sales target setting is reflected in the display mode of the preset keys G221 (S303).

[0144] The operator checks the sales target setting of the product from the display mode of the preset key G221, and when the operator determines that the product needs to be changed in setting, he / she requests the change of the designated sales target setting of the product using the preset key G221 (S304). In response to this, the setting unit 126 changes the sales target setting of the designated product and registers it in the product file 121 (S305). In response to the change of the sales target setting, the setting is reflected in the display mode of the preset key G221 of the target product (S306).

[0145] When an operation to end the third operation mode is performed, the operation mode switches to the first operation mode, and the label is printed via steps S106 and thereafter.

[0146] In the above example, the sales target setting is performed for each product by switching from the first operation mode to the third operation mode, but similar to the first operation mode and the second operation mode described above, it is also possible to image a product from a predetermined standby screen in the third operation mode, display the product preset key G221, and switch the sales target setting for each preset key G221. Furthermore, similar to the case where the operation mode is switched from a state in which the preset key G221 is displayed in the first operation mode to the third operation mode, it is also possible to switch from a state in which the preset key G221 is displayed in the second operation mode or a fourth operation mode described later to the third operation mode, and to set the sales target for each product displayed as the preset key G221.

[0147] <Process Flow 4 (Fourth Operation Mode)> Fig. 13 shows the process flow in the fourth operation mode. First, on a predetermined standby screen, products are read from the product file 121 and the preset keys G221 are displayed on the screen (S401). The products displayed as the preset keys G221 may be, for example, products extracted by a product number search or the like, or all products registered in the product file 121 may be extracted.

[0148] When the operator designates an arbitrary preset key G221 from the preset keys G221 displayed on the screen, the designation acceptance unit 131 accepts this (S402). When multiple preset keys G221 are designated and an operation to request grouping is performed by pressing a predetermined button on the screen (S403), the grouping unit 128 groups the designated multiple products and sets the group in the product file 121 (S404).

[0149] In addition, in the processing flow in any of the first to fourth operation modes, the order of steps can be changed as appropriate as long as no contradiction occurs.

[0150] The weighing device 1 according to the present embodiment described above allows for easy sales target setting for products extracted by object scanning. In particular, the extracted products are displayed as specifiable preset keys G221, allowing for easy updating of settings with a touch (press) operation, and the settings are displayed in a manner that is easy to understand at a glance, which is convenient. Furthermore, object scanning also extracts products similar to the captured product, which is useful because it allows for simultaneous updating of sales target settings for similar products. Furthermore, the settings can be made with the same operational feel as a normal transaction, eliminating the need to select target or non-target products from a product list.

[0151] Furthermore, for any product registered in the product file 121, learning images can be registered and the product can be set using a general-purpose and simple operation. In particular, since the process is performed by imaging the product and then specifying the product number (product), rather than by specifying the product number, imaging the product, and then inputting the image, it is possible to register learning images consecutively without switching product numbers. Furthermore, learning images are generated by the imaging unit 111 simply by placing the product on the weighing unit 109, and for products for which learning images have already been registered, candidate products for which learning images are to be registered are automatically extracted by object scanning, eliminating the need for time and effort when specifying the products for which learning images are to be registered.

[0152] Furthermore, even for products for which no learning images have been registered, even if the operator does not know whether they are registering a learning image for a product for which a learning image has been registered or not, only products for which no learning image has been registered are extracted as candidates based on the results of the object scan, making it easy to specify the product and immediately identifying that the learning image has not been registered. Furthermore, even if a product other than the intended product is accidentally captured during the process of registering a learning image, only a candidate product corresponding to the captured image is extracted, eliminating the risk of incorrectly registering the correspondence between the learning image and the product. This significantly reduces the need to delete a mistakenly registered learning image. Furthermore, the operation screen is not a special screen, and the user can operate the system while imagining a normal operation screen. Furthermore, since certain products are grouped, it is convenient to collectively register learning images for products with similar appearances.

[0153] Furthermore, sales target settings can be easily made for products extracted by object scanning. In particular, the extracted products are displayed as specifiable preset keys G221, allowing settings to be easily updated by touch (pressing) the keys, and the settings are displayed in a manner that is easy to understand at a glance, which is convenient. Furthermore, object scanning also extracts products similar to the captured product, which is useful because it allows sales target settings for similar products to be updated simultaneously. Furthermore, settings can be made with the same operational feel as normal transactions, and there is no need to go through the trouble of selecting target or non-target products from a product list.

[0154] In the above embodiment, if the weighing device 1 has the functionality of a POS terminal 3 or if the information processing device according to the present invention is configured as the POS terminal 3, the POS terminal 3 performs processes such as product extraction by object scanning, product registration as a purchase target product, and payment for registered products. The POS terminal 3 can be of various types, including a regular POS terminal where a store clerk performs operations from product registration to payment, a full-self-service POS terminal where customers register products and pay for them themselves, and a semi-self-service POS terminal where a store clerk registers products at a registration machine and customers pay at a payment machine. Each POS terminal has the same functional units as known POS terminals. That is, it includes a display / operation unit for displaying various information and performing operations, a reading unit for reading various codes to acquire information, a payment unit for accepting payments using various payment media and cash, and a communication unit for enabling data communication with the store controller 2, etc. In this case, the POS terminal 3 may not only register learning images but also, as described above, display multiple product candidates from captured product images as preset keys G221 and accept designation when registering products. Also, the POS terminal 3 may be configured to allow sales target settings to be performed in the same manner as described above.

[0155] In the above embodiment, learning images of products sold to customers are registered. However, the object scan target or the learning image registration target may be other items. For example, containers used in bulk sales (by weight) may be the target. In bulk sales, containers of various sizes are used to sell products such as prepared foods and fruit, and the stores may offer these containers to customers at different prices depending on the size. Furthermore, each container has a different tare weight depending on its size. In this case, it would be convenient to object scan the container when placing the product in the container, reflect the price of the container in the payment amount, or subtract the tare weight of the container from the measured value. Therefore, it would be useful to apply the above-described present invention to such situations, registering the price, tare weight, and learning image for each container size, and presenting candidate containers to the operator when registering images of containers of various sizes as learning images, allowing the operator to select any container.

[0156] Depending on the viewpoint, the present invention can be configured as a computer program or a method. The computer program can be provided by being stored on various data-readable recording media, or by being downloadable via a network such as the Internet. In other embodiments, the information processing device according to the present invention can be configured as a POS terminal or a display device as described above. However, any device that performs at least image processing, regardless of its name, can also constitute the information processing device according to the present invention.

[0157] Furthermore, these software resources can be distributed or consolidated into any of the hardware resources through appropriate design, and the hardware resources can also be configured as physically integrated devices or separate devices.

[0158] Specifically, for example, the weighing device 1 can be configured as an information processing system including a cloud server 4 positioned as a higher-level device of the weighing device 1. In this case, for example, the weighing device 1 includes at least a display touch panel 104, a weighing unit 109, an imaging unit 111, a weighing control unit 122, an imaging control unit 124, and a display control unit 132. The cloud server 4 also includes an extraction unit 125 that extracts product candidates based on product images captured by the imaging unit 111. Information required by the weighing device 1 and the cloud server 4 is transmitted and received via a network. That is, the weighing device 1 transmits to the cloud server 4 captured product images and information specifying products from the product candidates for which learning images are to be registered, and the cloud server 4 transmits information related to the product candidates extracted based on the captured product images to the weighing device 1.

[0159] Alternatively, the weighing device 1 transmits to the cloud server 4 images of the products and information regarding the operator's operations to set the products as sales targets, and the cloud server 4 transmits to the weighing device 1 information regarding candidate products extracted based on the images of the products.

[0160] <Additional Notes> An information processing device according to another embodiment of the present invention comprises an imaging means for imaging an item, an extraction means for extracting from a predetermined storage means an item corresponding to the item imaged by the imaging means based on a similarity using a learning image as a reference image, a designation receiving means for receiving designation of a predetermined item from one or more items extracted by the extraction means, and an acquisition means for acquiring, as the learning image, an image of the predetermined item designated by the designation receiving means and an item imaged by the imaging means, wherein if there is no learning image of the item imaged by the imaging means or if the extraction means cannot extract an item based on the learning image, the designation receiving means accepts designation from all items registered in the predetermined storage means, and if an item is extracted by the extraction means based on the image of the item imaged by the imaging means, it accepts designation from the extracted item.

[0161] The device may also be provided with a setting means for setting whether an item extracted by the extraction means based on a learning image of the item captured by the imaging means is to be a target or an exception to extraction by the extraction means in an operation mode for executing a transaction.

[0162] The device may be provided with a switching means for switching between a first mode and a second mode, and the switching means may cause the specification receiving means to, in the first mode, accept an instruction to acquire an image of the item captured by the imaging means as the learning image by the acquisition means, and, in the second mode, accept a setting as to whether the item should be a target or not to be extracted by the extraction means in the operation of executing the transaction.

[0163] Overview of the Embodiments The present invention relates to an information processing device, an information processing system, and a computer program.

[0164] 2. Description of the Related Art A so-called object scanning technique is known in which an image of an object is captured and compared with a learning image serving as a reference image for image recognition to identify the object.

[0165] For example, Japanese Patent Application Laid-Open No. 2017-162217 discloses an item identification device including a tray on which items are placed, a backlight source that illuminates the bottom of the tray with backlight, a digital camera that captures images of the items on the tray, and an identification computer that recognizes the images from the digital camera and identifies the items. This item identification device extracts the items on the tray from a differential image between an image taken when the backlight source is on and an image taken when the backlight source is off.

[0166] Image learning for object scanning requires capturing an image of each target product, specifying the product each time, and registering the image, which is extremely tedious.

[0167] Therefore, one of the objects of the present invention is to provide a device that can easily register learning images to be used in object scanning.

[0168] In order to achieve the above object, an information processing device according to one aspect of the present invention comprises an imaging means for imaging a product, an extraction means for extracting candidate products for which a learning image is to be added based on the image of the product imaged by the imaging means and by referring to a product information storage means for storing product information, a designation acceptance means for accepting designation of a product for which the learning image is to be added and registered from the candidate products extracted by the extraction means, and a registration means for registering the image of the product imaged by the imaging means as the learning image in the product information storage means in association with the product designated by the designation acceptance means.

[0169] If the extraction means is unable to extract a candidate product to which the learning image is to be added based on the image of the product captured by the imaging means, it may extract a product to which the learning image is not associated as a candidate product to which the learning image is to be newly registered, and the designation receiving means may receive a designation of a product to which the learning image is to be newly registered from the candidate products extracted by the extraction means.

[0170] The extraction means may be provided with a display means for displaying selectable products stored in the product information storage means as candidates for products for which the learning image is to be newly registered when it is not possible to extract a candidate product for which the learning image is to be added based on the image of the product captured by the imaging means, and the designation receiving means may be configured to receive a designation of a product for which the learning image is to be newly registered from the candidate products displayed by the display means.

[0171] The extraction means may extract candidate products to which the learning image is to be added based on the image of the product captured by the imaging means, by referring to a product information storage means that stores product information, when the weight of the product weighed by the weighing means is within the transaction weighing range.

[0172] The system may further include a grouping means for grouping specified products, and when the product for which designation has been accepted is one of the products grouped by the grouping means, the registration means may register an image captured by the imaging means as the learning image in the product information storage means in association with all products grouped with the product designated by the designation acceptance means.

[0173] An information processing system according to one aspect of the present invention is configured such that an information processing device and a higher-level management device can communicate with each other via a network, and the information processing device comprises an imaging means for imaging a product, and a designation receiving means for receiving designation of a product for which a learning image is to be added from candidate products for which the image of the product imaged by the imaging means is to be added as a learning image, and the management device comprises a product information storage means for storing product information, an extraction means for extracting candidate products for which the learning image is to be added by referring to the product information storage means based on the image of the product imaged by the imaging means, and a registration means for registering the image of the product imaged by the imaging means as the learning image in the product information storage means in association with the product specified by the designation receiving means.

[0174] A computer program according to one aspect of the present invention causes a computer to perform an imaging process for imaging a product, an extraction process for extracting candidate products to which the learning image is to be added based on the image of the product imaged by the imaging process and by referring to a product information storage means that stores product information, a designation acceptance process for accepting designation of a product to which the learning image is to be added from the candidate products extracted by the extraction process, and a registration process for registering the image of the product imaged by the imaging process as the learning image in the product information storage means in association with the product specified in the designation acceptance process.

[0175] A computer program according to one aspect of the present invention is provided in a system in which an information processing device and a higher-level management device are configured to be able to communicate with each other via a network, and causes the information processing device to perform an imaging process for imaging a product, and a designation acceptance process for accepting, from among candidate products to which the image of the product imaged by the imaging process is to be added as a learning image, a designation of a product to which the learning image is to be added and registered; and causes the management device to perform an extraction process for extracting candidate products to which the learning image is to be added by referring to a product information storage means that stores product information based on the image of the product imaged by the imaging process, and a registration process for registering the image of the product imaged by the imaging process as the learning image in the product information storage means, in association with the product specified in the designation acceptance process.

[0176] According to the above-described item identification device according to the embodiment of the present invention, learning images to be used for object scanning can be easily registered.

[0177] Furthermore, for example, Japanese Patent No. 7125179 discloses a product recognition device that includes a storage unit that pre-stores an image database in which at least training image information and product names are associated for a plurality of products, a learning unit that performs machine learning using the training image information and product names stored in the image database to generate a trained identification engine, and an identification unit that uses the identification engine to identify the product name of a target product based on the image information of the target product.

[0178] In this regard, when performing an object scan of products to be purchased in a store such as a supermarket, if all products registered in a predetermined storage device are subject to the object scan, there is a possibility that products that are not currently on sale will be extracted. Therefore, it is necessary to search for the target products in the predetermined storage device and set whether or not to make them available for sale. However, searching for and setting each target product is cumbersome and troublesome.

[0179] Therefore, one object of the present invention is to provide a device that can easily set product sales targets.

[0180] In order to achieve the above object, the information processing device of the present invention comprises an imaging means for imaging a product, an extraction means for extracting product candidates corresponding to the product image captured by the imaging means in a predetermined operation mode in which an operation for purchasing the product is performed, based on the product image captured by the imaging means, by referring to a product information storage means that stores product information, and a setting means for setting the product extracted by the extraction means in the predetermined operation mode as a product for sale or a product not for sale.

[0181] The system may further include a display control means for displaying an operator for setting using the setting means for each product extracted by the extraction means as a candidate product, and the display control means may vary the display mode of the operator depending on whether the product is set as a product for sale or a product not for sale.

[0182] The system may include the predetermined operation mode and a product information maintenance mode, and the extraction means may extract product candidates from products set as the products eligible for sale in the predetermined operation mode, and may extract product candidates from products set as the products eligible for sale and products set as products not eligible for sale in the maintenance mode.

[0183] The extraction means may be configured to, in the specified operation mode, when it is not possible to extract a candidate product from the products set as the products eligible for sale, extract a candidate product from the products set as the products not eligible for sale, and output a message to that effect.

[0184] The extraction means may refer to the product information storage means, use a learning image associated with the product as a reference image, and extract candidate products corresponding to the image of the product captured by the imaging means, and may exclude from the extraction targets any products among the products eligible for sale and the products not eligible for sale that are not associated with the learning image.

[0185] An information processing system according to one aspect of the present invention is configured so that an information processing device and a higher-level management device can communicate with each other via a network, the information processing device comprising an imaging means for capturing images of merchandise, and the management device comprising a product information storage means for storing product information, an extraction means for, in a predetermined operation mode in which operations for purchasing the merchandise are performed, referring to the product information storage means based on the image of the merchandise captured by the imaging means, extracting candidate products corresponding to the image of the merchandise captured by the imaging means, and a setting means for setting the merchandise extracted by the extraction means in the predetermined operation mode as a merchandise for sale or a merchandise not for sale.

[0186] A computer program according to one aspect of the present invention causes a computer to execute an imaging process for imaging a product, an extraction process for extracting product candidates corresponding to the product image captured by the imaging process in a predetermined operation mode in which an operation for purchasing the product is performed, by referring to a product information storage means that stores product information based on the product image captured by the imaging process, and a setting process for setting the product extracted by the extraction process in the predetermined operation mode as a product for sale or a product not for sale.

[0187] A computer program according to one aspect of the present invention is provided in a system in which an information processing device and a higher-level management device are configured to be able to communicate with each other via a network, and causes the information processing device to execute an imaging process to capture an image of a product, and causes the management device to execute, in a predetermined operation mode in which operations are performed to purchase the product, an extraction process to extract candidate products corresponding to the image of the product captured by the imaging process by referring to a product information storage means that stores product information based on the image of the product captured by the imaging process, and a setting process to set the product extracted by the extraction process in the predetermined operation mode as a product for sale or a product not for sale.

[0188] According to the above-described item identification device according to the embodiment of the present invention, it is possible to easily set up sales targets based on products extracted by object scanning.

[0189] DESCRIPTION OF SYMBOLS 1: Weighing device (information processing device) 104: Display touch panel (display means) 109: Weighing section (weighing means) 110: Printing section 111: Imaging section (imaging means) 121: Product file (product information storage means) 122: Weighing control section 123: Change detection section 124: Imaging control section 125: Extraction section (extraction means) 126: Setting section (setting means) 127: Registration section (registration means) 128: Grouping section (grouping means) 129: Calculation section 130: Printing control section 131: Designation acceptance section (designation acceptance means) 132: Display control section 2: Store controller 3: POS terminal 4: Cloud server G21: Product information display area G22: Preset key display area G221: Preset key

Claims

An imaging means for imaging a product; an extraction means for extracting candidates for products to which learning images are to be added, based on the images of the products captured by the imaging means and referring to a product information storage means for storing product information; a designation receiving means for receiving a designation of a product for which the learning image is to be additionally registered from among the product candidates extracted by the extraction means; and a registration means for registering an image of the product captured by the imaging means as the learning image in the product information storage means in association with the product designated by the designation acceptance means. Information processing device. 　 The extraction means includes: When it is not possible to extract a candidate product to which the learning image is to be added based on the image of the product captured by the imaging means, a product to which the learning image is not associated is extracted as a candidate product to which the learning image is to be newly registered; The designation receiving means is accepting a designation of a product for which the learning image is to be newly registered from the product candidates extracted by the extraction means; 2. The information processing device according to claim 1. 　 The extraction means includes: a display means for displaying, in a case where a candidate product to which the learning image is to be added cannot be extracted based on the image of the product captured by the imaging means, a product stored in the product information storage means as a candidate product for which the learning image is to be newly registered in a selectable manner; The designation receiving means is accepting a designation of a product for which the learning image is to be newly registered from the product candidates displayed by the display means; 2. The information processing device according to claim 1. 　 The extraction means includes: extracting candidates for a commodity to which the learning image is to be added, based on an image of the commodity captured by the imaging means, by referring to a commodity information storage means that stores commodity information, when the weight of the commodity measured by the weighing means is within a transaction weighing range; 4. The information processing device according to claim 1. 　 A grouping means for grouping predetermined products is further provided, The registration means includes: when the product designated by the designation acceptance means is any one of the products grouped by the grouping means, an image captured by the imaging means is registered as the learning image in the product information storage means in association with all products grouped with the product designated by the designation acceptance means; 4. The information processing device according to claim 1. 　 The information processing device and a higher-level management device are configured to be able to communicate with each other via a network, The information processing device includes: An imaging means for imaging a product; a designation receiving means for receiving, from among candidates for products to which the image of the product captured by the imaging means is to be added as a learning image, a designation of a product for which the learning image is to be additionally registered; Equipped with The management device includes: A product information storage means for storing product information; an extraction means for extracting candidates for a commodity to which the learning image is to be added, based on an image of the commodity captured by the imaging means and referring to the commodity information storage means; and a registration means for registering an image of the product captured by the imaging means as the learning image in the product information storage means in association with the product designated by the designation acceptance means. Information processing system. 　 For computers, An imaging process for imaging a product; an extraction process for extracting candidates for products to which the learning image is to be added, based on the image of the product captured by the imaging means and referring to a product information storage means that stores product information; A designation receiving process for receiving a designation of a product for which a learning image is to be additionally registered from the product candidates extracted by the extraction process; a registration process for registering an image of the product captured by the imaging means as the learning image, in association with the product designated in the designation reception process, in the product information storage means; Computer program. In a system in which an information processing device and a higher-level management device are configured to be able to communicate with each other via a network, The information processing device includes: An imaging process for imaging a product; A designation receiving process for receiving, from among candidates of products to which the image of the product captured by the imaging process is to be added as a learning image, a designation of a product for which the learning image is to be additionally registered; Run the command, The management device: an extraction process for extracting candidates for products to which the learning image is to be added, based on the images of the products captured by the imaging process, by referring to a product information storage means that stores product information; a registration process for registering an image of the product captured by the imaging process as the learning image in the product information storage means in association with the product designated in the designation reception process; Computer program. 　 An imaging means for imaging a product; an extracting means for extracting candidates for a commodity corresponding to the image of the commodity captured by the imaging means, by referring to a commodity information storage means for storing commodity information, based on the image of the commodity captured by the imaging means, in a predetermined operation mode in which an operation for purchasing the commodity is performed; and a setting unit for setting the product extracted by the extraction unit in the predetermined operation mode as a product to be sold or a product not to be sold. Information processing device. 　 a display control unit that displays an operator for setting the product by the setting unit for each product extracted as a product candidate by the extraction unit, The display control means The display mode of the operation element is changed depending on whether the product is set as a sale product or a non-sale product.

10. The information processing device according to claim 9. 　 The predetermined operation mode and a product information maintenance mode are provided, The extraction means includes: In the predetermined operation mode, a candidate product is extracted from the products set as the sales target products; In the maintenance mode, candidates for the product are extracted from the products set as the sale target products and the products set as the non-sale target products.

10. The information processing device according to claim 9. 　 The extraction means includes: when it is not possible to extract a candidate for the product from the products set as the products eligible for sale in the predetermined operation mode, extracting a candidate for the product from the products set as the products not eligible for sale and outputting a message to that effect; The information processing device according to claim 11. 　 The extraction means includes: extracting candidates for products corresponding to the image of the product captured by the imaging means by referring to the product information storage means and using a learning image associated with the product as a reference image, and excluding from the extraction targets, among the products to be sold and the products not to be sold, products to which the learning image is not associated; 13. The information processing device according to claim 9. 　 The information processing device and a higher-level management device are configured to be able to communicate with each other via a network, The information processing device includes: An imaging means for imaging a commodity, The management device includes: A product information storage means for storing product information; an extracting means for extracting, in a predetermined operation mode in which an operation for purchasing the product is performed, a candidate product corresponding to the image of the product captured by the imaging means, by referring to the product information storing means based on the image of the product captured by the imaging means; and a setting unit for setting the product extracted by the extraction unit in the predetermined operation mode as a product to be sold or a product not to be sold. Information processing system. 　 For computers, An imaging process for imaging a product; an extraction process for extracting candidates for a commodity corresponding to the image of the commodity captured by the imaging process, by referring to a commodity information storage means that stores commodity information, based on the image of the commodity captured by the imaging process, in a predetermined operation mode in which an operation for purchasing the commodity is performed; and executing a setting process for setting the products extracted by the extraction process as sale products or non-sale products in the predetermined operation mode. Computer program. 　 In a system in which an information processing device and a higher-level management device are configured to be able to communicate with each other via a network, The information processing device includes: Executing an imaging process for imaging the product; The management device: an extraction process for extracting candidates for a commodity corresponding to the image of the commodity captured by the imaging process, by referring to a commodity information storage means that stores commodity information, based on the image of the commodity captured by the imaging process, in a predetermined operation mode in which an operation for purchasing the commodity is performed; and executing a setting process for setting the products extracted by the extraction process as sale products or non-sale products in the predetermined operation mode. Computer program.

Images