Terminal device and program

Abstract

To provide a terminal device and a program that can reduce the possibility of picking a wrong product even for a beginner.SOLUTION: A terminal device according to an embodiment comprises: a storage unit that stores product identifying information that identifies a designated product; an interface that, for an image of a product captured by an imaging unit, receives the product identifying information related to the product that is recognized with a general object recognition technique; and notification means for providing information on whether or not the product of the image is a designated product and information indicating whether the freshness of the product is normal on the basis of the product identifying information received by the interface and the product identifying information stored in the storage unit.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The embodiment of the present invention relates to a terminal device and a program. [Background technology]

[0002] Recently, shopping centers, mass retailers, department stores, and other stores have adopted online shopping, where customers can order products from a terminal device (such as a personal computer, smartphone, or tablet device) at home. When a customer places an order through online shopping, a store clerk or other staff member uses a shopping basket or shopping cart (collectively referred to as "cart" below) to pick up the ordered product from the store's shelves and place it in the cart (hereinafter referred to as "picking").

[0003] When picking products, the staff member takes care to ensure that they are picking the correct product and, in the case of fresh food products, to ensure that the product is as fresh as possible. However, when the picking is done by a beginner or part-time worker with little experience (hereafter referred to as "beginners, etc."), there is a high likelihood that they will pick a product that is different from the ordered product or a product that is not fresh enough (for example, old) (hereafter referred to collectively as such products as "wrong product"). Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide a terminal device and a program that can reduce the possibility of even a beginner picking the wrong product. [Means for solving the problem]

[0005] The terminal device of the embodiment includes a memory unit that stores product identification information that identifies a pre-specified product, an interface that receives the product identification information related to the product recognized using general object recognition technology from an image of the product captured by an imaging unit, and a notification means that notifies information as to whether the imaged product is a specified product and information as to whether the freshness of the product is normal, based on the product identification information received by the interface and the product identification information stored in the memory unit. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram showing a system according to an embodiment. [Diagram 2] FIG. 2 is an explanatory diagram showing the principle of package recognition. [Diagram 3] FIG. 3 is a diagram showing an example in which the object detection model extracts a product image. [Figure 4] FIG. 4 is a diagram showing a cart used when picking up an item and a mobile terminal attached to the cart. [Diagram 5] FIG. 5 is a sequence diagram showing the flow of processing in the system. [Figure 6] FIG. 6 is a block diagram showing a hardware configuration of the terminal device. [Figure 7] FIG. 7 is a memory map showing the configuration of the list storage unit. [Figure 8] FIG. 8 is a block diagram illustrating a hardware configuration of the edge device. [Figure 9] FIG. 9 is a block diagram showing a hardware configuration of the server. [Figure 10] FIG. 10 is a functional block diagram showing the functional configuration of the mobile terminal. [Figure 11] FIG. 11 is a flowchart showing the flow of control processing of the mobile terminal. [Figure 12] FIG. 12 is a flowchart showing the flow of the product judgment process in the mobile terminal. [Figure 13]FIG. 13 is a flowchart showing the flow of the control process of the edge device. [Figure 14] FIG. 14 is a flowchart showing the flow of the control process of the server. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the embodiment, a mobile terminal detachably attached to a cart in a system for picking products related to online shopping will be described as an example of a terminal device. Note that the present invention is not limited to the embodiment described below.

[0008] Here, online shopping refers to a situation where a customer such as a member uses a network, etc. to access, for example, the homepage of a company that operates a store, and purchases products sold by that store or company online. The customer browses the homepage and specifies in advance the products they wish to purchase online, then places an order.

[0009] Fig. 1 is a diagram showing a system according to an embodiment. In Fig. 1, the system 1 is installed in a store such as a supermarket, a mass retailer, a convenience store, or a specialty store. The system 1 includes a mobile terminal 2, a server 4, and an edge device 6. The server 4 and the edge device 6 are communicatively connected via a communication line Na. In addition, one or more POS (Point of Sales) terminals 8 are communicatively connected to the system 1 via the communication line Na.

[0010] The communication line Na may be the Internet, a virtual private network (VPN), a local area network (LAN), a public communication network, a mobile communication network, or the like, either alone or in appropriate combination.

[0011] Furthermore, a wireless access point 7 (repeater) is connected to the communication line Na. The wireless access point 7 is connected to the mobile terminal 2 via a communication line Nb such as a wireless LAN. The mobile terminal 2 can communicate with the server 4, the edge device 6, and the POS terminal 8 via the wireless access point 7 using the communication line Nb.

[0012] The shopping cart 3 has a storage section 31 (see FIG. 4) that stores products displayed in the store. The top of the storage section 31 is open, and products are stored in the storage section 31 from the top. While moving around the store with the shopping cart 3, the staff picks out products that have been ordered via online shopping (products that have been specified in advance via online shopping) from among the displayed products and stores them in the storage section 31.

[0013] The mobile terminal 2 is provided so as to be attachable to the shopping cart 3. The mobile terminal 2 is a portable information terminal with a built-in microcomputer, such as a smartphone, a tablet terminal, or a PDA (Personal Digital Assistant). Application software for executing control, which will be described later with reference to FIG. 11 and FIG. 12, is installed in the mobile terminal 2. The mobile terminal 2 has (built-in) an imaging unit 29 (see FIG. 6). The imaging unit 29 is a camera that captures images and videos, for example, in which a large number of CCDs (Charge Coupled Devices: imaging elements) are arranged two-dimensionally (planarly). The imaging unit 29 captures images of products stored in the storage unit 31. In the embodiment, the imaging unit 29 is built in the mobile terminal 2, but the imaging unit 29 may be a device attached to a device other than the mobile terminal 2. For example, the imaging unit 29 may be one or more cameras attached to the shopping cart 3. The mobile terminal 2 transmits information on the image captured by the imaging unit 29 (hereinafter referred to as "image") to the edge device 6.

[0014] The edge device 6 is, for example, a computer provided adjacent to the server 4, or provided within the server 4, or installed on the cloud. The edge device 6 receives an image from the mobile terminal 2 and executes package recognition processing based on the image. That is, the edge device 6 receives an image captured by the imaging unit 29 from the mobile terminal 2, and executes package recognition processing using an object detection model 642, a feature extraction model 643, and a matching dictionary 644 (all of which refer to FIG. 8 ) described below, to identify one product. The edge device 6 transmits product identification information (for example, a product code) indicating the identified product to the mobile terminal 2.

[0015] The mobile terminal 2 executes a picking registration process for the product based on the product identification information received from the edge device 6. Here, the picking registration process refers to a process of adding information indicating that the product has been picked normally to a list storage unit 233 (see FIGS. 6 and 7) described later based on the received product identification information.

[0016] The server 4 is installed, for example, in the back yard of a store. Alternatively, it is a cloud server installed on a cloud. The server 4 manages the types and numbers of products sold in the store. The server 4 also manages sales in the store based on sales information received from the POS terminal 8. The server 4 also receives, from the edge device 6, images captured by the imaging unit 29 and decoded information obtained by decoding symbols, and performs machine learning to improve the accuracy of the matching dictionary 644 stored in the edge device 6.

[0017] The POS terminal 8 is a device installed in a store and executes accounting processes. The POS terminal 8 executes payment processing for a product based on product identification information of the product for which product registration processing has been performed by the mobile terminal 2. Product registration processing refers to processing for reading and storing product information (product name, product price, etc.) of an imaged product. Payment processing refers to processing for displaying the total price and tax amount of the products related to one transaction for which product registration processing has been performed, calculating and displaying change based on the deposit received from the customer, and issuing a receipt on which product information and payment information (total price, deposit amount, change amount, etc.) are printed. The POS terminal 8 transmits product information and payment information (i.e. sales information) of the product for which payment processing has been performed to the server 4.

[0018] Next, a description will be given of the package recognition process executed by the edge device 6. The package recognition process is a known technique that uses a general object recognition technique to recognize a product based on the shape, appearance, texture, and the like of an image of the product, and identifies one product.

[0019] The generic object recognition technology is described, for example, in "Yanai Keiji, 'The Current State and Future of Generic Object Recognition,' Transactions of the Information Processing Society of Japan, Vol. 48, No. SIG16 [searched August 10, 2010], Internet <URL: http: / / mm.cs.uec.ac.jp / IPSJ-TCVIM-Yanai.pdf>".

[0020] Here, the package recognition process will be described in detail. Fig. 2 is a diagram for explaining the package recognition process. As shown in Fig. 2, the package recognition process uses three types of dictionaries, namely, an object detection model 642, a feature extraction model 643, and a matching dictionary 644, to identify an image of a product.

[0021] The object detection model 642 is a dictionary for identifying an area (range) of an object (product) in a captured image. That is, the object detection model 642 is a dictionary for identifying an area in which an object (product) is captured from an input captured image based on the captured image. In other words, the object detection model 642 is a dictionary for extracting an image of a product (hereinafter referred to as an "object image") surrounded by a identified area from the captured image.

[0022] The feature extraction model 643 is a dictionary for calculating feature amounts effective for identifying a product shown in a specified area (object image). That is, the feature extraction model 643 outputs a feature vector indicating feature amounts indicating the features of a product whose area is specified, based on the input object image.

[0023] The matching dictionary 644 is a dictionary for identifying a product based on the output feature vector. In other words, the matching dictionary 644 stores a large number of products in association with feature amounts indicating the characteristics of the product according to product identification information that identifies each product, and calculates a similarity indicating how similar the large number of stored products are to the captured product by comparing the feature amounts indicated by the input feature vector with the stored feature amounts. The higher the similarity of a product, the higher the degree to which it resembles the captured product (i.e., the probability that it is the captured product). For example, a product with a similarity of 90% or more is highly likely to be the same product as the captured product.

[0024] In the package recognition process, the object detection model 642 is used to identify an area in which a product is captured from an image captured by an imaging unit such as a camera, and the feature extraction model 643 is used to extract the product's features, which represent the product's surface information such as the product's external shape, color, pattern, and unevenness captured in the area (object image) as parameters (the "feature vector" shown in FIG. 2), and the comparison dictionary 644 is used to compare the extracted features with the feature data of many products stored in the dictionary, to calculate a similarity indicating how similar the stored product is to the product, and to recognize the product according to the similarity. Based on the similarity calculated by the package recognition process, the system 1 identifies the recognized product as the captured product if there is one product whose similarity is equal to or greater than a threshold (e.g., 90% or more).

[0025] Next, the code recognition executed by the mobile terminal 2 will be described. In code recognition, when the imaging unit 29 captures an image of a symbol (such as a barcode or two-dimensional code) attached to a product, the mobile terminal 2 decodes the symbol based on the installed application software and generates decoded information. The decoded information includes product identification information that identifies the product to which the symbol is affixed. The product identification information is, for example, a product code that indicates the product, but it does not have to be a product code as long as it is information that identifies the product.

[0026] The edge device 6 extracts product images showing products from the images received from the mobile terminal 2. FIG. 3 shows an example of extracting product images from captured images. In FIG. 3, images of frames 1 to 9 are captured by the imaging unit 29. Frame 1 in FIG. 3 shows nothing, but in reality, the customer's hand for scanning the product and the background are captured, and the product is not captured. The same is true for frame 9. In contrast, frames 2 to 8 show the product in addition to the customer's hand and the background. Therefore, the mobile terminal 2 uses a known technique to extract the images of frames 2 to 8 showing the product from the images of frames 1 to 9 as product images.

[0027] Methods for extracting product images from captured images include, for example, a method of extracting images in which the product appears by distinguishing between the product and hands or the background. Other methods include a method of extracting images in which the product appears before and after the image in which symbol S in Figure 3 is recognized, a method of extracting images in which the product appears by going back from the image in which symbol S is recognized, and a method of extracting images in which the product appears after the image in which symbol S is recognized.

[0028] The edge device 6 executes package recognition processing. That is, the edge device 6 detects an object image in which the area (range) of the product is specified from the extracted product image using the object detection model 642 (A in FIG. 2).

[0029] Next, the edge device 6 uses the feature extraction model 643 to calculate feature amounts indicating the features of the product from the detected object image, and outputs a feature vector of the product (B in FIG. 2). Then, the edge device 6 uses the matching dictionary 644 to calculate a similarity indicating the degree of similarity between the stored product and the product based on the output feature vector. Then, the edge device 6 recognizes a product with a high similarity as the imaged product (C in FIG. 2). Then, the edge device 6 transmits product identification information for identifying the recognized product, and information on the feature amounts and similarity calculated during the package recognition process to the mobile terminal 2.

[0030] Furthermore, the edge device 6 transmits to the server 4 association information that associates the product image with the decoded information received from the mobile terminal 2. The edge device 6 then receives from the server 4 information on the matching dictionary 444 that has been machine-learned by the server 4 based on the transmitted association information, and stores it. The edge device 6 overwrites the matching dictionary 644 with the information on the matching dictionary received from the server 4 and stores it, thereby improving the accuracy of the stored matching dictionary 644.

[0031] The mobile terminal 2 executes a product judgment process (see FIG. 12) for the picked product (imaged product) based on the product identification information and the similarity and feature amount information received from the edge device 6. That is, it executes a process to judge whether the picked product is the ordered product and whether the freshness of the picked product is normal.

[0032] The server 4 additionally stores the association information received from the edge device 6 in the matching dictionary 444 (see FIG. 9) stored therein. Specifically, the server 4 extracts an object image from the product image included in the association information received from the edge device 6 using the object detection model 442 (see FIG. 9), and extracts a feature based on the object image using the feature extraction model 443 (see FIG. 9). The server 4 then adds and stores the extracted feature in the matching dictionary 444 stored therein based on the product identification information included in the decoded information associated with the product image. In this way, by increasing the feature stored for each product identification information in the matching dictionary 444, the matching dictionary 444 is machine-learned, and the accuracy of product identification by the matching dictionary 444 is improved. The server 4 then transmits the learned information of the matching dictionary 444 to the edge device 6.

[0033] In the embodiment, the object detection model 442, the feature extraction model 443, and the matching dictionary 444 are provided in the server 4, but the object detection model 442, the feature extraction model 443, and the matching dictionary 444 may be provided in a server other than the server 4 (for example, a headquarters server installed in a headquarters that manages and controls multiple servers 4, etc.).

[0034] Next, a shopping cart 3 will be described as an example of a cart. The shopping cart 3 is a device into which products purchased by a customer are placed while moving (roaming) around the store, but in the embodiment, the shopping cart 3 is a device that a staff member moves around the store to collect products ordered through online shopping. FIG. 4 is a side view showing an example of the overall configuration of the shopping cart 3. The shopping cart 3 is equipped with a basket-shaped storage section 31 with an open top, rotating casters 32 arranged below the storage section 31, and a handle section 33 that is connected to the storage section 31 and the casters 32 and that the customer holds when moving the shopping cart 3.

[0035] The storage section 31 has a generally rectangular side and bottom section surrounded by a mesh-like lattice 311 with an open top, allowing the products stored in the storage section 31 to be viewed from the outside. The side and bottom sections may be formed of walls instead of the mesh-like lattice 311.

[0036] Furthermore, the shopping cart 3 is provided with a jig near the handle 33 to which the portable terminal 2 can be attached and detached. The portable terminal 2 attached to the shopping cart 3 is installed such that the imaging direction of the imaging unit 29 faces the opening of the storage unit 31. In other words, the imaging direction is set so that at least a part of the imaging range of the imaging unit 29 includes the inside of the storage unit 31.

[0037] The imaging unit 29 captures images of the products to be stored in the storage unit 31 (i.e., the picked products) and outputs the captured images to the mobile terminal 2. The images captured by the imaging unit 29 are still images or moving images in frame units.

[0038] Next, a description will be given of the flow of processing in the system 1. Fig. 5 is a sequence diagram showing the flow of processing in the system 1. As shown in Fig. 5, the system 1 includes a mobile terminal 2, an edge device 6, and a server 4.

[0039] When picking up an ordered product, the mobile terminal 2 captures an image of the product to be picked using the imaging unit 29 (T1). Next, the mobile terminal 2 outputs the captured image to the edge device 6 (T2).

[0040] The edge device 6, which has received the image, performs package recognition using the image (T3). That is, the edge device 6 uses the object detection model 642 to generate an object image showing the product by identifying the area of ​​the product included in the product image, uses the feature extraction model 643 to extract a feature vector showing the features of the product based on the object image, and uses the matching dictionary 644 to calculate the similarity of the product to the captured product, thereby extracting products similar to the product (high similarity) (T3). Then, the edge device 6 transmits product identification information of the identified product to the mobile terminal 2 (T4). Note that at T4, the edge device 6 transmits information on the feature amount and similarity of the product calculated in the process of identifying the product identification information by package recognition, together with the product identification information, to the mobile terminal 2.

[0041] Meanwhile, the mobile terminal 2 that output the product image at T2 waits for the imaging unit 29 to capture the image of the symbol. When the image of the symbol is captured, the symbol is decoded to generate decoded information including the product identification information (T5). Note that the order of input of the product identification information by T4 and generation of the decoded information by T5 may differ depending on the timing of the image of the symbol. If the person in charge does not correctly face the imaging unit 29, the image of the symbol will be captured late, and if the generation of the decoded information is delayed, the input of the product identification information from the edge device 6 may come first. Conversely, if the image of the symbol is captured early, the generation of the decoded information may come earlier than the input of the product identification information from the edge device 6.

[0042] The portable terminal 2 receives and stores information related to the ordered products (product identification information, feature values, etc. of the ordered product) from the server 4 in advance. Then, when the portable terminal 2 receives the product identification information, feature values, and similarity from the edge device 6, it stores the similarity related to the product included in the order information in the RAM 23 (see FIG. 6) (T6). Then, the portable terminal 2 performs a product judgment to determine whether the received product identification information is included in the stored information related to the ordered products (whether the imaged product is the ordered product) (T7). If the received product identification information is included in the stored information related to the ordered products, the portable terminal 2 determines whether the freshness of the product is normal based on the stored similarity (T8). If the freshness is normal, the portable terminal 2 performs a picking registration indicating that the product has been picked (T9).

[0043] Meanwhile, the mobile terminal 2 transmits decoded information generated by decoding the symbol included in the image captured by the imaging unit 29 to the edge device 6 (T11). The timing of T11 at which the decoded information is transmitted to the edge device 6 may be any timing after T4 at which the mobile terminal 2 receives the product identification information and T5 at which the mobile terminal 2 generates the decoded information. The edge device 6 transmits association information that associates the product image generated from the image received at T2 with the decoded information received at T11 to the server 4 (T12).

[0044] The server 4, which has received the association information from the edge device 6, trains the matching dictionary 444 based on the association information (T13). The server 4 then transmits the trained information of the matching dictionary 444 to the edge device 6 (T14). The edge device 6 trains the matching dictionary 644 based on the information of the matching dictionary 444 received from the server 4 (T15). By training the matching dictionary 644 in this manner, the accuracy of package recognition using the matching dictionary 644 is improved.

[0045] Next, the hardware configuration of the mobile terminal 2 will be described. FIG. 6 is a block diagram showing the hardware configuration of the mobile terminal 2. As shown in FIG. 6, the mobile terminal 2 includes a CPU (Central Processing Unit) 21, which is an example of a processor, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, a memory unit 24, and the like. The CPU 21 is the main controller of the mobile terminal 2. The ROM 22 stores various programs. The RAM 23 expands programs and various data. The memory unit 24 stores various programs. The CPU 21, the ROM 22, the RAM 23, and the memory unit 24 are connected to each other via a bus 25. The CPU 21, the ROM 22, and the RAM 23 constitute a control unit 200. That is, the control unit 200 executes a control process of the mobile terminal 2, which will be described later, by the CPU 21 operating according to a control program stored in the ROM 22 or the memory unit 24 and expanded in the RAM 23.

[0046] The RAM 23 has an image storage unit 231, a decoded information unit 232, and a list storage unit 233 (storage unit). The image storage unit 231 stores images captured by the imaging unit 29. The decoded information unit 232 stores decoded information obtained by decoding a symbol captured by the imaging unit 29. The list storage unit 233 stores order information received from the server 4, which is related to a customer identified by a customer card read by the card reading unit 30 (described later), among the order information for online shopping stored in the server 4. The list storage unit 233 will be described later with reference to FIG. 7.

[0047] The memory unit 24 is composed of a non-volatile memory such as a hard disk drive (HDD) or a flash memory that retains stored information even when the power is turned off, and has a control program unit 241 that stores a control program (including installed application software) that operates the mobile terminal 2, a product master 242, and a threshold storage unit 243. The product master 242 stores product information of each product in association with product identification information that identifies the product. The threshold storage unit 243 stores a similarity threshold for determining whether the freshness of a product is normal or abnormal.

[0048] In addition, the control unit 200 is connected to an operation unit 27, a display unit 28, an imaging unit 29, and a card reading unit 30 via a bus 25 and a controller 26.

[0049] The operation unit 17 has a start key 271 and an end key 272. The start key 271 is operated when starting picking of ordered items. When the start key 271 is operated, the control unit 200 starts the picking process. The end key 272 is operated when ending picking. When the end key 272 is operated, the control unit 200 ends the picking process. The display 18 is, for example, a liquid crystal display, and displays information to a person in charge (for example, a store clerk) who operates the mobile terminal 2 (moves the shopping cart 3).

[0050] The card reading unit 30 reads card information including a customer code from a customer card (one example of a customer medium, which may be a medium other than a card) that stores a customer code (customer identification information) that identifies a customer. The card reading unit 30 reads card information from, for example, an inserted customer card or a scanned card.

[0051] The control unit 200 is also connected to a communication unit 34 (interface) via the bus 25. The communication unit 34 is communicatively connected to the edge device 6 and the server 4 via the communication line Nb and the wireless access point 7. The communication unit 34 receives, from the server 4, order information corresponding to the customer code read by the card reading unit 30. The communication unit 34 also receives, from the edge device 6, product identification information, feature amount, and similarity information of the package-recognized product.

[0052] Next, the list storage unit 233 will be described. FIG. 7 is a memory map showing the configuration of the list storage unit 233. The list storage unit 233 has a product code unit 2331, a product name unit 2332, a feature unit 2333, and a picking unit 2334. The product code unit 2331 stores product identification information for identifying products ordered by a customer identified by a customer code in online shopping. The product name unit 2332 stores the product name of the product in association with the product identification information stored in the product code unit 2331. The feature unit 2333 stores information on feature amounts (e.g., feature vectors) indicating the characteristics of the product in association with the product identification information stored in the product code unit 2331. The product identification information stored in the product code unit 2331, the product name information stored in the product name unit 2332, and the feature amount information stored in the feature unit 2333 are all received from the server 4 and stored.

[0053] When a product is registered for picking, the picking unit 2334 stores information (e.g., flag information) indicating that the product has been registered for picking. For example, when a product is registered for picking, the picking unit 2334 corresponding to the product stores flag information "1" indicating that the product has been registered for picking. The picking unit 2334 corresponding to a product that has not been registered for picking stores flag information "0" indicating that the product has not been registered for picking.

[0054] Next, the hardware configuration of the edge device 6 will be described. FIG. 8 is a block diagram showing the hardware configuration of the edge device 6. As shown in FIG. 8, the edge device 6 includes a CPU 61, which is an example of a processor, a ROM 62, a RAM 63, a memory unit 64, and the like. The CPU 61 is the main controller of the edge device 6. The ROM 62 stores various programs. The RAM 63 expands programs and various data. The memory unit 64 stores various programs. The CPU 61, the ROM 62, the RAM 63, and the memory unit 64 are connected to each other via a bus 65. The CPU 61, the ROM 62, and the RAM 63 constitute a control unit 600. That is, the control unit 600 executes a control process of the edge device 6, which will be described later, by the CPU 61 operating in accordance with a control program stored in the ROM 62 or the memory unit 64 and expanded in the RAM 63.

[0055] The RAM 63 has an object image storage unit 631, a feature amount unit 632, and a product identification information unit 633. The object image storage unit 631 stores an object image extracted based on an image received from the mobile terminal 2. The feature amount unit 632 stores a feature vector indicating the feature amount of the product calculated by the feature extraction model 643 based on the object image stored in the object image storage unit 631. The product identification information unit 633 compares the feature vector calculated by the feature extraction model 643 with the feature amount of the product stored in the comparison dictionary 644, calculates the similarity of the product close to the feature vector, and stores the product identification information of the product with high similarity. When the calculated similarity is equal to or greater than a threshold, the product identification information unit 633 stores the product identification information of the product with the highest similarity. The product identification information unit 633 also stores information on the calculated similarity.

[0056] The memory unit 64 is composed of a non-volatile memory such as an HDD or a flash memory in which stored information is retained even when the power is turned off, and has a control program unit 641, an object detection model 642, a feature extraction model 643, and a matching dictionary 644. The control program unit 641 stores a control program for driving the edge device 6.

[0057] The control unit 600 is also connected to the bus 65 and the controller 66. The control unit 600 is also connected to a communication unit 67 via the bus 65. The communication unit 67 is communicatively connected to the POS terminal 8 and the server 4 via a communication line Na. The communication unit 67 is also communicatively connected to the mobile terminal 2 via a communication line Nb and a wireless access point 7.

[0058] Next, the hardware configuration of the server 4 will be described. FIG. 9 is a block diagram showing the hardware configuration of the server 4. As shown in FIG. 9, the server 4 includes a CPU 41, which is an example of a processor, a ROM 42, a RAM 43, a memory unit 44, and the like. The CPU 41 is the main controller of the server 4. The ROM 42 stores various programs. The RAM 43 expands programs and various data. The memory unit 44 stores various programs. The CPU 41, the ROM 42, the RAM 43, and the memory unit 44 are connected to each other via a bus 45. The CPU 41, the ROM 42, and the RAM 43 constitute a control unit 400. That is, the control unit 400 executes a control process of the server 4, which will be described later, by the CPU 41 operating according to a control program stored in the ROM 42 or the memory unit 44 and expanded in the RAM 43.

[0059] The RAM 43 has a sales information section 431, an associated information section 432, and a list information section 433. The sales information section 431 stores sales data received from the POS terminal 8 and manages sales of the store. The associated information section 432 stores associated information received from the edge device 6. The list information section 433 stores product information (product identification information, product name, feature amount, etc.) of products ordered by customers through online shopping by customer code in association with the customer code that identifies the customer. For example, when a customer orders a product using a PC from home, information on the ordered product (hereinafter referred to as "order information") and the customer code are transmitted to the server 4 via, for example, an Internet line, and the server 4 stores the order information in association with the transmitted customer code. The feature amount for each product is stored in advance by the server 4 in a product master or the like, and when the server 4 receives order information, it calls up the feature amount corresponding to the product included in the order information from the product master and stores it in association with the product identification information of each product in the received order information.

[0060] The memory unit 44 is composed of a non-volatile memory such as an HDD or a flash memory in which stored information is retained even when the power is turned off, and has a control program unit 441, an object detection model 442, a feature extraction model 443, and a matching dictionary 444. The control program unit 441 stores a control program for driving the server 4. The object detection model 442 is a dictionary with the same content as the object detection model 642, the feature extraction model 443 is a dictionary with the same content as the feature extraction model 643, and the matching dictionary 444 is a dictionary with the same content as the matching dictionary 644.

[0061] In addition, the control unit 400 is connected to a display unit 47 and an operation unit 48 via a bus 45 and a controller 46. The display unit 47 is, for example, a liquid crystal display, and displays information to an operator who operates the server 4. The operation unit 48 is, for example, a keyboard, and is operated by the operator of the server 4.

[0062] The control unit 400 is also connected to a communication unit 49 via a bus 45. The communication unit 49 is communicatively connected to the POS terminal 8 and the edge device 6 via a communication line Na. The communication unit 49 is also communicatively connected to the mobile terminal 2 via a wireless access point 7.

[0063] From here, we will explain the functional configuration of the mobile terminal 2. Fig. 10 is a functional block diagram showing the functional configuration of the mobile terminal 2. The control unit 200 of the mobile terminal 2 functions as an order information acquisition means 201, an image transmission means 202, a product determination means 203, a freshness determination means 204, a candidate selection means 205, and a notification means 206 by following the control programs stored in the ROM 22 or the control program unit 241 of the memory unit 24.

[0064] The order information acquisition means 201 acquires order information of the product ordered by the customer through online shopping from the server 4. Specifically, the order information acquisition means 201 transmits the identified customer code (customer identification information) to the server 4, and receives and stores the order information extracted by the server 4 based on the customer code. More specifically, the order information acquisition means 201 transmits to the server 4 the customer code stored in the customer card read by the card reading unit 30, and receives from the server 4 the order information extracted by the server 4 based on the customer code, and stores it in the list storage unit 233.

[0065] The image transmission means 202 transmits the image captured by the imaging unit 29 to the edge device 6 .

[0066] The product determination means 203 determines whether the product identified by the edge device 6 is an ordered product. Specifically, the product determination means 203 determines whether product identification information indicating the product identified by the edge device 6 is included in the order information stored in the list storage unit 233.

[0067] The freshness determination means 204 determines whether the freshness of the product is normal based on the similarity of the product calculated when the imaged product is determined as one product, which is received by the communication unit 34 (interface). Specifically, the freshness determination means 204 determines whether the freshness of the product is normal or abnormal based on the similarity of the determined product to a product to be compared, which is received by the communication unit 34 and calculated when the edge device 6 determines one product using package recognition. More specifically, since the product to be compared is a product with normal freshness, the freshness determination means 204 determines whether the freshness of the imaged product is normal or abnormal based on whether the similarity of the product calculated when the edge device 6 determines one product using package recognition is equal to or greater than a threshold value (a boundary value between a product with normal freshness and a product with abnormal freshness) stored in the threshold storage unit 243. A predetermined similarity value that exceeds the similarity value at which the product can be recognized as one product is set as a threshold value, and if the received similarity value is equal to or greater than the threshold value, the freshness of the imaged product is determined to be normal. If the received similarity is equal to or greater than the similarity required for recognizing the image as one product, but is less than the threshold value, the image of the product is determined to have abnormal freshness.

[0068] For example, if the similarity of a certain product as a result of package recognition exceeds 90%, which is the level at which the product can be recognized as a single product, and is equal to or greater than a threshold value of 95% (e.g., 98%), the product is determined to be of normal freshness.On the other hand, if the similarity is equal to or greater than 90% but less than 95% (e.g., 91%), the product is determined to be of abnormal freshness.

[0069] The freshness determination means 204 may determine whether the freshness of a product is normal or abnormal as follows. The comparison dictionary 644 of the edge device 6 prestores the feature amounts of products with normal freshness and the feature amounts of products with abnormal freshness, which are related to the same product. The control unit 600 of the edge device 6 uses the comparison dictionary 644 to compare the feature amounts calculated when the imaged products received by the communication unit 34 (interface) are determined as one product with both the feature amounts of products with normal freshness and the feature amounts of products with abnormal freshness. The control unit 600 then calculates the similarity for each feature amount, and recognizes a product with a high similarity (either a product with normal freshness or a product with abnormal freshness) as the product. Note that a product with normal freshness and a product with abnormal freshness are recognized as products with the same product identification information because they are the same type of product. However, when the control unit 600 recognizes a product with normal freshness, it adds additional information indicating that the freshness is normal to the identified product identification information, for example. Furthermore, when the control unit 600 recognizes that the freshness of the product is abnormal, the control unit 600 adds additional information indicating that the freshness is abnormal to the identified product identification information, for example.

[0070] The freshness determination means 204 of the mobile terminal 2 determines whether the freshness of the product of the product identification information is normal or abnormal based on the additional information added to the product identification information received from the edge device 6. The freshness determination means 204 determines that the freshness of the product is normal when additional information indicating normal freshness is added. The freshness determination means 204 determines that the freshness of the product is abnormal when additional information indicating abnormal freshness is added.

[0071] When the determined product identification information is not stored in the list storage unit 233, the candidate selection means 205 selects a candidate product from the products stored in the list storage unit 233 based on the feature amount of the product calculated when the imaged product was determined as one product and received by the communication unit 34. Specifically, when the determined product identification information is not stored in the list storage unit 233, the candidate selection means 205 selects, from the products stored in the list storage unit 233, the candidate product whose feature amount is most similar (has the highest similarity) to the feature amount of the product calculated when the imaged product was determined as one product and received by the communication unit 34.

[0072] The notification means 206 notifies information as to whether the imaged product is a specified product and information as to whether the freshness of the product is normal, based on the product identification information received by the communication unit 34 and the product identification information stored in the list storage unit 233. Specifically, the notification means 206 notifies information as to whether the imaged product is an ordered product and information as to whether the freshness of the product is normal, based on the product identification information received by the communication unit 34 and the product identification information of a product ordered through online shopping stored in the list storage unit 233.

[0073] Specifically, the notification means 206 notifies information on whether the imaged product is the ordered product, depending on whether the product identification information received by the communication unit 34 is included in the order information stored in the list storage unit 233. When the product identification information received by the communication unit 34 is included in the product identification information stored in the list storage unit 233, the notification means 206 notifies information that the imaged product is the ordered product. When the product identification information received by the communication unit 34 is not included in the product identification information stored in the list storage unit 233, the notification means 206 notifies information indicating that the imaged product is not the ordered product (is a wrong product). More specifically, when the product identification information received by the communication unit 34 is not included in the product identification information stored in the list storage unit 233 and the candidate selection means 205 selects a candidate product, the notification means 206 notifies information indicating that the imaged product is not the ordered product (is a wrong product) and information on the selected candidate product. In addition, if the product identification information received by the communication unit 34 is not included in the product identification information stored in the list memory unit 233, and if the candidate selection means 205 does not select a candidate product, the notification means 206 notifies information indicating that the imaged product is not the ordered product (that it is the wrong product).

[0074] Furthermore, when the product identification information received by the communication unit 34 is included in the product identification information stored in the list storage unit 233, the notification means 206 notifies whether the freshness of the product is normal. Specifically, the notification means 206 notifies whether the freshness of the product is normal depending on whether the similarity of the product received by the communication unit 34 is equal to or greater than a threshold. When the freshness determination means 204 determines that the similarity of the product received by the communication unit 34 is equal to or greater than a threshold, the notification means 206 notifies that the freshness of the product is normal. When the freshness determination means 204 determines that the similarity of the product received by the communication unit 34 is less than the threshold, the notification means 206 notifies that the freshness of the product is abnormal (for example, a product that has lost its freshness or is degraded) (a wrong product) Alternatively, when the freshness determination means 204 determines that additional information indicating that the freshness is normal is added to the received product specifying information, the notification means 206 notifies that the freshness of the product is normal. Also, when the freshness determination means 204 determines that additional information indicating that the freshness is abnormal is added to the received product specifying information, the notification means 206 notifies that the freshness of the product is abnormal.

[0075] Next, the control of the mobile terminal 2 will be described. FIG. 11 is a flowchart showing the flow of the control process of the mobile terminal 2. As shown in FIG. 11, the control unit 200 of the mobile terminal 2 judges whether the start key 271 has been operated (S11). The control unit 200 waits until the start key 271 has been operated (No in S11), and when it is judged that the start key 271 has been operated (Yes in S11), the control unit 200 requests a list of order information from the server 4 (S12). Specifically, in order to obtain order information related to the customer from the server 4, the control unit 200 transmits the customer code read from the customer card by the card reading unit 30 to the server 4 (S12). The server 4 performs the process of S74 described later, extracts order information corresponding to the customer code, and transmits it to the mobile terminal 2 that transmitted the customer code. The order information obtaining means 201 receives the order information related to the customer code received from the server 4, and stores it in the list storage unit 233 (S13). Then, the control unit 200 displays a list related to the stored order information on the display unit 28 (S14).

[0076] Next, the control unit 200 judges whether the end key 272 has been operated (S21). If it is judged that the end key 272 has not been operated (No in S21), it judges whether one product to be picked from the displayed list has been designated by the person who will pick the products (S22). The control unit 200 waits until it is designated (No in S22), and if it is judged that one product has been designated (Yes in S22), the control unit 200 starts capturing an image by the imaging unit 29 (S23). Then, the control unit 200 stores the image captured by the imaging unit 29 in the image storage unit 231 (S24). Then, the image transmission means 202 transmits the stored image to the edge device 6 (S25). The edge device 6 performs package recognition processing based on the received image, the details of which will be described later with reference to FIG. 13.

[0077] The control unit 200 also determines whether a symbol attached to the product has been detected from the image stored in S24 (S26). If a symbol has been detected (Yes in S26), the control unit 200 decodes the detected symbol (S27). The control unit 200 then stores the decoded information in the decoded information unit 232 (S27). The control unit 200 then transmits the stored decoded information to the edge device 6 (S28).

[0078] If it is determined that a symbol has not been detected (No in S26), or after performing the process of S28, the control unit 200 determines whether or not it has received product identification information, similarity, and feature amount of a product identified by package recognition from the edge device 6 in conjunction with the image transmission in S25 (S29).The control unit 200 waits until such information is received (No in S29), and if it has been determined that such information has been received (Yes in S29), it then executes product determination processing (S30).

[0079] 12 is a flow chart showing the flow of the product determination process (determination of whether the product is a wrong product) executed by the control unit 200 in S30. The control unit 200 compares the product identification information determined to have been received from the edge device 6 in S29 with the product identification information of the ordered product stored in the list storage unit 233 (S41). Then, the product determination means 203 determines whether the received product identification information is included in the product identification information stored in the list storage unit 233 (S42). If it is determined that the received product identification information is included (Yes in S42), the freshness determination means 204 next determines whether the freshness of the product of the received product identification information is normal or abnormal based on the received similarity received at the same time and the threshold value stored in the threshold storage unit 243 (S43). If the similarity is equal to or greater than the threshold value, the freshness determination means 204 determines that the freshness of the product is normal. If the similarity is less than the threshold value, the freshness determination means 204 determines that the freshness of the product is normal.

[0080] When it is determined that the freshness of the product is normal (Yes in S43), the control unit 200 executes the process of S31 in FIG. 11. That is, the control unit 200 performs a picking registration process for the product in the picking unit 2334 related to the product identification information in FIG. 7 (S31). That is, the control unit 200 stores a flag "1" in the picking unit 2334 corresponding to the product identification information. Then, the control unit 200 returns to S21. Note that when the control unit 200 determines that the freshness of the product is normal (Yes in S43), it displays a message on the display unit 28 indicating that the correct product has been picked. The correct product means a product that is included in the order information and has the correct freshness.

[0081] On the other hand, if it is determined in S43 that the freshness of the product is abnormal (No in S43), the notification means 206 displays a message indicating that the picked product has an abnormality in freshness (wrong product) on the display unit 28 (S44). The message is, for example, "Freshness abnormal. Please discard this product," or "Freshness abnormal (wrong product). Please discard this product." Upon seeing the message, the picker discards the product and picks another product again. Then the control unit 200 returns to S21.

[0082] In addition, in S42, when it is determined that the received product identification information is not included in the product identification information stored in the list storage unit 233 (No in S42), the control unit 200 determines whether there is product identification information whose feature amount is similar to the order information stored in the list storage unit 233 (S45). The determination in S45 is made by comparing the feature amount determined to be received in S29 with each feature amount stored in the feature amount unit 2333 of the list storage unit 233, and determining whether there is a product whose similarity exceeds a predetermined level. If it is determined that there is product identification information whose feature amount is similar (Yes in S45), the candidate selection means 205 selects the product whose feature amount is determined to be similar as a candidate product (S46). Then, the notification means 206 displays a message indicating that the picked product is an incorrect product and a message indicating the selected candidate product on the display unit 28 (S47). The message is, for example, "The product is incorrect. Isn't the correct product XX?". When the picker sees the message, he or she discards the product and picks the displayed candidate product again. Then, the control unit 200 returns to S21.

[0083] Furthermore, if it is determined that there is no product identification information with similar features (No in S45), the notification means 206 displays a message indicating that the picked product is the wrong product on the display unit 28 (S48). The message may be, for example, "This is the wrong product. Please discard this product." Upon seeing the message, the picker discards the product and picks another product instead. Then, the control unit 200 returns to S21.

[0084] Return to the description of FIG. 11. In S21, when it is determined that the end key 272 has been operated (Yes in S21), the control unit 200 executes a process of confirming whether all the ordered items have been picked (S35). When picking registration information is stored in all picking units 2334 for the item identification information stored in the list storage unit 233, the control unit 200 determines that all the ordered items have been picked. When picking registration information is not stored in the picking unit 2334 corresponding to any item identification information, the control unit 200 determines that all the ordered items have not been picked. When it is determined that all the ordered items have been picked, the control unit 200 transmits picking completion information indicating that picking has been completed for the ordered items together with the customer code to the server 4 (S36). When the server 4 receives the picking completion information, it executes a picking completion process indicating that the ordered items have been picked. Then, the control unit 200 ends the imaging by the imaging unit 29 (S37). Then, the control unit 200 ends the process and returns to S11.

[0085] According to the mobile terminal 2 of this embodiment, package recognition is used to determine whether an image of a product captured for picking is an ordered product, and if the product is different from the ordered product, information is displayed indicating that the product is not the ordered product (the wrong product has been picked). Furthermore, if the freshness of the imaged product is abnormal, information is displayed indicating that the freshness is abnormal (the wrong product). This makes it possible to reduce the possibility that the person performing the picking task will pick the wrong product.

[0086] Next, the control of the edge device 6 will be described. FIG. 13 is a flowchart showing the flow of the control process of the edge device 6. As shown in FIG. 13, the control unit 600 of the edge device 6 judges whether an image captured by the imaging unit 29 has been received from the mobile terminal 2 (S51). If it is judged that an image has been received from the mobile terminal 2 (Yes in S51), the control unit 600 extracts a product image based on the received image (S52). The images of frames 0002 to 0008 in FIG. 3 (images showing products) are product images. Next, the control unit 600 uses the object detection model 642 to specify a product area for the extracted product image, and stores an object image surrounded by the specified area in the object image storage unit 631 (S53).

[0087] Next, the control unit 500 calculates a feature vector of the object image stored in the object image storage unit 631 using the feature extraction model 643 (S54). Then, the control unit 600 stores the calculated feature vector in the feature amount unit 632 (S55). Next, the control unit 500 uses the comparison dictionary 644 to compare the feature vector stored in the feature amount unit 632 with the feature amount stored for each item identification information stored in the comparison dictionary 644 to calculate a similarity, and extracts item identification information of items with high similarity (S56). For example, when there is an item with a similarity of 90% or more, the item identification information extracted in S56 is extracted. The calculated similarity and the extracted item identification information are stored in the item identification information unit 633. Then, the control unit 600 transmits the feature amount stored in the feature amount unit 632 and the item identification information and similarity stored in the item identification information unit 633 in association with each other to the mobile terminal 2 (S57). Then, the control unit 600 returns to S51.

[0088] Furthermore, if it is determined that the image has not been received from the portable terminal 2 (No in S51), the control unit 600 determines whether decoded information has been received from the portable terminal 2 (S61). If the control unit 600 has received decoded information from the portable terminal 2 by the process of S28, it generates association information associated with the product image already extracted in S52 (S62). Then, the control unit 600 transmits the generated association information to the server 4 (S63). Then, the control unit 600 returns to S51.

[0089] Also, if it is determined that the decoded information has not been received from the mobile terminal 2 (No in S61), the control unit 600 determines whether information of the matching dictionary 444 has been received from the server 4 (S64). The information of the matching dictionary 444 is information that has been machine-learned in the server 4. If it is determined that the information of the matching dictionary has been received (Yes in S64), the control unit 600 replaces the received information of the matching dictionary 444 with information of the matching dictionary 644 and stores it (S65), thereby improving the accuracy of the matching dictionary 644. Then, the control unit 600 returns to S51. Also, if it is determined that the information of the matching dictionary has not been received (No in S64), the control unit 600 returns to S51.

[0090] Next, the control of the server 4 will be described. FIG. 14 is a flowchart showing the flow of the control process of the server 4. As shown in FIG. 14, the control unit 400 of the server 4 judges whether order information for online shopping has been received from an information processing device such as a personal computer, tablet terminal, or smartphone operated by a customer (S71). If it is judged that the order information has been received (Yes in S71), the control unit 400 associates the received order information with the customer code received at the same time, and stores it in the list information unit 433 (S72). Then, the control unit 400 returns to S71.

[0091] Furthermore, if it is determined that the received information is not order information from a customer (No in S71), the control unit 400 determines whether or not a customer code associated with the operation in S12 has been received from the mobile terminal 2 (S73). If it is determined that a customer code has been received (Yes in S73), the control unit 400 transmits the order information stored in the list information unit 433 in association with the received customer code to the mobile terminal 2 (S74). Then, the control unit 400 returns to S71.

[0092] Also, if it is determined that the customer code is not received from the mobile terminal 2 (No in S73), the control unit 400 determines whether association information has been received from the edge device 6 (S81). If it is determined that association information has not been received (No in S81), the control unit 400 returns to S71, and if it is determined that association information has been received (Yes in S81), the control unit 400 stores the received association information in the association information unit 432 (S82). Then, the control unit 400 trains the matching dictionary 444 by machine learning based on the stored association information, and updates the information of the matching dictionary 444 (S83). Then, the control unit 400 transmits the updated information of the matching dictionary 444 to the edge device 6 (S84). Then, the control unit 400 returns to S71.

[0093] As described above, the mobile terminal 2 of the embodiment includes a list memory unit 233 that stores product identification information that identifies an ordered product, a communication unit 34 that receives product identification information related to the product that is determined using general object recognition on an image of the product captured by the imaging unit 29, and a notification means 206 that notifies information as to whether the imaged product is an ordered product and information as to whether the freshness of the product is normal, based on the product identification information received by the communication unit 34 and the product identification information stored in the list memory unit 233.

[0094] The mobile terminal 2 configured in this way notifies the user when a product different from the ordered product has been picked, or when a product with abnormal freshness has been picked. This makes it possible to reduce the possibility of even a beginner picking the wrong product.

[0095] In addition, the mobile terminal 2 of the embodiment is equipped with a candidate selection means 205 that, when the product identification information received by the communication unit 34 is not stored in the memory unit, selects a candidate product from the products stored in the list memory unit 233 based on the feature values ​​of the product calculated when determining the imaged product as a single product and received by the interface.

[0096] When a product different from the ordered product is picked, the mobile terminal 2 configured as above displays candidate products that are considered to be the correct product. Therefore, even a beginner can reduce the possibility of picking the wrong product.

[0097] Although the embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. This embodiment can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations can be made without departing from the gist of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims.

[0098] For example, in the embodiment, when picking a product, the product to be picked is specified in advance. However, this is not limited to this, and specifying the product to be picked is not a mandatory requirement. In this case, the control unit 200 only needs to determine whether the picked product is included in the list.

[0099] In the embodiment, when a symbol is read from an image of a product to be picked, the symbol is decoded. However, the process of decoding the symbol is necessary for machine learning of the matching dictionary 444 and the matching dictionary 644, and is not an essential requirement of the present invention.

[0100] In the embodiment, the freshness of the picked product is judged to be normal or abnormal depending on whether the similarity is equal to or greater than a threshold or less than a threshold. However, the present invention is not limited to this. Products with normal freshness and products with abnormal freshness may be stored in advance as data to be compared, and the feature amount of the package-recognized product may be judged to be closer to which data based on the similarity, and the control unit 200 may judge whether the freshness of the product is normal or abnormal based on the closer data. In this case, the control unit 200 judges that the freshness is normal when the feature amount is closer to that of a product with normal freshness, and judges that the freshness is abnormal when the feature amount is closer to that of a product with abnormal freshness.

[0101] In the embodiment, picking of products to be sold in a store has been described as an example, but the present invention is not limited to this and can also be used for packing picked products into boxes, for example.

[0102] In the embodiment, the product is designated in advance in the online shopping and the product is photographed when it is picked up. However, the present invention is not limited to this. For example, the product may be designated in advance when inventorying the product, and the product to be inventoryed may be photographed.

[0103] In the embodiment, the mobile terminal 2 that can be attached to the shopping cart 3 has been described as an example of the terminal device. However, the present invention is not limited to this, and attachment to a shopping cart is not a mandatory requirement for the terminal device, and the terminal device does not have to be a mobile terminal.

[0104] In the embodiment, as an example of the notification by the notification means 206, a display is provided on the display unit 28. However, the present invention is not limited to this, and the notification means 206 may notify by a means other than a display, for example, by generating a sound different from a normal sound when it is determined that the product is a wrong product, thereby notifying information as to whether the imaged product is the ordered product and information as to whether the freshness of the product is normal.

[0105] The program executed by the mobile terminal 2 of the embodiment is provided as a file in an installable or executable format recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD).

[0106] The program executed by the portable terminal 2 of the embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program executed by the portable terminal 2 of the embodiment may be provided or distributed via a network such as the Internet.

[0107] Furthermore, the program executed by the mobile terminal 2 of the embodiment may be provided by being pre-installed in a ROM or the like. [Explanation of symbols]

[0108] 1 System 2. Mobile devices 3. Shopping Cart 4 Server 6 Edge Devices 8. POS terminals 21 CPU 24 Memory section 27 Control section 28 Display section 29 Imaging unit 30 Card reader 31 Storage area 34 Communications Department 41 CPU 44 Memory section 47 Display section 48 Control section 49 Communications Department 61 CPU 64 Memory section 67 Communications Department 200 Control section 201 Order information acquisition method 202 Image transmission means 203 Product Judgment Means 204 Method of determining freshness 205 Candidate selection means 206 Notification means 231 Image storage unit 232 Decode Information Section 233 List Storage Unit 243 Threshold memory unit 400 Control section 431 Sales Information Department 432 Related Information Section 433 List Information Department 442 Object Detection Model 443 Feature Extraction Model 444 Matching Dictionary 500 Control section 600 Control section 631 Object Image Storage Unit 632 Feature Quantity Section 633 Product Specific Information Department 642 Object Detection Model 643 Feature Extraction Model 644 Matching Dictionary 2331 Product Code Section 2332 Product name section 2333 Feature Quantity Section 2334 Picking Department [Prior art documents] [Patent documents]

[0109] [Patent Document 1] JP 2011-66573 A [Patent Document 2] JP 2022-87366 A

Claims

1. A storage unit that stores product identification information that identifies a product designated in advance; an interface for receiving the product identification information relating to the product recognized using a generic object recognition technology from an image of the product captured by an imaging unit; a notification means for notifying information as to whether the imaged product is a specified product and information as to whether the freshness of the product is normal, based on the product identification information received by the interface and the product identification information stored in the storage unit; A terminal device comprising:

2. the notification means, when the product identification information received by the interface is not stored in the storage unit, notifies a message that the imaged product is different from the designated product. The terminal device according to claim 1 .

3. the notification means, when the product identification information received by the interface is stored in the storage unit, issues a message to that effect if the freshness of the imaged product is abnormal.

3. The terminal device according to claim 1 or 2.

4. a freshness determination means for determining whether the freshness of the product is normal based on a similarity of the product calculated when the imaged product received by the interface is recognized as a single product, The notification means notifies a message to that effect when the freshness determination means determines that the freshness of the product is not normal. The terminal device according to claim 3.

5. a candidate selection means for selecting a candidate product from the products stored in the storage unit based on information on a feature amount of the product that is calculated when the imaged product is recognized as a single product and that is received by the interface when the product identification information received by the interface is not stored in the storage unit, The notification means notifies information on the candidate products selected by the candidate selection means together with the message. The terminal device according to claim 2.

6. A computer as a terminal device including a storage unit that stores product identification information that identifies a product designated in advance, and an interface that receives the product identification information related to the product that is recognized by using a general object recognition technology for an image of the product captured by an imaging unit, a notification means for notifying information as to whether the imaged product is a specified product and information as to whether the freshness of the product is normal, based on the product identification information received by the interface and the product identification information stored in the storage unit; A program to make it function as such.

Images