Information processing apparatus, control method, and program

Abstract

To provide a mechanism that can specify commodities while considering the number of types of commodities specified at a time.SOLUTION: The present invention comprises: management means that manages the upper limit number of the types of commodities specified at a time; acquisition means that acquires a result of the specification of commodities from images of commodities; and display control means that performs control to display the images of commodities and information on the specified commodities. The display control means identifies and displays if the number of types of the specified commodities exceeds the upper limit number.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for identifying a product from a product image. [Background technology]

[0002] In retail businesses, it is mandatory to attach product labels to products to identify them. Currently, staff visually check products to determine the type of product, then print and attach product labels that correspond to the product type, which can lead to printing and attachment errors. Product label errors not only result in lost revenue due to discrepancies in the listed price and complaints from customers, but also pose a risk of serious accidents in the case of food products due to misidentification of allergens.

[0003] One possible solution would be to use object detection technology based on machine learning to automatically recognize the type of product from a photograph of the product, and then output a product label based on the results.

[0004] Patent Document 1 discloses a technique for identifying a product based on a product recognition result obtained by a global model common to all stores and a product recognition result obtained by a local model specific to each store. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2022-32962 DISCLOSURE OF THEINVENTION [Problem to be solved by the invention]

[0006] However, the technology of Patent Document 1 does not take into consideration the recognition of multiple products at once, and therefore does not take into consideration the number of types of products that can be identified at once.

[0007] Therefore, an object of the present invention is to provide a mechanism for identifying products while taking into consideration the number of types of products identified at one time. [Means for solving the problem]

[0008] The present invention comprises a management means for managing the maximum number of product types that can be identified at one time, an acquisition means for acquiring identification results of products identified from product images, and a display control means for controlling the display of the product images and information related to the identified products, and is characterized in that the display control means identifies and displays whether the number of identified product types exceeds the maximum number. Effect of the Invention

[0009] According to the present invention, it is possible to provide a mechanism for identifying products while taking into consideration the number of types of products to be identified at one time. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an example of a system configuration of an information processing system. [Diagram 2] FIG. 1 is a diagram illustrating an example of a hardware configuration applicable to a learning server, a business server, a store edge device, and a store client. [Diagram 3] 13 is a flowchart showing an example of a process for linking a learning model and a similar products list between the head office and the store. [Figure 4] 13 is a flowchart showing an example of a process for linking a product table, a product price table, and a schedule list between a head office and a store. [Diagram 5] 13 is a flowchart showing an example of a process for operating the system on the store side. [Figure 6] 13 is a flowchart illustrating an example of a product determination process. [Figure 7] 13 is a flowchart illustrating an example of a product determination process. [Figure 8] FIG. 13 is a diagram showing an example of an image. [Figure 9]FIG. 13 is a diagram illustrating an example of a tray list screen. [Figure 10] FIG. 13 is a diagram showing an example of a candidate product list screen. [Figure 11] 13A and 13B are diagrams showing an example of a candidate product list screen and a product designation screen. [Figure 12] FIG. 13 is a diagram showing an example of a candidate product list screen. [Figure 13] FIG. 13 is a diagram showing an example of a similar products list screen. [Figure 14] FIG. 13 is a diagram illustrating an example of a tray list screen. [Figure 15] FIG. 4 is a diagram showing examples of various tables. [Figure 16] FIG. 4 is a diagram showing examples of various tables. [Figure 17] 13 is a flowchart illustrating an example of a process for considering an upper limit on the number of product varieties. [Figure 18] FIG. 13 is a diagram illustrating an example of a tray list screen. [Figure 19] 13 is a flowchart showing an example of a process for updating a similar products list during system operation. [Figure 20] 13A and 13B are diagrams showing examples of a candidate product list screen and a similar product registration confirmation screen. [Figure 21] 13A and 13B are diagrams showing examples of a candidate product list screen, a product designation screen, and a similar product registration confirmation screen. [Figure 22] FIG. 11 illustrates an example of a batch printing result. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0012] FIG. 1 is a diagram showing an example of a system configuration of an information processing system 100 according to an embodiment of the present invention.

[0013] The learning server 101 and business server 102, which are centrally managed at the head office (or main branch), are provided in a cloud environment, with their actual locations located in any data center, connected to the Internet or a dedicated line 107, and capable of communicating with each store.

[0014] A store edge device 103, store client 104, network camera 105, and label output device 106 managed in each store are connected to each other via an in-store LAN 108, and can further communicate with the head office via the Internet or a dedicated line 107.

[0015] FIG. 2 is a block diagram showing an example of a hardware configuration applicable to each of the information processing devices of the learning server 101, the business server 102, the store edge device 103, and the store client 104 in the embodiment of the present invention.

[0016] As shown in FIG. 2, the information processing device includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an input controller 205, a video controller 206, a memory controller 207, and a communication I / F controller 208 connected via a system bus 204.

[0017] The CPU 201 comprehensively controls each device and controller connected to a system bus 204 .

[0018] ROM 202 or external memory 211 holds the BIOS (Basic Input / Output System) and OS (Operating System), which are control programs executed by CPU 201, computer-readable and executable programs for realizing this information processing method, and various necessary data (including data tables).

[0019] The RAM 203 functions as a main memory, a work area, etc., for the CPU 201. The CPU 201 loads programs and the like required for executing processes from the ROM 202 or the external memory 211 into the RAM 203, and executes the loaded programs to realize various operations.

[0020] An input controller 205 controls input from input devices such as a keyboard 209 and a pointing device such as a mouse (not shown).

[0021] When the input device is a touch panel, the user can give various instructions by pressing (touching with a finger or the like) icons, cursors, or buttons displayed on the touch panel.

[0022] The touch panel may be a touch panel capable of detecting positions touched by multiple fingers, such as a multi-touch screen.

[0023] A video controller 206 controls display on an external output device such as a display 210. The display also includes a display of a notebook computer that is integrated into the main body.

[0024] The external output device is not limited to a display, and may be, for example, a projector. In addition, for the device capable of receiving the above-mentioned touch operation, an input device is also provided.

[0025] The video controller 206 is capable of controlling a video memory (VRAM) for display control, and can use part of the RAM 203 as the video memory area, or can provide a separate dedicated video memory.

[0026] The memory controller 207 controls access to the external memory 211. As the external memory, an external storage device (hard disk) for storing a boot program, various applications, font data, user files, edit files, and various data, a flexible disk (FD), or a compact flash (registered trademark) memory connected to a PCMCIA card slot via an adapter, etc., can be used.

[0027] The communication I / F controller 208 connects and communicates with external devices via a network, and executes communication control processing on the network. For example, communication using TCP / IP, telephone lines such as ISDN, and 3G lines for mobile phones are possible.

[0028] The CPU 201 enables display on the display 210 by, for example, executing a process of expanding (rasterizing) an outline font in a display information area in the RAM 203. The CPU 201 also enables a user to give instructions using a mouse cursor (not shown) on the display 210.

[0029] It is preferable that the learning server 101 and the store edge device 103 are configured using a GPU (Graphics Processing Unit) that excels in real-time image processing, rather than a CPU.

[0030] Next, the processes executed by the information processing system 100 in the embodiment of the present invention will be described with reference to the flowcharts shown in FIGS.

[0031] 3 is a flowchart showing an example of a process for linking a learning model and a similar product list between the head office and the store. This process flow is executed when a new learning model is generated or when a learning model is additionally trained.

[0032] In step S101, the learning server 101 acquires learning data. Here, learning data refers to data in which product images are associated with information identifying the type of product (product number and product name), and in the case where multiple product images exist in one image, regions within the image are designated by annotation, and the above-mentioned identification information is associated with each region. Here, learning data covering all products is acquired, but differential learning data is acquired during additional learning.

[0033] In step S102, the learning server 101 generates a learning model using all the acquired learning data. The learning model thus generated makes it possible to detect the position of a product and the product type (candidate) in an image.

[0034] In step S103, the learning server 101 transmits the learning model generated in step S102 to the store edge device 103, and in step S104, the store edge device 103 receives the transmitted learning model. The store edge device 103 is now able to detect products using the received learning model.

[0035] In step S105, the learning server 101 generates a similar products list. An example of the similar products list is shown in FIG.

[0036] Similar product list 1530 has similar group 1531 and product number group 1532 as items. Similar group 1531 is an alphanumeric string that identifies a group of similar products. Product number group 1532 corresponds to product number 1501 in product table 1500 shown in Fig. 15(a), and product numbers of products with similar appearances are registered in order. Note that the similar product list does not have to have this format, and may instead be in a format in which the product number of one product is used as a key and the product numbers of products with similar appearances are registered in order.

[0037] For details of step S105, the learning server 101 secondarily calculates the image features of each product when generating the learning model in step S102, and creates similarity groups by grouping products with similar features. In this way, similarity groups are created for all products, thereby generating a similar product list. Note that the image features of each product may be calculated using a known technique separately from step S102.

[0038] In step S106, the learning server 101 transmits the similar products list to the business server 102, and in step S107, the business server 102 receives the similar products list.

[0039] In step S108, if the similar products list needs to be modified or added, the business server 102 displays an editing screen (not shown) and accepts updates to the similar products list from the user. By accepting updates, additions or deletions to the similar groups 1531 in the similar products list 1530 and additions, deletions, or changes to product numbers in the product number groups 1532 are implemented.

[0040] In step S109, the business server 102 transmits the similar products list to the store client 104, and in step S110, the store client 104 receives the similar products list.

[0041] In step S111, the shop client 104 accepts updates to the similar products list as necessary, similar to step S108.

[0042] Note that the updating of the similar products list in step S108 may be executed by the learning server 101, and the similar products list may be transmitted from the learning server 101 to the store client 104. Furthermore, the updating of the similar products list in steps S108 and S111 may be executed not only when a new learning model is generated or additional learning is performed, but also whenever an update is required.

[0043] This completes the explanation of FIG.

[0044] 4 is a flow chart showing an example of a process for linking a product table, a product price table, and a schedule list between the head office and the store. This process flow is executed every specific period (week or specific cycle).

[0045] In step S201, the business server 102 transmits a product table to the store client 104, and in step S202, the store client 104 receives the product table. An example of the product table is shown in Fig. 15(a).

[0046] In step S202, the business server 102 transmits the product price table to the store client 104, and in step S202, the store client 104 receives the product price table. An example of the product price table is shown in Fig. 15(b).

[0047] Product table 1500 has items such as product number 1501, JAN code 1502, product name 1503, and representative image 1504. Product number 1501 is a unique number assigned to each product to identify the product within an organization (such as a company). JAN code 1502 is a globally recognized product identification code. Product name 1503 is text that describes the product. Representative image 1504 is a representative image file that shows the appearance of the product. Product number 1501 is used in this embodiment as information to identify the product. JAN code 1502 and product name 1503 are used to output related items on the product label. Representative image 1504 is used to visually confirm the product on a screen display, etc.

[0048] Product price table 1510 has the following items: JAN code 1511, applicable period From 1512, applicable period To 1513, and price 1514. JAN code 1511 corresponds to JAN code 1502 in product table 1500. Applicable period From 1512 and applicable period To 1513 indicate the start and end dates of the period to which the price information of the record applies, and in both cases, a record with "-" indicates the default value, that is, standard price information. Price 1514 indicates the price that applies to the product with JAN code 1511 during the period indicated by applicable period From 1512 and applicable period To 1513. These are used to output the price on the product label.

[0049] In step S205, the business server 102 generates a schedule list for each store. An example of the schedule list is shown in FIG.

[0050] The schedule list 1520 has items such as scheduled period From 1521, scheduled period To 1522, product number 1523, and scheduled production quantity 1524. Scheduled period From 1521 and scheduled period To 1522 indicate the start date and end date of the period for which the production schedule is managed by the record. Product number 1523 corresponds to product number 1501 in product table 1500, and indicates the product number of the product scheduled to be produced during the period indicated by scheduled period From 1521 and scheduled period To 1522. Planned production quantity 1524 indicates the planned production quantity of the product during the aforementioned period.

[0051] In step S205, a schedule list may be generated by extracting production schedules for the relevant period for the store from already created annual or monthly schedules, or an editing screen (not shown) may be displayed to accept input for each item of the schedule list.

[0052] In step S206, the business server 102 transmits the schedule list to the store client 104, and in step S207, the store client 104 receives the schedule list.

[0053] In step S208, if the plan list needs to be revised due to circumstances on the store side, the store client 104 displays an edit screen (not shown) and accepts updates to the plan list. By accepting updates, records are added or deleted from the plan list 1520, and each item is updated.

[0054] This completes the explanation of FIG.

[0055] 5 is a flow chart showing an example of a process for operating the system on the store side. This process flow is executed every day.

[0056] In step S301, the store client 104 sends an instruction to the network camera 105 to capture an image of a product, and in step S302, acquires the image captured by the network camera 105. As shown in Fig. 1, the network camera 105 captures images of trays 111 on which products 110 are placed, and the images acquired are also tray-by-tray. Fig. 8 shows an example of the image.

[0057] Image 800 is an image of a tray 801 with a product 802 placed on it. The outer frame of the image is trimmed to match the outer frame of the tray. There is no limit to the number of products that can be placed on the tray, but the products are placed so that they do not overlap. In this embodiment, images are handled as tray units, but an image of one product may also be extracted and handled as a product image such as 810. In this case, image 810 contains only image 811 of one product.

[0058] In step S303, the store client 104 transmits the acquired image to the store edge device 103, and in step S304, the store edge device 103 receives the image.

[0059] In step S305, the store edge device 103 receives the image as an input and detects products included in the image using the learning model received in step S104.

[0060] In step S306, the store edge device 103 transmits the detection result of the product detected in step S305 to the store client 104, and in step S307, the store client 104 receives the detection result. Fig. 16(a) shows an example of a detection result table storing the detection results.

[0061] The detection result table 1600 includes the following items: position ID 1601, rank 1602, coordinate X 1603, coordinate Y 1604, width 1605, height 1606, product number 1607, and reliability 1608. The position ID 1601 is identification information indicating the position (rectangular area) of the product placed on the tray in the image, and in this embodiment, the first two digits indicate the position from the top to the bottom of the image, and the last two digits indicate the position from the left to the right. The rank 1602 is the rank of product candidates detected at the same position in the image, arranged in descending order of reliability 1608. The coordinate X 1603 and the coordinate Y 1604 are numbers indicating the position where the product was detected in the image, that is, the coordinates of a specific point (the upper left corner in this embodiment) of the rectangular area, in pixels, with the upper left corner of the image as the origin. The width 1605 and the height 1606 are numbers indicating the position where the product was detected in the image, that is, the width and height of the rectangular area, in pixels. Product number 1607 indicates the product number of the product detected in step S305, and reliability 1608 indicates the reliability of detecting the product. Note that even if the type of product cannot be detected or the reliability is extremely low, the position ID 1601, coordinate X 1603, coordinate Y 1604, width 1605, and height 1606 are output from the outer frame of the image of the product. In that case, product number 1607 becomes "undetermined."

[0062] 8, when an image of only one product is included, there is only one type of position ID 1601. Also, in step S305, when multiple product candidates are not detected from the same position but one product is specified, the rank 1602 is only "1". Also, when detecting a product from a shape other than a rectangular area of ​​the image, an item indicating the shape can be set in the items corresponding to 1603 to 1606.

[0063] In this embodiment, the reliability 1608 is the probability for product candidates detected at the same position. Therefore, the sum of the reliability for product candidates detected at the same position does not exceed 1. However, other indices may be used as long as they can determine the relative merits of the product candidates.

[0064] Incidentally, since handling detection results with extremely low reliability would make the confirmation process complicated, in this embodiment, a reliability lower limit 1541 can be set as shown in Fig. 15(e). This value is stored in parameter table 1540 and used during operation. As a result, detection results with reliability lower than the reliability lower limit 1541 are not registered in detection result table 1600, or only items related to location (1601, 1603 to 1606) are registered, and product number 1607 becomes "undetermined."

[0065] In step S308, the store client 104 determines the product based on the received detection result. This process will be described in detail with reference to FIGS.

[0066] 6 and 7 are flowcharts showing an example of the product selection process.

[0067] In step S401, the shop client 104 performs initial setting of product information for each location. Here, the product determination table is updated. An example of the product determination table is shown in FIG. 16(b).

[0068] The product determination table 1610 has the following items: position ID 1611, coordinate X 1612, coordinate Y 1613, width 1614, height 1615, product number 1616, and printed 1617. The position ID 1611, coordinate X 1612, coordinate Y 1613, width 1614, and height 1615 correspond to the position ID 1601, coordinate X 1603, coordinate Y 1604, width 1605, and height 1606 in the detection result table 1600, respectively. The product number 1616 indicates the product number of the product determined or provisionally determined for the position ID 1611. Therefore, one record is registered for each position ID 1611. The printed 1617 indicates whether or not a product label with the product number 1616 has been printed for the position ID 1611 (including an error).

[0069] In step S401, the coordinates X1603, Y1604, width 1605, and height 1606 of a record whose position ID 1601 in the detection result table 1600 matches the position ID 1611 in the product determination table 1610 are set to coordinates X1612, Y1613, width 1614, and height 1615, respectively, and the product number 1607 of a record whose rank 1602 is "1" among those records is set to product number 1616 as an initial value. In other words, the product number of the product with the highest reliability of the detection result is set to the initial value. Note that if a product cannot be detected in the product detection, or if it can be detected but the reliability is lower than the reliability lower limit 1541, that is, if only the items related to the position (1601, 1603 to 1606) are registered in the detection result table 1600 and the product number 1607 is "undecided", the product number 1616 in the product determination table 1610 is also set to "undecided".

[0070] In step S402, the store client 104 displays a tray list screen. An example of the tray list screen is shown in FIG.

[0071] The tray list screen 901 is a screen that displays a list of the detection results of all products placed on the tray. A message 902 according to the detection result, a rectangular area frame 904 in which the product was detected, and the product number, product name, reliability, etc. as the detection result 905 are displayed superimposed on the captured image 903. This display data is based on the product determination table 1610 and references necessary data from the detection result table 1600 and the product table 1500. The reliability can be displayed not only as a numerical value, but also by changing the display color of the rectangular area frame 904 or by displaying it in a shaded area depending on the numerical value range, so that it can be intuitively recognized.

[0072] Furthermore, other candidate products present category 906 and similar products present category 907 are displayed with marks. Other candidate products present category 906 indicates that there are candidates for detected products other than the product displayed as detection result 905. In other words, it indicates that there are other records for the same position ID 1601 in detection result table 1600. Similar products present category 907 indicates that there are products similar to the product displayed as detection result 905. In other words, it indicates that the product number of the product is registered in product number group 1532 of similar products list 1530, and that product numbers of products other than the product are also registered.

[0073] Furthermore, when the displayed product is not scheduled for production, that is, when the product number of the product is not registered in product number 1523 of the record corresponding to the current period in schedule list 1520, tray list screen 901 displays “Not scheduled” in a superimposed manner at the relevant position, as in 909.

[0074] In addition, if a product cannot be detected during product detection, or if the product is detected but the reliability is lower than the reliability lower limit 1541, that is, if the product number 1607 in the product determination table 1610 is “Undetermined,” the tray list screen 901 will display “Undetermined” superimposed at that position, as in 908.

[0075] The tray list screen 901 has various function buttons (911 to 915) at the bottom, and the functions of each will be explained in the following processing flow.

[0076] In step S403, the store client 104 determines whether a product designation has been accepted, that is, whether any position in the image 903 on the tray list screen 901 has been selected and the product designation button 911 has been pressed. If accepted, the process proceeds to step S404; if not, the process proceeds to step S406.

[0077] In step S404, the store client 104 displays a product designation screen and accepts product designation. The product designation screen will be described later with reference to FIG.

[0078] In step S405, the store client 104 updates the product information for the location, that is, the product number 1607 in the product determination table 1610, based on the received content, and returns to step S402.

[0079] In step S406, the store client 104 determines whether a request to display a candidate product list has been accepted, that is, whether any position in the image 903 on the tray list screen 901 has been selected and the candidate product list display button 912 has been pressed. If accepted, the process proceeds to step S411, and if not, the process proceeds to step S407. In step S411, the candidate product list screen is displayed, and the candidate product list screen will be described later with reference to FIG. 10.

[0080] In step S407, the store client 104 determines whether a similar product list display has been accepted, that is, whether any position in the image 903 on the tray list screen 901 has been selected and the similar product list display button 913 has been pressed. If accepted, the process proceeds to step S421, and if not, the process proceeds to step S408. In step S421, a similar product list screen is displayed, which will be described later with reference to FIG. 13.

[0081] In step S408, the store client 104 judges whether the batch printing or the designated printing has been accepted, that is, whether any position in the image 903 of the tray list screen 901 has been selected (multiple selections are possible) and the designated printing button 914 has been pressed, or whether the batch printing button 915 on the tray list screen 901 has been pressed. If the request has been accepted, the process flow is terminated and the process flow of FIG. 5 is returned to. If not, the process returns to step S402. If the batch printing button 915 is pressed, product labels are printed for products corresponding to all positions displayed in the image 903, but in the example of FIG. 9, since there is a position where a product is not yet determined ("not yet determined" is displayed superimposed), the batch printing button 915 is grayed out so that it cannot be pressed. If the designated printing button 914 is pressed, product labels are printed for products corresponding to the positions selected in the image 903. However, if the selected positions include a position where a product is not yet determined, a warning message (not shown) is output, but the designated printing button 914 may be controlled so that it cannot be pressed.

[0082] When returning to step S402 from each step, if the product determination table 1610 has been updated, the contents of the tray list screen 901 are updated and displayed in accordance with the updated contents.

[0083] In step S411, the shop client 104 displays a candidate product list screen. An example of the candidate product list screen is shown in FIG.

[0084] Candidate product list screen 1001 is a screen that displays a list of candidate products detected for a selected position. It displays a message 1002 according to the status of the candidate product, a photographed product image 1003, a detected candidate product image 1004, and a detection result 1005. Image 1003 is an image cut out from the image 903 of the tray list screen 901 for the selected position, candidate product image 1004 is a representative image 1504 in product table 1500 that corresponds to the product number of the detected product, and detection result 1005 references necessary data from detection result table 1600 and product table 1500.

[0085] Furthermore, a mark is displayed for a similar product present category 1006. The similar product present category 1006 is similar to the similar product present category 907 on the tray list screen 901, and therefore a description thereof will be omitted.

[0086] Furthermore, on the candidate product list screen 1001, a frame 1007 indicating that the product is being selected is displayed for the selected position. In addition, if the displayed product is not scheduled for production, that is, if the product number of the product is not registered in product number 1523 of the record corresponding to the current period in schedule list 1520, the product is superimposed with the message "Not scheduled" as in 1008.

[0087] The candidate product list screen 1001 has various function buttons (1011 to 1015) at the bottom, and the functions of each will be explained in the following processing flow.

[0088] In step S412, the store client 104 determines whether a selection decision has been received, that is, whether any candidate product on the candidate product list screen 1001 has been selected and the selection decision button 1011 has been pressed. If the selection decision has been received, the store client 104 proceeds to step S413; if not, the store client 104 proceeds to step S414.

[0089] In step S413, the shop client 104 updates the product information for the location, that is, the product number 1607 in the product determination table 1610, based on the received content, and returns to step S411.

[0090] In step S414, the store client 104 determines whether a product designation has been accepted, that is, whether the product designation button 1012 on the candidate product list screen 1001 has been pressed. If accepted, the process proceeds to step S415; if not, the process proceeds to step S417.

[0091] In step S415, the store client 104 displays a product designation screen and accepts product designation. An example of the product designation screen will be described with reference to FIG.

[0092] Fig. 11 shows a display form of the candidate product list screen 1001 different from that of Fig. 10. On the candidate product list screen 1001 in Fig. 11, the candidate product is not displayed because the product for the position selected on the tray list screen 901 has not been determined. Therefore, in order to specify the product for that position, the product specification button 1012 is pressed, and the product specification screen 1101 is displayed.

[0093] The product designation screen 1101 displays a product number input field 1102, a decision button 1103, a search button 1104, and a product image 1105. By manually entering a product number in the product number input field 1102, or by pressing the search button 1104 to display a product number search screen (not shown) to execute a search and enter the product number in the product number input field 1102, a product image 1105 is displayed for confirmation, and the product number is confirmed by pressing the decision button 1103. After that, when the user returns to the candidate product list screen 1001, the product with the product number confirmed as a candidate product is displayed in a selected state.

[0094] It should be noted that the same product designation screen is also displayed when the product designation button 911 is pressed on the tray list screen 901 (step S404).

[0095] In step S416, the shop client 104 updates the product information for the location, that is, the product number 1607 in the product determination table 1610, based on the received content, and returns to step S411.

[0096] In step S417, the store client 104 judges whether a similar product list display has been accepted, that is, whether any candidate product has been selected on the candidate product list screen 1001 and the similar product list display button 1013 has been pressed. If accepted, the process proceeds to step S421, and if not, the process proceeds to step S418. In step S421, the similar product list screen is displayed, and the similar product list screen will be described later with reference to FIG.

[0097] In step S418, the store client 104 determines whether the tray list display has been accepted, that is, whether the tray list display button 1014 on the candidate product list screen 1001 has been pressed. If accepted, the process proceeds to step S402; if not, the process proceeds to step S419.

[0098] In step S419, the store client 104 determines whether the designated print has been accepted, that is, whether the designated print button 1015 on the candidate product list screen 1001 has been pressed. If accepted, this processing flow ends and the processing flow returns to that of FIG. 5. If not, the processing returns to step S411. When the designated print button 1015 is pressed, a product label is printed for the currently selected product, but when a product is undecided, as in the case before product selection in FIG. 11, that is, when no product has been selected, the designated print button 1015 may be controlled so that it cannot be pressed, or a warning message (not shown) may be output when the button is pressed.

[0099] When returning to step S411 from each step, if the product determination table 1610 has been updated, the contents of the candidate product list screen 1001 are updated and displayed in accordance with the updated contents.

[0100] Fig. 12 shows a display form of the candidate product list screen 1001 different from Fig. 10 and Fig. 11. On the candidate product list screen 1001 in Fig. 12, since only one candidate product has been detected at the position selected on the tray list screen 901, that is, there are no other candidate products, a confirmation message such as 1201 is output, and if the selected candidate product is acceptable, pressing of a confirmation button 1202 is accepted, and the screen returns to the tray list screen. If not, operations such as product designation are accepted.

[0101] In step S421, the store client 104 displays a similar products list screen. An example of the similar products list screen is shown in FIG.

[0102] The similar product list screen 1301 is a screen that displays a list of products similar to a selected product. A photographed product image 1302 corresponding to the selected product, a selected product image 1303 and a detection result 1304, and a similar product image 1305 and a detection result 1306 are displayed in a list. The image 1302 is an image cut out from the image 903 of the tray list screen 901 at the selected position, or the product image 1003 of the candidate product list screen 1001 is displayed. The selected product image 1303 and the similar product image 1305 refer to the representative image 1504 of the product table 1500 corresponding to the product number of the product, and the detection results 1304 and 1306 refer to necessary data from the detection result table 1600 and the product table 1500. Here, in the case of a detected product, the reliability is displayed as in 1304, but in the case of a product that was not detected, the reliability is blank as in 1306.

[0103] In addition, a frame 1307 is displayed to indicate whether the displayed product is selected.

[0104] In addition, if the displayed product is not scheduled for production, that is, if the product number of the product is not registered in product number 1523 of the record corresponding to the current period in schedule list 1520, the product is superimposed with the message "Not scheduled" as shown in 1308.

[0105] The similar products list screen 1301 has various function buttons (1311 to 1314) at the bottom, and the functions of each will be explained in the following processing flow.

[0106] In step S422, the store client 104 determines whether the selection decision has been accepted, that is, whether any similar product has been selected on the similar products list screen 1301 and the selection decision button 1311 has been pressed. If the selection decision has been accepted, the process proceeds to step S423; if not, the process proceeds to step S424.

[0107] In step S423, the shop client 104 updates the product information for the location, that is, the product number 1607 in the product determination table 1610, based on the received content, and returns to step S421.

[0108] In step S424, the store client 104 determines whether a product designation has been accepted, that is, whether the product designation button 1312 on the similar products list screen 1301 has been pressed. If accepted, the process proceeds to step S425; if not, the process proceeds to step S427.

[0109] In step S425, the store client 104 displays a product designation screen to accept product designation. This process and the product designation screen are similar to step S415 and the product designation screen 1101 in FIG. 11, so a description thereof will be omitted.

[0110] In step S426, the shop client 104 updates the product information for the location, that is, the product number 1607 in the product determination table 1610, based on the received content, and returns to step S421.

[0111] In step S427, the store client 104 determines whether a request to display a candidate product list has been accepted, that is, whether the candidate product list display button 1313 on the similar products list screen 1301 has been pressed. If accepted, the process proceeds to step S411; if not, the process proceeds to step S428.

[0112] In step S428, the store client 104 determines whether the tray list display has been accepted, that is, whether the tray list display button 1314 on the similar products list screen 1301 has been pressed. If accepted, the process proceeds to step S402; if not, the process returns to step S421.

[0113] When returning to step S421 from each step, if the product determination table 1610 has been updated, the contents of the similar products list screen 1301 are updated and displayed in accordance with the updated contents.

[0114] An example of the tray list screen after the product has been determined is shown in Fig. 14. In the tray list screen 901 in Fig. 14, the "undetermined" product position has been resolved (compared to Fig. 9), so the batch print button 915 can be pressed and the message 1401 has also been changed.

[0115] This completes the explanation of Figs. 6 and 7.

[0116] Returning to the explanation of Figure 5.

[0117] In step S309, the store client 104 transmits the determined product information to the label output device 106, and in step S310, the label output device 106 receives the transmitted product information. The product information transmitted and received here is information necessary for label printing, and includes information on all positions in the product determination table 1610 when the batch print button (915) is pressed, and information on the selected position in the product determination table 1610 when the designated print button (914, 1015) is pressed, as well as the JAN code 1502 and product name 1503 in the product table 1500 corresponding to the product number, and further the price 1514 in the product price table 1510 corresponding to the JAN code 1502 and the current date. Note that, in the second and subsequent printing operations, when the batch print button is pressed, only information corresponding to records in which the printed 1617 in the product determination table 1610 is "error" or blank may be transmitted and received.

[0118] In step 311, the label output device 106 prints a product label based on the received product information. An example of the printed product label in the case of batch printing is shown in FIG.

[0119] In the printout 2200, product labels 2201 are printed consecutively in the order in which the foods are arranged on the tray. In this embodiment, the order is from the rear left of the tray (top left in the image) to the right, then to the front (bottom in the image) and then from left to right. This allows the person in charge to attach product labels to the products in the order in which they were printed.

[0120] In step S312, the label output device 106 transmits the print result to the store client 104, and in step S313, the store client 104 receives the transmitted print result. The contents transmitted and received include the location ID, the product number, and the print status (normal / error).

[0121] In step S314, the store client 104 tallyes up the results based on the received print results. First, for the relevant location ID 1611 and product number 1616 in the product determination table 1610, the print status is set in Printed 1617. That is, if the result is normal, "Printed" is set, and if an error occurs, "Error" is set. If the Batch Print button is pressed again, records with "Printed" in Printed 1617 are excluded from the list, so that duplicate product labels can be prevented from being printed. Next, the print history table is updated. An example of the print history table is shown in FIG.

[0122] Printing result table 1620 is a table for managing the schedule and results for a target scheduled period, and has items including scheduled period From 1621, scheduled period To 1622, product number 1623, planned production quantity 1624, and printing result 1625. These correspond to scheduled period From 1521, scheduled period To 1522, product number 1523, and planned production quantity 1524 in schedule list 1520. Printing result 1625 indicates the number of prints in the target scheduled period.

[0123] In step S314, for the product number 1623 during the target period, the number of prints for which the print status has become “printed” is added to the print result 1625.

[0124] This completes the explanation of FIG.

[0125] In the above display example, even if the product number of the target product is not registered in the product number 1523 of the record corresponding to the current period in the plan list 1520, the product is displayed and "Not scheduled" is superimposed on it, but if there is no possibility that an unscheduled product will be a candidate, the product may be excluded from the display targets. In that case, the tray list screen will display products ranked second or lower, or if there are no products ranked second or lower, it will display "Undecided."

[0126] Next, we will explain the case where there is a limit to the number of types of products that can be placed on one tray. For example, the basic rule is that only the same type of products are allowed, but up to two types are allowed to be mixed.

[0127] FIG. 17 is a flow chart showing an example of the process in this case.

[0128] In step S501, the store client 104 acquires the upper limit of the number of types as a parameter. The upper limit of the number of types is registered in the upper limit of the number of types 1542 of the parameter table 1540 shown in Fig. 15(e). In addition, it may be set daily or periodically using an editing screen (not shown). The upper limit of the number of types is a value of 1 when mixed loading is not permitted, and is a value of 2 or more when mixed loading is permitted.

[0129] In step S502, the store client 104 displays the tray list screen 901 (FIG. 18). This process and the processes before and after are similar to step S401 (FIG. 6).

[0130] In step S502, the store client 104 determines whether the number of types of products is greater than the upper limit. If so, the process proceeds to step S504, otherwise the process flow ends with the tray list screen remaining as it is.

[0131] In step S504, the store client 104 displays a warning on the tray list screen. An example of the warning display is shown in Fig. 18. On the tray list screen 901 in Fig. 18, the upper limit of the number of types is three, but four types of products have been detected or determined, so a message 1801 indicating this is displayed. Also, to indicate where each type of product is placed on the tray, a number is assigned to each type of product and displayed as shown in 1802. Note that the type of product may be identified using a figure or mark instead of a number.

[0132] This is the end of the explanation of FIG.

[0133] Next, a case where the similar products list is updated during system operation will be described.

[0134] FIG. 19 is a flow chart showing an example of the process in this case.

[0135] In step S601, the shop client 104 displays the candidate product list screen 1001 (FIGS. 20 and 21). This process and the processes before and after are similar to step S411 (FIG. 6).

[0136] In step S602, the store client 104 determines whether a selection decision has been received, that is, whether any candidate product has been selected on the candidate product list screen 1001 (FIG. 20) and the selection decision button 1011 has been pressed. If the selection decision has been received, the process proceeds to step S603; if not, the process proceeds to step S604.

[0137] In step S603, the store client 104 judges whether the candidate product for which the selection has been accepted is a similar product to the selected product. In the example of Fig. 20, the store client 104 judges whether the product with product number 0006 for which the selection has been accepted is a similar product to the selected product with product number 0002, that is, whether it is registered in the same similar group 1531 in the similar product list 1530. If it is a similar product, it has already been registered, so the process flow ends without doing anything. If not, the process proceeds to step S606.

[0138] In step S604, the store client 104 determines whether a product designation has been accepted, that is, whether the product designation button 1012 on the candidate product list screen 1001 (FIG. 21) has been pressed. If accepted, the process proceeds to step S605; if not, the process flow ends.

[0139] In step S605, the store client 104 judges whether the product for which the product designation has been accepted is a similar product to the selected product. In the example of Fig. 21, the store client 104 judges whether the product with product number 0009 for which the product designation has been accepted on the product designation screen 1101 is a similar product to the selected product with product number 0002, that is, whether it is registered in the same similar group 1531 in the similar product list 1530. If it is a similar product, it has already been registered, so nothing is done and this processing flow ends. If not, the process proceeds to step S606.

[0140] In step S606, the store client 104 displays the similar product registration confirmation screen 2001 (FIGS. 20 and 21) and accepts the confirmation result. If the "Register" button 2002 is pressed, the process proceeds to step S607, and if the "Do not register" button 2003 is pressed, this process flow is terminated. In step S607, the store client 104 additionally registers the product number of the product for which the selection or product designation has been accepted in the target similar group 1531 of the similar products list 1530, and ends this processing flow.

[0141] This is the end of the explanation of FIG.

[0142] The above describes an embodiment, but the program in the present invention is a program that enables a computer to execute the processing method of the flowcharts shown in the figures, and the storage medium of the present invention stores a program that enables a computer to execute the processing method of the figures.

[0143] The program in the present invention may be a program for each processing method of each device in each drawing.

[0144] As described above, it goes without saying that the object of the present invention can be achieved by supplying a recording medium on which a program that realizes the functions of the above-mentioned embodiments is recorded to a system or device, and having the computer (or CPU or MPU) of that system or device read and execute the program stored on the recording medium.

[0145] In this case, the program itself read out from the recording medium will realize the novel functions of the present invention, and the recording medium on which the program is recorded will constitute the present invention.

[0146] Examples of recording media for supplying the program include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EPROMs, silicon disks, etc.

[0147] Furthermore, it goes without saying that the functions of the above-mentioned embodiments are not only realized by the computer executing a program it has read, but also includes cases where an OS (operating system) running on the computer performs some or all of the actual processing based on the instructions of the program, and the functions of the above-mentioned embodiments are realized through that processing.

[0148] Furthermore, it goes without saying that this also includes cases where a program read from a recording medium is written into a memory provided on a function expansion board inserted into a computer or a function expansion unit connected to a computer, and then a CPU or the like provided on the function expansion board or function expansion unit performs some or all of the actual processing based on the instructions of the program code, thereby realizing the functions of the above-mentioned embodiments.

[0149] Furthermore, the present invention may be applied to a system made up of a plurality of devices, or to an apparatus made up of a single device.

[0150] Needless to say, the present invention can also be applied to a case where the invention is achieved by supplying a program to a system or device. In this case, the effect of the present invention can be achieved by reading a recording medium storing a program for achieving the present invention into the system or device.

[0151] Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. Note that the present invention also includes configurations that combine the above-mentioned embodiments and their modified examples. [Explanation of symbols]

[0152] 100 Information Processing Systems 101 Learning Server 102 Business Server 103 Store Edge Device 104 store clients 105 Network Camera 106 Label output device 107 Internet 108 LAN

Claims

1. a control means for controlling the maximum number of product types that may be identified at one time; An acquisition means for acquiring an identification result of an item identified from an image of the item; A display control means for controlling display of an image of the product and information related to the specified product; Equipped with The information processing device is characterized in that the display control means displays whether the number of the specified product types exceeds the upper limit number in an identifiable manner.

2. The information processing apparatus according to claim 1 , wherein the display control means displays the number of types of the identified products.

3. The information processing apparatus according to claim 1 , wherein the display control means displays the upper limit number.

4. 2. The information processing apparatus according to claim 1, wherein the display control means displays an identification object for each type of product.

5. A method for controlling an information processing device having a management unit for managing an upper limit of the number of types of products that can be specified at one time, comprising: An acquisition step in which an acquisition means acquires an identification result of identifying a product from an image of the product; a display control step in which a display control means controls to display an image of the product and information related to the specified product; Equipped with The control method for an information processing device, wherein in the display control step, whether the number of types of the identified products exceeds the upper limit number is displayed in an identifiable manner.

6. An information processing device, a control means for controlling the maximum number of product types that may be identified at one time; An acquisition means for acquiring an identification result of an item identified from an image of the item; A display control means for controlling display of an image of the product and information related to the specified product; A program for causing the device to function as a The display control means displays whether the number of types of the identified products exceeds the upper limit number.

Images