Information display system, processing device and method used in said information display system

Abstract

This reduces the processing load of overlaying virtual labels containing discount information onto camera footage of products. [Solution] The information display system comprises a camera that photographs multiple products and generates camera images, a display that displays the camera images, and a processing device that recognizes and reads codes attached to each product from the camera images, and overlays a virtual label containing discount information for each product onto the camera images and displays it on the display based on the information recorded in the codes. The processing device detects changes in the position of each code in the camera images by comparing multiple frames of the camera images, and for each code, if the amount of change in position exceeds a threshold, it rereads the code and updates the display of the virtual label, and if the amount of change in position does not exceed the threshold, it maintains the display content of the virtual label without rereading the code, and updates the display of discount information on the virtual label corresponding to the code at a predetermined timing based on the information recorded in the code.

Description

Technical Field

[0001] The present invention relates to an information display system, a processing apparatus, and a method used in the information display system.

Background Art

[0002] In stores that sell foods such as boxed lunches and prepared dishes, when the expiration date of the displayed products approaches, a label indicating a price discount is often attached to the products. Such labels describe information indicating the degree of price discount, such as "100 yen off", "30% discount", or "50% discount". The store staff compares the expiration date of each product with the current time and performs the task of attaching a label displaying an appropriate discount amount or discount rate to each product at an appropriate timing.

[0003] Patent Document 1 discloses a technique for reducing the burden of the above-described label attachment work. The system disclosed in Patent Document 1 aims to enable a customer who visits the store to confirm the details of the benefits of the product without attaching a label on which the details of the benefits such as a price discount are described. Specifically, the system includes a storage unit, a photographing unit, a display unit, and a processing unit. The storage unit stores in association with each other product ID information set for each product, additional information set for each product, and benefit information set for each additional information. The photographing unit acquires an image including a first code attached to the product and representing the product ID information and a second code attached to the product and representing the additional information. The display unit displays the image acquired by the photographing unit as a real-time video. The processing unit acquires the product ID information represented by the first code and the additional information represented by the second code from the image acquired by the photographing unit, determines the benefit based on the benefit information stored in the storage unit in association with the acquired product ID information and additional information, and causes the details of the benefit to be displayed at a predetermined position with respect to the second code in the real-time video.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

[0005] In a system like the one exemplified in Patent Document 1, the product may be interrupted or moved by a consumer or staff member while the details of a discount or other benefit are being displayed in a video. Even in such cases, it is required that the display of the benefit in the video (hereinafter referred to as the "virtual label") be updated in real time. However, if there are many products being filmed, it becomes difficult to update the display of the virtual label corresponding to the code in real time, and a time lag may occur in the display of the virtual label.

[0006] This invention provides a technology that enables high-speed processing of multiple codes in order to keep the display of discounts and other benefits in a video in near real-time. [Means for solving the problem]

[0007] An information display system according to one aspect of the present invention is a system that photographs a plurality of displayed products, reads a code attached to each product, and displays a virtual label including discount information for each product overlaid on the camera image of the plurality of products based on the information recorded in the code. The system comprises a camera that photographs the plurality of products and generates a camera image, a display that displays the camera image, and a processing device that recognizes and reads a code attached to each product from the camera image, and displays a virtual label including discount information for each product overlaid on the camera image on the display based on the information recorded in the code. The processing device detects changes in the position of each code in the camera image by comparing a plurality of frames of the camera image, and for each code, if the amount of change in position exceeds a threshold, it rereads the code and updates the display of the virtual label, and if the amount of change in position does not exceed the threshold, it maintains the display content of the virtual label without rereading the code. For each code, the processing device updates the display of discount information on the virtual label corresponding to the code at a predetermined timing based on the information recorded in the code.

[0008] The comprehensive or specific embodiments of this disclosure may be implemented by apparatus, systems, methods, integrated circuits, computer programs, or computer-readable non-temporary storage media, or any combination thereof. Computer-readable storage media may include volatile storage media or non-volatile storage media. Apparatus may consist of multiple devices. If apparatus consists of two or more devices, these two or more devices may be located in a single device or in two or more separate devices. [Effects of the Invention]

[0009] According to the present invention, even if the photograph of the product is obstructed by the hands of a consumer or staff member, or by another object, or if the product moves, it becomes easy to keep the display of the virtual label in real time. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of a store where an information display system is used. [Figure 2] This figure shows an example of a product. [Figure 3] This figure shows an example of a virtual label. [Figure 4] This figure shows an example of camera footage below the display. [Figure 5] This is a block diagram showing an example configuration of an information display system. [Figure 6] This figure shows an example of a database stored in a storage device. [Figure 7] This figure shows an example of a display screen for editing the correspondence between identifiers and discount conditions. [Figure 8] This flowchart shows a concrete example of the processing performed by the processing unit. [Figure 9A] This figure shows an example of a detection area that contains multiple products in a camera image and the codes attached to them. [Figure 9B] This figure shows an example where the detected area does not overlap with the previously detected area. [Figure 10] This flowchart shows the process flow for reading code and displaying virtual labels. [Figure 11] This figure shows an example of a situation where the recording of the code is being obstructed by human hands. [Modes for carrying out the invention]

[0011] Exemplary embodiments of the present invention will be described below. However, detailed descriptions may be omitted if they are not necessary. For example, detailed descriptions of well-known matters or redundant descriptions of substantially identical configurations may be omitted. This is to avoid making the following descriptions unnecessarily redundant and to facilitate the understanding of those skilled in the art. The inventors provide the accompanying drawings and the following descriptions to enable those skilled in the art to fully understand the present disclosure, and do not intend to limit the subject matter described in the claims thereby. In this specification, the same or similar components are denoted by the same reference numerals.

[0012] The following embodiments are exemplary, and the present invention is not limited to the following embodiments. For example, the numerical values, shapes, steps, order of steps, layout of the display screen, etc. shown in the following embodiments are merely examples, and various modifications are possible. Also, it is possible to combine one aspect with another aspect.

[0013] FIG. 1 is a diagram showing an example of a store in which an information display system according to an embodiment of the present invention is used. The store is a retail store that sells food products, such as a supermarket, a bento shop, a prepared food store, a discount store, etc. The store sells food products with a defined expiration date or shelf life, such as bento, prepared foods, fresh foods, processed foods, beverages, etc.

[0014] In the example shown in FIG. 1, a plurality of products 50 (food products), such as bento and prepared foods, are displayed in the store. Near the product 50, a display 120 and a processing device 130 are installed. A camera 110 is installed on the ceiling of this store.

[0015] Camera 110 is positioned so that multiple products 50 are within its field of view. Camera 110 may be a camera that is already installed in the store, such as a surveillance camera, or it may be a dedicated camera installed for this system. Camera 110 may be mounted on a stand that is positioned to photograph multiple displayed products 50, not just on the ceiling. Camera 110 is configured to photograph multiple products 50 and generate camera footage (moving images). Display 120 is a display such as an LCD or OLED, and displays the camera footage generated by camera 110. Processing unit 130 is a computer that processes the camera footage and displays it on display 120.

[0016] The camera 110 and the processing unit 130 are connected by wire or wireless, and the camera video generated by the camera 110 is processed by the processing unit 130. The camera 110 and the processing unit 130 may also be connected via relay equipment such as a switching hub or a wireless access point. The processing unit 130 is connected to the display 120 via a cable, and the camera video can be displayed on the display 120.

[0017] FIG. 2 is a diagram showing an example of the product 50. The product 50 in this example is a bento box, and on its package, there are a label 51 including information such as the product name, price, raw materials, nutritional components, expiration date, etc., and a barcode 52, and a label 53 including a two-dimensional code 54 attached thereto. The two-dimensional code 54 in the example of FIG. 2 is a Data Matrix code. The two-dimensional code 54 is not limited to the Data Matrix code and may be other types of two-dimensional codes such as a QR code (registered trademark). Instead of providing the label 53 including the two-dimensional code 54 independently, the label 51 may include the two-dimensional code 54. When the amount of information to be recorded is small, a one-dimensional code such as the barcode 52 may also serve the role of the two-dimensional code 54 instead of the two-dimensional code 54. The description regarding the two-dimensional code 54 in the following explanation is similarly applicable even when the two-dimensional code 54 is replaced with a one-dimensional code. In the following explanation, the two-dimensional code 54 may sometimes be simply referred to as "code 54".

[0018] The code 54 records information including an identifier (ID) associated with the discount information of the product 50. The processing device 130 reads the code 54 attached to the product 50 photographed by the camera 110, and determines whether there is a discount for the product 50 based on the information recorded in the code 54. The presence or absence of a discount and the content of the discount are separately determined for each identifier recorded in the code 54 and are recorded in a database. The database is stored in a storage device inside or outside the processing device 130. The processing device 130 refers to the database to determine the presence or absence of a discount for each product 50, and when there is a discount, overlays a virtual label including the discount information on the camera image and displays it on the display 120.

[0019] Figure 3 shows an example of a virtual label 70. The virtual label 70 is not an actual label, but an image that mimics a label displayed only on the display 120. The processing unit 130 displays the virtual label 70 at the position of code 54 in the image, or at a position shifted by a predetermined amount in a predetermined direction from that position. In the example in Figure 3, the virtual label 70 includes a description indicating a discount rate of "20% off". Not limited to "20% off", virtual labels 70 indicating any discount rate or discount amount, such as "10% off", "30% off", "50 yen off", "100 yen off", or "half price", can be superimposed on the camera image. The processing unit 130 determines the virtual label 70 by referring to a database that shows which product corresponds to which discount amount or discount rate.

[0020] Figure 4 shows an example of camera footage displayed on display 120. In this example, the camera footage includes five different products 50A, 50B, 50C, 50D, and 50E with different discount information. Product 50A does not have a virtual label indicating a discount. Product 50B has a virtual label 70B indicating "10% off". Product 50C has a virtual label 70C indicating "20% off". Product 50E has a virtual label 70E indicating "50 yen off". Product 50E also has a virtual label 70E indicating "half price". Thus, whether or not a virtual label is displayed, and the content of the discount included in the virtual label, may vary depending on the product. The presence or absence of a discount and the content of the discount are determined for each product based on the product's manufacturing date, or best-before or expiration date, and the current date and time.

[0021] Customers can check whether each of the 50 displayed items is discounted, as well as the discount amount or percentage, by looking at the display 120. In this way, by overlaying virtual labels 70 on each discounted item 50 in the camera image, customers can be informed of the presence and details of discounts without actually attaching labels containing discount information. This eliminates the need to attach discount labels to each item 50, leading to labor savings for store staff and reductions in label costs.

[0022] In such a system, while multiple products 50 are being photographed and virtual labels 70 are being displayed on the display 120, the photography of the products 50 may be obstructed or the products 50 may move due to the actions of customers or staff. Even in such cases, it is required that the display of the virtual labels 70 in the camera image be updated in real time. Therefore, in this embodiment, the processing unit 130 is configured to track the position of the code 54 attached to each product 50 in the camera image, and to reread only the code 54 whose position has changed relatively significantly and update the virtual label 70. This reduces the amount of processing and enables real-time tracking of the display of the virtual labels 70.

[0023] The configuration and operation of the information display system 100 of this embodiment will be described in more detail below.

[0024] Figure 5 is a block diagram showing an example configuration of the information display system 100. In this example, the information display system 100 includes a camera 110, a display 120, a processing unit 130, as well as a storage device 140 and an input device 150. The storage device 140 is a device that stores data (database) showing the correspondence between identifiers (IDs) recorded in code 54 and discount information. In the example of Figure 5, the storage device 140 is a device independent of the processing unit 130, but it may be built into the processing unit 130. The input device 150 is a device used by a user (e.g., store staff) to edit the discount information for each product. The input device 150 may be, for example, a keyboard, a mouse, or a touchscreen.

[0025] The processing unit 130 is any computer, such as a personal computer (PC). In this embodiment, the processing unit 130 is connected to the display 120, but a computer with a built-in display 120 may be used as the processing unit 130. Alternatively, an external computer (e.g., a server) that communicates with the computer connected to the display 120 via wired or wireless connection may be used as the processing unit 130.

[0026] Figure 6 shows an example of a database (table) stored in the storage device 140. The table shown in Figure 6 includes information such as an identifier (ID), manufacturing date and time, expiration date, first discount timing, first discount details, second discount timing, second discount details, third discount timing, and third discount details. The table may also include discount timings and details for the fourth and subsequent discounts. Depending on the product, the best-before date may be recorded instead of the expiration date. Each discount timing may be specified by a number indicating how many hours (or days) before the expiration date (or best-before date) or how many hours (or days) after the manufacturing date and time. Each record in the table may include information such as the product identifier and product name.

[0027] Thus, code 54 records an identifier for determining a virtual label 70 containing discount information. The storage device 140 stores data that defines the correspondence between the identifier, one or more timings in which the discount is applied, and the discount amount or discount rate at each timing. The processing device 130 can refer to this data to determine the virtual label 70 containing discount information corresponding to the identifier recorded in code 54. More specifically, the processing device 130 determines the corresponding virtual label 70 containing discount information based on this data and the current time. The current time information can be obtained from a timing circuit such as a real-time clock provided by the processing device 130.

[0028] Figure 7 shows an example of a display screen for editing the above correspondence. Such a display screen can be shown on display 120 or other displays. By displaying a display screen like the one exemplified in Figure 7 on display 120 or other displays, the processing unit 130 can provide a user (for example, a system user such as store staff) with the function to set discount conditions for each ID. The user can use the input device 150 to input the ID, manufacturing date and time, expiration date, and one or more discount conditions on the display screen shown in Figure 7. Each discount condition can be set, for example, by specifying whether to use the manufacturing date and time or the expiration date (or best before date) as the reference date and time, how many hours before or after the reference date and time the discount should be applied, and the discount rate or discount amount. When the user enters each input item for each ID and presses the "OK" button, a record is added to the table shown in Figure 6. In this way, the processing unit 130 can be configured to edit the correspondence between ID, discount timing, and discount amount or discount rate based on user input. Note that such editing functionality may be provided by a computer separate from the processing unit 130.

[0029] In this embodiment, the processing unit 130 recognizes and reads the code 54 attached to each product 50 from the camera image, and displays a virtual label 70 containing discount information for each product 50 on the display 120, overlaid on the camera image, based on the information (ID, etc.) recorded in the code 54. Furthermore, the processing unit 130 detects changes in the position of each code 54 in the camera image by comparing multiple frames of the camera image, and performs the following processing for each code 54. If the change in position exceeds the threshold, reread code 54 and update the display of virtual label 70. If the change in position does not exceed the threshold, the display content of the virtual label 70 is maintained without rereading code 54.

[0030] In other words, the processing unit 130 rereads the code 54 and updates the display on the virtual label 70 only when the position of the code 54 in the camera image changes relatively significantly. This reduces the frequency of reading the code 54, thereby suppressing processing volume and processing time. As a result, even when there are many products 50 being photographed, the display on the virtual label 70 can be made to follow the changes in the position of the products 50.

[0031] In this embodiment, the processing unit 130 hides the virtual label 70 corresponding to a code 54 in the camera image when the code 54 is obscured by the consumer's hand or another object, and re-reads the code 54 and re-displays the virtual label 70 corresponding to the code 54 when the code 54 becomes included in the camera image again. In other words, the virtual label 70 is hidden when the code 54 is hidden in the camera image, and the virtual label 70 is re-displayed when the code 54 is re-displayed. This processing makes it possible to achieve a display of the virtual label 70 that is natural for the user.

[0032] The processing unit 130 updates the display of discount information on the virtual label 70 corresponding to code 54 at a predetermined timing based on the information recorded in code 54. For example, the processing unit 130 may be configured to update the display of discount information (discount amount or discount rate) on the virtual label 70 based on a table as shown in Figure 6 and the current time.

[0033] In this embodiment, the processing unit 130 displays a virtual label 70 corresponding to each code 54 on the display 120 so as to overlap the position of the code 54 in the camera image. As a result, the virtual label 70 is displayed without the code 54 being displayed in the camera image, thus achieving a natural display for the user.

[0034] The processing unit 130 may be configured to extract frames from camera images at predetermined intervals, extract a region containing each code 54 for each frame, and determine the amount of change in the position of each code 54 based on the change in the region's position between frames. The predetermined interval is, for example, 0.1 seconds or more and 0.5 seconds or less, and in one example, 0.2 seconds. By processing at such a relatively short interval, the changes in the display of the virtual label 70 can be made smoother.

[0035] The processing unit 130 may perform a binarization process for each frame, extract a rectangular region having the size and shape corresponding to each code 54 from the binarized frame, and determine the amount of change in the position of each code 54 based on the change in the position of the rectangular region between frames. Alternatively, the processing unit 130 may perform a resolution reduction process for each frame before extracting the rectangular region. The resolution reduction process may be performed before the binarization process. For example, the processing unit 130 may perform a resolution reduction process for each frame to 1 / 2 or less and 1 / 5 or more (1 / 3 in one example), then perform the binarization process and extract the rectangular region. Such processing allows for faster reading of the code 54 and updating of the virtual label 70.

[0036] The following describes a more specific example of the virtual label update process performed by the processing unit 130, with reference to Figures 8 and 9.

[0037] Figure 8 is a flowchart illustrating a specific example of processing performed by the processing unit 130. The flowchart shown in Figure 8 includes the processing steps S101 to S114. The processing unit 130 may be configured to perform these processes at a frequency of, for example, about 5 times per second (once every 200 milliseconds). The operation of each step will be described below.

[0038] In step S101, the processing unit 130 acquires image data for one frame (hereinafter also referred to as "camera image") from the camera video data generated by the camera 110.

[0039] In step S102, the processing unit 130 reduces the resolution of the camera image. For example, it uses a predetermined filter to reduce the resolution of the camera image to 1 / 2 or less and 1 / 5 or more (for example, 1 / 3). This reduces the load on each subsequent process.

[0040] In step S103, the processing unit 130 performs a binarization process on the camera image with reduced resolution. This separates the camera image into black regions and white regions.

[0041] In step S104, the processing unit 130 extracts black regions from the binarized camera image.

[0042] In step S105, the processing device 130 analyzes the size and shape (rectangularity) of each extracted black region.

[0043] In step S106, the processing unit 130 identifies an image region that is thought to correspond to code 54 based on the size and shape analysis results.

[0044] In step S107, the processing unit 130 determines the reading angle and direction of the code 54 for each image region that is thought to correspond to the code 54. This determination is made by detecting characteristic black pattern portions in the code 54. For example, as shown in Figure 2, if the code 54 is a data matrix code, the L-shaped black pattern in the lower left is detected, and the reading angle of the code 54 is determined based on its orientation.

[0045] In step S108, the processing unit 130 determines a detection area for each code 54. The detection area is an area that encloses the code 54 and bulges outward in the left-right and up-down directions by a predetermined offset amount from the outer edge of the code 54. The offset amount may be set to a value of, for example, half the length of one side of the code 54. The offset amount is set to determine whether or not there has been a change in the position of the code 54.

[0046] Figure 9A is a diagram illustrating an example of a detection region. Figure 9A shows an example of a detection region 74 that encompasses multiple products 50 in a camera image and the codes 54 attached to them. As shown by the dotted line frame in Figure 9A, the processing device 130 determines a detection region 74 that encompasses each code 54 detected from the camera image. The detection region 74 is an example of the "rectangular region having a size and shape corresponding to the code 54" mentioned above.

[0047] In step S109, the processing unit 130 determines whether there is a code 54 in which the detected area in the current frame does not overlap with the corresponding detected area in the previous frame. If such a code 54 exists, the process proceeds to step S110; otherwise, the process proceeds to step S113.

[0048] Figure 9B shows an example where the detected area in the current detection does not overlap with the corresponding detected area in the previous detection. In the example shown in Figure 9B, the lower left product 50 has changed from its corresponding position in the previous frame, and the detection area 74 containing code 54 does not overlap with the corresponding detected area 74p in the previous detection. In this case, it is determined that the amount of change in the position of code 54 of the lower left product 50 exceeds the threshold, and the process proceeds to step S110, where only the code 54 of that product 50 is reread.

[0049] In the initial processing, since there is no previously detected area, it is determined that all codes 54 do not overlap with the corresponding previously detected area, and the processing from steps S110 to S112 is executed for all codes 54.

[0050] In step S110, the processing unit 130 corrects the reading angle and direction of each code 54 that is determined not to overlap with the previously detected area. This process aligns the vertical and horizontal directions of the code 54 with the horizontal and vertical directions in the image, based on the angle and direction determined in step S107.

[0051] In step S111, the processing unit 130 performs noise reduction on the image region of code 54 while preserving edges using a bilateral filter.

[0052] In step S112, the processing unit 130 reads the code 54 and determines whether to display the virtual label. Details of the process in step 112 will be described later. After step S112, the process proceeds to step S113.

[0053] In step S113, the processing unit 113 updates the display of the virtual label 70. Here, for codes 54 whose detection area overlaps with the previous detection area (i.e., small movement), the code 70 is not reread, and only the display position of the corresponding virtual label 70 is adjusted. On the other hand, for codes 54 whose detection area does not overlap with the previous detection area (i.e., large movement), the code 70 is reread in step S112, and in addition to the display position of the corresponding virtual label 70, the display content is also updated as necessary. After step S113, the process proceeds to step S114.

[0054] In step S114, the processing unit 130 stores the position information (coordinates) of the detected region in the image determined in step S108 (for example, by storing it in memory). Then, the process returns to step S101.

[0055] By performing the above processing, the processing unit 130 reads the code 54 only for products 50 where the amount of change in the position of the code 54 exceeds a threshold. This allows the virtual label 70 to be displayed with near real-time smoothness.

[0056] In this embodiment, whether the change in the position of code 54 exceeds a threshold is determined based on whether there is overlap between corresponding detection regions 74 between frames, but this is only one example. For example, the change in the position of a specific point such as the center point or corner of code 54 may be calculated between frames, and if that change exceeds a threshold, it may be determined that the position of code 54 has changed. Since the size of the detection region 74 is fixed, the method of determining whether the change is based on whether or not the detection regions 74 overlap can also be said to be substantially determining whether the change in the position of code 54 exceeds a predetermined threshold.

[0057] Next, the process in step S112 will be explained in more detail with reference to Figure 10.

[0058] Figure 10 is a flowchart showing the process flow for reading code 54 and determining whether or not to display the virtual label 70. The flowchart shown in Figure 10 includes the processes from steps S201 to S205. The processes from steps S201 to S205 are performed for each code 54 in step S109 in which it was determined that the detection area does not overlap with the previously detected area.

[0059] In step S201, the processing unit 130 reads the code 54 and obtains the identifier (ID) information recorded in the code 54.

[0060] In step S202, the processing unit 130 refers to the database stored in the storage device 140 and identifies the discount conditions corresponding to the identifier. The discount conditions are, for example, the timing and content of each discount as shown in Figure 6.

[0061] In step S203, the processing unit 130 determines whether the code 54 satisfies the display conditions for the virtual label 70. For example, the processing unit 130 determines whether to display the virtual label 70 containing the discount information on the display 120 based on the current time and the discount conditions. If the display conditions for the virtual label 70 are met, the process proceeds to step S204; otherwise, the process proceeds to step S205.

[0062] In step S204, the processing unit 130 decides to display the virtual label 70. If the virtual label 70 corresponding to the code 54 was displayed in the previous frame, the display position and content of the virtual label 70 will be updated.

[0063] In step SS205, the processing unit 130 decides to hide the virtual label 70. If the virtual label 70 corresponding to its code 54 was displayed in the previous frame, the virtual label 70 is hidden.

[0064] Through the above process, the processing unit 130 can read the code 54 and display the virtual label 70 corresponding to the code 54 on the display 120 if the predetermined display conditions are met.

[0065] In the initial processing, the processing shown in Figure 10 is performed for all codes 54 in the image. At that time, in step S202, the processing unit 130 may store the discount conditions corresponding to each code 54 in a storage device such as memory. By doing so, even for codes 54 that do not move in the image, the virtual label 70 containing the discount information can be displayed when it becomes necessary to display the virtual label 70, without having to read the code 54 again. This further reduces the processing load and enables smooth display. Alternatively, the processing unit 130 may also perform the processing shown in Figure 10 for codes 54 that do not move in the image at predetermined intervals (e.g., 1 minute, 10 minutes, 30 minutes, 1 hour, etc.). By doing so, even if the contents of the database that defines the discount conditions are changed during the operation of the system, it will be possible to display the virtual label 70 that reflects the latest discount conditions. In this specification, "codes that do not move" or "codes with little movement" means codes in which the amount of change in position between frames in the camera image is less than a threshold.

[0066] Figure 9B shows an example where the position of code 54 changes, but code 54 may be covered by a person's hand or other object, such as a consumer or staff member, preventing it from being photographed. Figure 11 shows an example of such a case. In such a case, in step S108, code 54 and its corresponding detection area, which were detected in the previous frame, will not be detected. Thus, if code 54 that was detected in the previous frame is no longer detected, and the virtual label 70 corresponding to code 54 was displayed in the previous frame, the processing unit 130 may hide the virtual label 70. Then, in the frame in which code 54 becomes detectable again, the processing unit 130 may read code 54 again and redisplay the virtual label 70. If the product 50 is lifted and returned to its original position or another position while code 54 remains detectable, the processing unit 130 may move the display of the virtual label 70 in the image to follow the movement. Such processing enables a near real-time display of the virtual label 70 that responds to the actual movement of the product 50.

[0067] The processing unit and computer program used in the information display system in the above embodiments may be manufactured and sold independently. The computer program may be provided, for example, stored in a computer-readable non-temporary storage medium. The computer program may also be provided by download via a telecommunications line (e.g., the Internet). Each of the above processes may be performed by the processor of the processing unit executing such a computer program.

[0068] As described above, this disclosure includes control systems, work vehicles, control methods, and computer programs as described in the following items.

[0069] As described above, this disclosure includes the following information display systems, processing devices, and methods.

[0070] [Item 1] An information display system that photographs multiple displayed products, reads the code attached to each product, and overlays a virtual label containing discount information for each product onto the camera image of the multiple products based on the information recorded in the code, A camera that photographs the aforementioned multiple products and generates camera images, A display that shows the camera image, A processing device that recognizes and reads the code attached to each product from the camera image, and overlays a virtual label including discount information for each product onto the camera image and displays it on the display based on the information recorded in the code. Equipped with, The aforementioned processing apparatus is By comparing multiple frames of the camera image, changes in the position of each code in the camera image are detected. For each code, If the amount of change in position exceeds the threshold, the code is reread and the display of the virtual label is updated. If the amount of change in position does not exceed the threshold, the display content of the virtual label is maintained without rereading the code. At a predetermined timing based on the information recorded in the code, the display of discount information on the virtual label corresponding to the code is updated. Information display system.

[0071] [Item 2] The information display system according to item 1, wherein the processing device displays a virtual label corresponding to each code on the display so as to overlap the position of the code in the camera image.

[0072] [Item 3] The aforementioned processing apparatus is In the aforementioned camera footage, if a code is obstructed by a consumer's hand or another object, the virtual label corresponding to the code is hidden. When the code is again included in the camera image, the code is read again and the virtual label corresponding to the code is redisplayed. The information display system described in item 1.

[0073] [Item 4] The aforementioned processing apparatus is Frames are extracted from the aforementioned camera image at predetermined intervals, For each frame, extract the region containing each code, Based on the positional changes between frames in the aforementioned region, the amount of change in the position of each code is determined. The information display system described in item 1.

[0074] [Item 5] The information display system described in item 4, wherein the period is 0.1 seconds or more and 0.5 seconds or less.

[0075] [Item 6] The aforementioned processing unit, for each frame, Perform binarization, From the frame after the binarization process, a rectangular region having a size and shape corresponding to each code is extracted for each code. Based on the positional changes between frames in the aforementioned rectangular region, the amount of change in the position of each code is determined. The information display system described in item 4.

[0076] [Item 7] The information display system described in item 6, wherein the processing device performs a resolution reduction process on each frame, and then performs the binarization process.

[0077] [Item 8] The information display system described in item 6, wherein the processing device performs a process to reduce the resolution of each frame to 1 / 2 or less and 1 / 5 or more, and then performs the binarization process.

[0078] [Item 9] The code records an identifier for determining a virtual label containing the discount information, The information display system further includes a storage device that stores data defining the correspondence between the identifier, one or more timings at which a discount is applied, and the discount amount or discount rate at each timing. The processing device refers to the data and determines a virtual label including discount information corresponding to the identifier. The information display system described in item 1.

[0079] [Item 10] The information display system according to item 9, wherein the processing device determines the virtual label, which includes corresponding discount information, based on the data and the current time.

[0080] [Item 11] The processing device is an information display system as described in item 10, which edits the correspondence relationship based on input from the user.

[0081] [Item 12] The aforementioned code is a two-dimensional code, and is part of an information display system described in any of items 1 through 11.

[0082] [Item 13] A processing device used in an information display system that photographs multiple displayed products, reads a code attached to each product, and overlays a virtual label containing discount information for each product onto the camera image of the multiple products based on the information recorded in the code, The camera recognizes and reads the code attached to each product from the camera images of the multiple products captured by the camera, and displays a virtual label containing discount information for each product overlaid on the camera image on the display based on the information recorded in the code. By comparing multiple frames of the camera image, changes in the position of each code in the camera image are detected. For each code, If the amount of change in position exceeds the threshold, the code is reread and the display of the virtual label is updated. If the amount of change in position does not exceed the threshold, the display content of the virtual label is maintained without rereading the code. At a predetermined timing based on the information recorded in the code, the display of discount information on the virtual label corresponding to the code is updated. Processing device.

[0083] [Item 14] A method for photographing multiple displayed products, reading the code attached to each product, and overlaying a virtual label containing discount information for each product onto the camera images of the multiple products based on the information recorded in the code, The system recognizes and reads the code attached to each product from the camera images of the multiple products captured by the camera, and displays a virtual label containing discount information for each product overlaid on the camera image on a display based on the information recorded in the code. By comparing multiple frames of the aforementioned camera image, changes in the position of each code in the camera image are detected, For each code, If the amount of change in position exceeds the threshold, the code is reread and the display of the virtual label is updated. If the amount of change in position does not exceed the threshold, the display content of the virtual label is maintained without rereading the code. The discount information displayed on the virtual label corresponding to the code is updated at a predetermined timing based on the information recorded in the code. A method that includes this. [Industrial applicability]

[0084] The technology disclosed herein can be used in information display systems in stores that sell food products such as bento boxes and prepared foods. [Explanation of Symbols]

[0085] 50: Product, 51: Label, 52: Barcode, 53: Label, 54: 2D code, 70: Virtual label, 100: Information display system, 110: Camera, 120: Display, 130: Processing unit

Claims

1. An information display system that photographs multiple displayed products, reads the code attached to each product, and overlays a virtual label containing discount information for each product onto the camera image of the multiple products based on the information recorded in the code, A camera that photographs the aforementioned multiple products and generates camera images, A display that shows the camera image, A processing device that recognizes and reads the code attached to each product from the camera image, and overlays a virtual label including discount information for each product onto the camera image and displays it on the display based on the information recorded in the code. Equipped with, The aforementioned processing apparatus is By comparing multiple frames of the camera image, the amount of change in the position of each of the multiple codes in the camera image is determined. For each code, If the amount of change in the position exceeds the threshold, the code is reread and the display of the virtual label is updated. If the amount of change in position does not exceed the threshold, the display content of the virtual label is maintained without rereading the code. Regardless of whether the amount of change in position exceeds the threshold, the display of discount information on the virtual label corresponding to the code is updated at a predetermined timing based on the information recorded in the code. Information display system.

2. The processing apparatus is When each code in the aforementioned camera image is read, the discount conditions corresponding to the code are stored. For codes in which the amount of change in position does not exceed the threshold, the display of the discount information in the virtual label corresponding to the code is updated at a predetermined timing based on the stored discount conditions, without reloading the code. The information display system according to claim 1.

3. The information display system according to claim 1, wherein the processing device displays a virtual label corresponding to each code on the display so as to overlap the position of the code in the camera image.

4. The aforementioned processing apparatus is In the aforementioned camera footage, if a code is obstructed by a consumer's hand or another object, the virtual label corresponding to the code is hidden. When the code is again included in the camera image, the code is read again and the virtual label corresponding to the code is redisplayed. The information display system according to claim 1.

5. The aforementioned processing apparatus is Frames are extracted from the aforementioned camera image at predetermined intervals, For each frame, extract the region containing each code, Based on the positional changes between frames in the aforementioned region, the amount of change in the position of each code is determined. The information display system according to claim 1.

6. The information display system according to claim 5, wherein the period is 0.1 seconds or more and 0.5 seconds or less.

7. The aforementioned processing unit, for each frame, Perform binarization, From the frame after the binarization process, a rectangular region having a size and shape corresponding to each code is extracted for each code. Based on the positional changes between frames in the aforementioned rectangular region, the amount of change in the position of each code is determined. The information display system according to claim 5.

8. The information display system according to claim 7, wherein the processing device performs a resolution reduction process on each frame and then performs the binarization process.

9. The information display system according to claim 7, wherein the processing device performs a process to reduce the resolution of each frame to 1 / 2 or less and 1 / 5 or more, and then performs the binarization process.

10. The code records an identifier for determining a virtual label containing the discount information, The information display system further includes a storage device that stores data defining the correspondence between the identifier, one or more timings at which a discount is applied, and the discount amount or discount rate at each timing. The processing device refers to the data and determines a virtual label including discount information corresponding to the identifier. The information display system according to claim 1.

11. The information display system according to claim 10, wherein the processing device determines the virtual label including corresponding discount information based on the data and the current time.

12. The information display system according to claim 11, wherein the processing device edits the correspondence relationship based on input from the user.

13. The information display system according to any one of claims 1 to 12, wherein the code is a two-dimensional code.

14. A processing device used in an information display system that photographs multiple displayed products, reads a code attached to each product, and overlays a virtual label containing discount information for each product onto the camera image of the multiple products based on the information recorded in the code, The camera recognizes and reads the code attached to each product from the camera images of the multiple products captured by the camera, and displays a virtual label containing discount information for each product overlaid on the camera image on the display based on the information recorded in the code. By comparing multiple frames of the camera image, the amount of change in the position of each of the multiple codes in the camera image is determined. For each code, If the amount of change in the position exceeds the threshold, the code is reread and the display of the virtual label is updated. If the amount of change in position does not exceed the threshold, the display content of the virtual label is maintained without rereading the code. Regardless of whether the amount of change in position exceeds the threshold, the display of discount information on the virtual label corresponding to the code is updated at a predetermined timing based on the information recorded in the code. Processing device.

15. A method performed by a computer that photographs multiple displayed products, reads the code attached to each product, and overlays a virtual label containing discount information for each product onto the camera images of the multiple products based on the information recorded in the code, The system recognizes and reads the code attached to each product from the camera images of the multiple products captured by the camera, and displays a virtual label containing discount information for each product overlaid on the camera image on a display based on the information recorded in the code. By comparing multiple frames of the aforementioned camera image, the amount of change in the position of each of the multiple codes in the camera image is determined, For each code, If the amount of change in the position exceeds the threshold, the code is reread and the display of the virtual label is updated. If the amount of change in position does not exceed the threshold, the display content of the virtual label is maintained without rereading the code. Regardless of whether the amount of change in position exceeds the threshold, the display of discount information on the virtual label corresponding to the code is updated at a predetermined timing based on the information recorded in the code. A method that includes this.

Images