Processing device, processing method, and program

Abstract

To recognize a status of a sell-by date of each commodity on display.SOLUTION: A processing apparatus includes: an acquisition unit that acquires a plurality of time-series images including a display area; a detection unit that detects an object region including a commodity from each of the time-series images; a determination unit that determines identity of object regions detected from different images; a management unit that manages a display time of the detected object region, based on a result of the detection and a result of the identity determination; and an output unit that outputs information related to the display time.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a processing device, a processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses a technology that recognizes ingredients in each of a time-series group of images taken inside a refrigerator, determines whether the ingredients have been moved based on the recognition results, and determines that ingredients that have not been moved for a period of time exceeding a threshold have been left unattended and issues a warning. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-57022 A Summary of the Invention [Problem to be solved by the invention]

[0004] Some stores set sell-by dates for products in order to avoid accidents. In such stores, a means is required for store clerks to know which products have passed their sell-by date or are close to their sell-by date. For example, when each product is given a label with various information such as a best-before date or expiration date printed on it, a means for checking the labels of each product on display is available. However, for products such as prepared foods and fresh foods that are not individually packaged, such as croquettes and yakitori, it may not be possible to give labels. In addition, the task of checking the labels of each product on display is very time-consuming. Patent Document 1 does not disclose the problems and solutions faced by such stores.

[0005] An object of the present invention is to provide a technique for grasping the status of the sell-by date of each product on display. [Means for solving the problem]

[0006] According to the present invention, An acquisition means for acquiring a plurality of time-series images including a display area; A product detection means for detecting a product from each of the plurality of time-series images; An identity determination means for determining identity of products detected from the different images; a display time management means for managing the display time of each of the detected products based on the detection result and the identity determination result; an output means for outputting information regarding the display time of each of the detected products; A processing apparatus is provided having:

[0007] Further, according to the present invention, The computer Acquire a plurality of time-series images including the display area; Detecting products from each of the plurality of time-series images; determining the identity of the products detected from the different images; managing the display time of each of the detected products based on the detection result and the result of the determination of identity; A processing method is provided that outputs information regarding the display time of each detected product.

[0008] Further, according to the present invention, Computer, An acquisition means for acquiring a plurality of time-series images including the display area; a product detection means for detecting a product from each of the plurality of time-series images; an identity determination means for determining the identity of products detected from the images that are different from each other; a display time management means for managing the display time of each of the detected products based on the detection result and the identity determination result; an output means for outputting information regarding the display time of each of the detected products; A program is provided to function as a Effect of the Invention

[0009] According to the present invention, a technique for grasping the status of the sell-by date of each product on display is realized. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of a processing device according to the present embodiment. [Diagram 2] FIG. 2 is a functional block diagram of a processing apparatus according to an embodiment of the present invention; [Diagram 3] FIG. 4 is a diagram illustrating an example of information processed by the processing device of the present embodiment. [Figure 4] 4 is a flowchart showing an example of a processing flow of the processing device of the present embodiment. [Diagram 5] FIG. 4 is a diagram showing an example of an image output by the processing device of the present embodiment. [Figure 6] FIG. 4 is a diagram illustrating an example of information processed by the processing device of the present embodiment. [Figure 7] FIG. 4 is a diagram showing an example of an image output by the processing device of the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] "Assumptions" In a store that uses the processing device of this embodiment, the sales deadline for at least some of the products is determined based on the display time (the time the product is displayed). Specifically, the sales deadline is the time when the display time exceeds the upper limit.

[0012] At least some of the products whose sell-by dates are managed based on the display time may be products to which labels cannot be attached, such as croquettes and yakitori, or may be products to which labels can be attached. The products in question may be, for example, food products, but are not limited to this, and may be other products such as home appliances and stationery.

[0013] Merchandise that has passed its sell-by date or is nearing its sell-by date is removed from the shelves or sold at a discount.

[0014] "overview" The processing device of this embodiment detects products from multiple time-series images including a product display area, determines the identity of the products detected from the different images, and manages the display time of each product displayed in the display area based on the detection result and the identity determination result.The processing device then outputs information regarding the display time of each product displayed.

[0015] Based on the information output by the processing device, the store clerk can grasp the sell-by date status of each product on display, i.e., whether the sell-by date has passed or whether the sell-by date is approaching.

[0016] "Hardware Configuration" Next, an example of the hardware configuration of the processing device will be described. Each functional part of the processing device is realized by any combination of hardware and software, centering on a CPU (Central Processing Unit) of any computer, memory, programs loaded into the memory, a storage unit such as a hard disk for storing the programs (programs stored beforehand at the stage of shipping the device, as well as programs downloaded from storage media such as CDs (Compact Discs) or servers on the Internet, can be stored), and a network connection interface. Those skilled in the art will understand that there are various variations in the method of realizing the above and the device.

[0017] FIG. 1 is a block diagram illustrating a hardware configuration of a processing device. As shown in FIG. 1, the processing device has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The processing device does not have to have the peripheral circuit 4A. The processing device may be composed of multiple devices that are physically and / or logically separated, or may be composed of a single device that is physically and / or logically integrated. When the processing device is composed of multiple devices that are physically and / or logically separated, each of the multiple devices can have the above hardware configuration.

[0018] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to transmit and receive data to each other. The processor 1A is an arithmetic processing device such as a CPU or a GPU (Graphics Processing Unit). The memory 2A is a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The input / output interface 3A includes an interface for acquiring information from an input device, an external device, an external server, an external sensor, a camera, etc., and an interface for outputting information to an output device, an external device, an external server, etc. Examples of the input device include a keyboard, a mouse, a microphone, a physical button, a touch panel, etc. Examples of the output device include a display, a speaker, a printer, a mailer, etc. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0019] "Function Configuration" 2 shows an example of a functional block diagram of the processing device 10. As shown in the figure, the processing device 10 has an acquisition unit 11, a product detection unit 12, an identity determination unit 13, a display time management unit 14, and an output unit 15.

[0020] The acquisition unit 11 acquires a plurality of time-series images including a display area. The display area is an area where products are displayed, and is realized by, for example, a product display shelf. In a store that uses the processing device 10, a camera is installed at a position and orientation that captures the products displayed in the display area (preferably, all the products displayed in the display area).

[0021] The camera may capture moving images all the time (for example, during business hours) or may continuously capture still images at a time interval longer than the frame interval of the moving images. The acquisition unit 11 acquires images generated by such a camera.

[0022] The commodity detection unit 12 detects commodities from each of a plurality of time-series images. Then, as a detection result, the commodity detection unit 12 outputs, for each detected commodity, position information indicating the position in the image (hereinafter, "commodity position information") and commodity identification information (commodity code, commodity name, etc.).

[0023] Product detection can be realized by using any known product recognition technology. For example, the product detection unit 12 detects an object region (a region where an object exists) in an image by using a known object detection technology. The product detection unit 12 may detect a region including an object and its periphery (e.g., a rectangular region) as the object region. Alternatively, the product detection unit 12 may detect a region having a shape following the contour of an object where only the object exists as the object region. The latter can be realized by using, for example, a method for detecting a pixel region where a detection target exists, called semantic segmentation or instance segmentation.

[0024] Then, the commodity detection unit 12 recognizes the commodity included in the image of the object region by using a well-known commodity recognition technology (feature point matching, deep learning, etc.).

[0025] The product position information is indicated, for example, by coordinates of a two-dimensional coordinate system set on the image. The product position information may be indicated by a point in the image, or may be indicated by an area occupying a part of the image. For example, the product position information may be information indicating a representative point (center, center of gravity, etc.) of the object area, or may be information indicating the object area (a part of the image).

[0026] The identity determination unit 13 determines the identity of products detected from different images. That is, the identity determination unit 13 determines whether the product detected from the first image and the product detected from the second image are identical. Here, "identical" means that they are the same individual. In other words, even if the product identification information matches, if they are different individuals, they do not fall under the definition of "identical" here. An example of the process of determining identity will be described below.

[0027] <Judgment Example 1> The identity determination unit 13 can determine the identity of products based on the position of the product in the image detected from each of the different images. If the positions of the product in the image detected from each of the multiple images are the same or the difference is within a threshold, the products detected from each of the multiple images can be determined to be the same product. In this example, two images whose image generation timing (shooting timing) difference is equal to or less than a reference time become the processing targets for identity determination (the above-mentioned first image and second image). The reference time is a design factor, but is a relatively small value.

[0028] When the product position information is indicated by a single point in the image, the identity determination unit 13 may set a condition for determining that the products are the same if "the distance between the points is equal to or less than a threshold value." In addition, when the product position information is occupying a partial area in the image, the identity determination unit 13 may set a condition for determining that the products are the same if "the overlapping areas occupy a reference ratio or more of the area indicated by each product position information." The conditions for determining that the products are the same given here are merely examples, and are not limited thereto.

[0029] The identity determination unit 13 may determine that the products are the same if the above conditions are satisfied and the product identification information is identical.

[0030] <Judgment Example 2> The identity determination unit 13 can determine the identity of products based on the feature quantities (shape, size, color, etc.) of the appearance of the products detected from each of the different images. Even if the products are of the same type (the product identification information matches), individual differences may appear in the appearance of the products. The identity determination unit 13 can determine the identity of products based on such individual differences in appearance.

[0031] The display time management unit 14 manages the display time of each of the products detected by the product detection unit 12 based on the detection result by the product detection unit 12 and the identity determination result by the identity determination unit 13.

[0032] For example, the display time management unit 14 can start managing the display time of a product detected from an image that is not determined to be the same as any of the products detected from previous images, as a product newly displayed in the display area. Also, the display time management unit 14 can end management of the display time of a product detected from an image that is not determined to be the same as any of the products detected from subsequent images.

[0033] Here, a specific example of processing by the display time management unit 14 will be described with reference to Figs. 3 and 4. First, the display time management unit 14 manages display time management information as shown in Fig. 3. The display time management information shown in the figure links serial numbers, product identification information, product position information, and display start time with each other. The serial numbers are information for identifying products managed by the display time management information. The product identification information is product identification information of products managed by the display time management information. The product position information is the latest product position information of each product managed by the display time management information. The display start time is the time when the display of each product managed by the display time management information started. Note that instead of the display start time, a display start date and time or a display start date may be used.

[0034] As shown in Figure 4, when the product detection unit 12 detects products from the first image (S10), the identity judgment unit 13 judges the identity of each of the products detected from the first image with the products managed by the display time management information based on the product position information of each of the detected products and the product position information of each of the products managed by the display time management information as shown in Figure 3 (S11).

[0035] Then, if there is a product detected from the first image that is not determined to be the same as any product managed in the display time management information (Yes in S12), the display time management unit 14 starts managing the display time of that product (S13). Specifically, the display time management unit 14 registers the product identification information and product position information of that product by linking it to a new serial number, and registers the generation timing (photographing timing) of the first image or a timing nearby the generation timing as the display start time.

[0036] It should be noted that, if there are no products managed by the display time management information at the time of judgment in S11, the display time management unit 14 determines that there is a product among the products detected from the first image that is not determined to be identical to any of the products managed by the display time management information (Yes in S12), and begins managing the display times of all products detected in the first image (S13).

[0037] On the other hand, if there is no product detected from the first image that is not determined to be identical to any of the products managed in the display time management information (No in S12), the display time management unit 14 does not execute the processing of S13.

[0038] Furthermore, if there is a product that is not determined to be the same as any of the products detected from the first image among the products managed by the display time management information (Yes in S14), the display time management unit 14 ends the management of the display time of that product (S15). For example, the display time management unit 14 may delete the information of that product from the display time management information. Additionally, the display time management unit 14 may register information indicating that the management of the display time has ended, linked to that product.

[0039] The above-mentioned "information indicating that management of display time has ended" may be time information or date and time information indicating the generation timing (photographing timing) of the first image or a timing close thereto. By registering such information, the display start time and display end time of each of a plurality of products can be recorded. This information can then be used for marketing, etc.

[0040] On the other hand, if there is no product among the products managed by the display time management information that is not determined to be identical to any of the products detected from the first image (No in S14), the display time management unit 14 does not execute the processing of S15.

[0041] Furthermore, if there is a product among the products managed by the display time management information that is determined to be identical to any of the products detected from the first image (Yes in S16), the display time management unit 14 updates the product position information of that product managed by the display time management information to the product position information of the product detected from the first image (S17).

[0042] On the other hand, if there is no product among the products managed by the display time management information that is determined to be identical to any of the products detected from the first image (No in S16), the display time management unit 14 does not execute the processing of S17.

[0043] The order of determinations in S12, S14, and S16 is not limited to the example shown in the figure.

[0044] Returning to Fig. 2, the output unit 15 outputs information relating to the display time of each product detected by the product detection unit 12. For example, the output unit 15 outputs a processed image on the image acquired by the acquisition unit 11, in which information relating to each product's display time is displayed, linked to each product detected from the image. The output unit 15 may output the information via an output device such as a display, projection device, printer, or mailer connected to the processing device 10, or may transmit the information to another external device. An example of information relating to display time will be described below.

[0045] <Example 1 of information about display hours> The output "information regarding display time" may be "information indicating an approximate display time." For example, the display time is divided into a number of groups, such as "less than M1," "M1 or more and less than M2," "M2 or more," etc. The number of groups may be any number equal to or greater than two. The "information indicating an approximate display time" is information indicating which group the current display time of each product belongs to.

[0046] An example of a processed image displaying the information is shown in Fig. 5. In the illustrated example, a frame F is displayed in association with each product P detected by the product detection unit 12. The display position of the frame F is determined, for example, based on the product position information of each product P. The display mode (line color, line shading, line type, etc.) of the frame F of each product P changes depending on which group the display time of each product P belongs to. The display mode of the frame F according to the display time as shown in Fig. 6 is determined in advance, and the frame F of each product P is displayed in the corresponding display mode.

[0047] 6 may be different for each product P. That is, the values ​​of M1 and M2 of the frame display criteria may be different for each product P. In addition, although the example shown in FIG. 6 is divided into three groups, the number of groups may also be different for each product P.

[0048] Furthermore, the output unit 15 may highlight the product P whose display time exceeds the upper limit. There are various methods for highlighting, but for example, the frame F of the relevant product P may be blinked or displayed in a predetermined color (red). The upper limit may be different for each product P.

[0049] <Example 2 of information about display hours> The output "information regarding display time" may be "information indicating an estimate of the remaining time until the sales deadline." The remaining time until the sales deadline (hereinafter sometimes simply referred to as "remaining time") is calculated by subtracting the display time from the upper limit value mentioned above. For example, the remaining time is divided into multiple groups such as "less than L1," "L1 or more and less than L2," "L2 or more," etc. The number of groups may be any number equal to or greater than two. The "information indicating an estimate of the remaining time" is information indicating which group the remaining time of each product belongs to.

[0050] An example of a processed image displaying the information is shown in Fig. 5. In the illustrated example, a frame F is displayed in association with each product P detected by the product detection unit 12. The display position of the frame F is determined, for example, based on the product position information of each product P. The display mode (line color, line shading, line type, etc.) of the frame F of each product P changes depending on which group the remaining time of each product P belongs to. The display mode of the frame F according to the remaining time is determined in advance, and the frame F of each product P is displayed in the corresponding display mode.

[0051] The frame display criteria may be different for each product P. The number of groups may also be different for each product P. The output unit 15 may highlight a product P whose remaining time falls below a lower limit. There are various methods for highlighting, and for example, the frame F of the corresponding product P may be made to blink or displayed in a predetermined color (red).

[0052] <Example 3 of information about display hours> The output "information regarding display time" may be the "display time" or the "remaining time until the sales deadline" itself. Figure 7 shows an example of a processed image displaying this information. The number displayed in association with frame F indicates the "display time" or the "remaining time until the sales deadline." The "display time" or the "remaining time until the sales deadline" is shown in any unit such as "month," "day," "hour," "minute," or "second." For example, a product associated with "1" indicates that the display time or remaining time is "1 month," "1 day," "1 hour," "1 minute," or "1 second."

[0053] In the above examples 1 to 3, the frame F may be changed to another mark.

[0054] <Example 4 of information about display hours> In this example, the display time management unit 14 manages the display time of each product in a plurality of display areas for each display area. The output unit 15 outputs display area identification information of a product whose display time exceeds a reporting reference value as information related to the display time. The display area identification information is information for identifying the plurality of display areas from each other. The reporting reference value may be different for each product.

[0055] "Action and effect" As described above, according to the processing device 10 of this embodiment, it is possible to manage the display time of products displayed in a display area based on a plurality of time-series images including the display area. The processing device 10 can output information related to the display time of each displayed product. Based on the information output by the processing device 10, the store clerk can grasp the status of the sell-by date of each displayed product, i.e., "whether the sell-by date has passed" or "whether the sell-by date is approaching" and the like.

[0056] Furthermore, the processing device 10 can output a processed image in which information indicating the display time is displayed in association with each of the products detected from the image including the display area, as shown in Fig. 5 and Fig. 7. With such a processed image, the store clerk can intuitively grasp the sales status of each of the displayed products.

[0057] In this specification, "acquisition" includes at least one of the following: "the device itself goes to retrieve data stored in another device or storage medium (active acquisition)" based on user input or based on program instructions, such as receiving data by making a request or inquiry to another device, or accessing another device or storage medium and reading it, and "the device itself inputs data output from another device (passive acquisition)" based on user input or based on program instructions, such as receiving data that is distributed (or transmitted, push notification, etc.), and selecting and acquiring data from among the received data or information, and "editing data (converting it to text, rearranging data, extracting some data, changing the file format, etc.) to generate new data and acquiring the new data."

[0058] Although the present invention has been described above with reference to the embodiments (and examples), the present invention is not limited to the above-mentioned embodiments (and examples). Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0059] A part or all of the above-described embodiments can be described as follows, but is not limited to the following. 1. An acquisition means for acquiring a plurality of time-series images including a display area; A product detection means for detecting a product from each of the plurality of time-series images; An identity determination means for determining identity of products detected from the different images; a display time management means for managing the display time of each of the detected products based on the detection result and the identity determination result; an output means for outputting information regarding the display time of each of the detected products; A processing device having 2. The processing device described in 1, wherein the output means outputs a processed image on which information indicating the display time is displayed and linked to each product detected from the image. 3. The processing device according to claim 2, wherein the output means outputs the processed image in which information corresponding to the length of time the product has been on display is displayed, linked to each product detected from the image. 4. The processing device according to claim 2, wherein the output means outputs the processed image that highlights products whose display time has exceeded the upper limit. 5. The processing device according to claim 4, wherein the upper limit value differs for each product. 6. A processing device described in any one of 1 to 5, wherein the identity determination means determines the identity of products based on the positions of the products within the images detected from each of the images which are different from each other. 7. A processing device according to any one of 1 to 6, wherein the display time management means starts managing the display time of any product detected from the image that is not determined to be identical to any product detected from the previous image as a new product displayed in the display area. 8. A processing device described in any one of 1 to 7, wherein the display time management means terminates management of the display time of a product detected from the image that is not determined to be identical to any of the products detected from subsequent images. 9. The computer: Acquire a plurality of time-series images including the display area; Detecting products from each of the plurality of time-series images; determining the identity of the products detected from the different images; managing the display time of each of the detected products based on the detection result and the result of the determination of identity; A processing method that outputs information regarding the display time of each detected product. 10. Computer An acquisition means for acquiring a plurality of time-series images including the display area; a product detection means for detecting a product from each of the plurality of time-series images; an identity determination means for determining the identity of products detected from the images that are different from each other; a display time management means for managing the display time of each of the detected products based on the detection result and the identity determination result; an output means for outputting information regarding the display time of each of the detected products; A program that functions as a

Claims

1. An acquisition means for acquiring a plurality of time-series images including a display area; A detection means for detecting an area in an image where a product exists from each of the plurality of time-series images; A determination means for determining the identity of a product present in the image area detected from each of the different images; a management means for managing the display time of a product present in the detected image area based on the result of the detection of the image area and the result of the determination of the identity; an output means for outputting information regarding the display time; A processing device having

2. The processing device according to claim 1 , wherein the output means outputs a processed image on which information indicating the display time is displayed and linked to the area within the image detected from the image.

3. The processing device according to claim 2 , wherein the output means outputs the processed image on which information corresponding to the length of time the item has been on display is displayed, in association with the area within the image detected from the image.

4. The processing device according to claim 2 , wherein the output means outputs the processed image that highlights an area in the image in which a commodity that has been on display for a period exceeding the upper limit is present.

5. The processing device according to claim 4 , wherein the upper limit value is different for each of the regions within the image.

6. The processing device according to claim 1 , wherein the determining means determines the identity of a product present in the intra-image area based on a position within the image of the intra-image area detected from each of the mutually different images.

7. The processing device according to any one of claims 1 to 6, wherein the management means starts managing the display time of a product that is detected in the image area from the image and is not determined to be identical to any of the products detected in the image area from a previous image as a new product.

8. A processing device as described in any one of claims 1 to 7, wherein the management means terminates management of the display time of a product that is present in the image area detected from the image and is not determined to be identical to any of the products present in the image area detected from any subsequent image.

9. The computer Acquire a plurality of time-series images including the display area; Detecting an area in an image where a product exists from each of the plurality of time-series images; determining the identity of the products present in the image regions detected from the different images; managing the display time of the product present in the detected area of ​​the image based on the detection result of the area of ​​the image and the determination result of the identity; A processing method for outputting information regarding the display time.

10. Computer, An acquisition means for acquiring a plurality of time-series images including the display area; A detection means for detecting an area in an image where a product exists from each of the plurality of time-series images; A determination means for determining the identity of a product present in the image area detected from each of the different images; a management means for managing the display time of a commodity present in the detected area of ​​the image based on the result of the detection of the area of ​​the image and the result of the determination of the identity; an output means for outputting information regarding the display time; A program that functions as a

Images