Target commodity authorization program and target commodity authorization device

Abstract

To provide a target commodity authorization program and a target commodity authorization device which analyze commodity data of display shelves for usage by a company manufacturing products for consumers.SOLUTION: A target commodity authorization program is a program for allowing a computer including a storage medium and an arithmetic processing unit to operate each of the following steps. The steps include: an image data storage step of storing image data of shop shelves obtained by each photographing means in the storage medium; a shelf contents grasping step of grasping shelf steps of shelves, and commodities arranged on the shelf steps; and a ratio calculation step of specifying a previously set target commodity among the grasped commodities, grasping the number of commodities of the target commodity, and calculating a ratio of the target commodity among all commodities on the shelves for a previously set commodity category or each of sub-categories being subordinate categories of the category.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a target product certification program and a target product certification device, and more particularly to a target product certification program and a target product certification device that analyzes product data on display shelves for use by companies that manufacture consumer products. [Background technology]

[0002] In the retail industry, product shelf display determines sales, so various methods have been proposed and implemented to efficiently manage the number of products on display and manage out-of-stock items. Such methods are important not only for retailers, but also for companies that manufacture products for consumers. This is because a manufacturer's product sales at retail locations are determined by the percentage of their own products they sell, known as "in-store share." Currently, the following devices have been proposed as devices that use image processing to check the display state of a display shelf and make some kind of suggestion. Patent Document 1 proposes a display situation analysis system, a display situation analysis method, and a program capable of analyzing the display situation of items. Specifically, the proposed display situation analysis system is a display situation analysis device that includes an item recognition means that recognizes items in a display image taken of items displayed on a display shelf, an analysis means that detects missing items based on the recognition results of the item recognition means, and an output means that outputs a display image on which an image that enables the detected missing items to be recognized is superimposed. Patent Document 2 proposes a product display information compilation system that compiles the display status of products in a store. Specifically, the compilation system has an intermediate format information generation processing unit that generates product identification information, shelf position, and face number information of products displayed on display fixtures based on display fixture image information obtained by photographing the display status of products in a store, and an interface generation processing unit that generates a screen that displays the face number of the product according to the shelf position based on the generated product identification information, shelf position, and face number information. Patent Document 3 proposes a tallying system that correlates the relationship between product display status and sales amount. Specifically, the proposed product display information tallying method includes a step of extracting product images of products displayed on a product display shelf from image data of the products, a step of searching a product database for product data corresponding to the extracted product images, a step of determining the number of shelves on the product display shelf and generating shelf number information, and a recording step of recording the product codes, product image position data in the image data of the extracted product images, the shelf number information, and auxiliary information as product shelf information. Patent Document 4 proposes an information processing system, an occupancy rate calculation method, and a program capable of calculating an appropriate occupancy rate according to the products stored on a shelf. Specifically, the proposed shelf management system acquires size information of the shelf and size information of the products stored on the shelf, calculates the occupancy rate of the shelf by the stored products based on at least the size information of the shelf and the size information of the products, and calculates a corrected occupancy rate by correcting the occupancy rate using a correction coefficient according to the products stored. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-126679 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-71782 [Patent Document 3] Japanese Patent Application Laid-Open No. 2013-250647 [Patent Document 4] Japanese Patent Publication No. 2023-65008 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the proposals mentioned above were aimed at retail businesses, and were not designed to analyze data and provide various proposals for use by companies that manufacture consumer products. In short, there is a current demand for the development of a system that can analyze product data on shelves for use by companies that manufacture consumer products.

[0005] Therefore, an object of the present invention is to provide a target product certification program and a target product certification device for analyzing product data on display shelves, for use by companies that manufacture products for consumers. [Means for solving the problem]

[0006] As a result of intensive research into resolving the above-mentioned problems, the inventors discovered that by not simply identifying the products on the shelves but also focusing on the shelves themselves and identifying the type of product etc. through the shelves, it would be possible to contribute to understanding the display situation not only in retail stores but also in stores of companies that manufacture and sell products for consumers, thereby achieving the above-mentioned objective, and thus completed the present invention. That is, the present invention provides the following inventions. 1. A program that causes a computer having a storage medium and an arithmetic processing unit to perform the following steps: Each of the above steps is an image data storage step of storing image data of store shelves obtained by various photographing means in the storage medium; a shelf content recognition step of analyzing the image data to recognize the number of shelves on the shelf and the products placed on the shelves; A target product certification program comprising a proportion calculation step of identifying predetermined target products from the identified products, determining the number of the target products, and calculating the proportion of the target products among the total number of products on the shelf for each predetermined product category or subcategory that is a lower concept of the category. 2. The target product certification program according to 1, wherein the shelf content understanding step is composed of the following four processes (modules): Module 1: Recognizing the location of each product in shelf image data Module 2: Recognizing shelf position in shelf image data Module 3: Recognizing price tag locations in shelf image data Module 4: Recognizing missing parts in shelf image data 3. The target product certification program described in 1 further includes an output preparation step of organizing the number of shelves, the number of products, the number of target products, and the ratio so that they can be output for each category or subcategory. A computer unit storing the program described in 4.1; and an information terminal connected to the computer unit directly or via a network, On the display unit of the information terminal, The number of shelves, the number of products, the number of target products, and the ratio are configured to be displayed for each category or subcategory. Approved product equipment. [Effects of the Invention]

[0007] The target product certification program and target product certification device of the present invention can analyze product data on display shelves for use by companies that manufacture consumer products. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram showing a target product certification device of the present invention. [Figure 2] FIG. 2 is a schematic diagram showing a computer used in the target product certification device of the present invention. [Figure 3] FIG. 3 is an explanatory diagram showing a flow sheet of the target product certification program of the present invention. [Figure 4] FIG. 4 is a front view of a shelf for schematically explaining certification of a shelf in the target product certification program of the present invention. [Figure 5] FIG. 5 is an explanatory diagram for schematically explaining an algorithm for certifying a product in the target product certification program of the present invention. [Explanation of symbols]

[0009] 100 Target product certification device, 1 Computer, 13 Central processing unit (CPU), 11 Memory, 15 Storage medium, Information terminal 50, Display unit 51, 60 Camera DETAILED DESCRIPTION OF THE INVENTION

[0010] The present invention will now be described in further detail. <Device> As shown in Fig. 1, the target product certification device 100 of this embodiment comprises a computer 1 as a computer unit in which the program of the present invention described below is stored, and an information terminal 50 connected to the computer unit directly or via a network (in this embodiment, wirelessly via the Internet 101), and is configured to display the number of shelves, the number of products, and the number and proportion of target products for each product category or subcategory on a display unit 51 of the information terminal 50. Furthermore, as shown in Fig. 1, the target product certification device 1 of this embodiment also comprises a camera 60 as a photographing means for acquiring product shelf information as image data and transferring it to the computer 1 via the Internet or the like.

[0011] 〔computer〕 The computer 1 used in this embodiment is not particularly limited as long as it includes a storage medium and a processing unit. Specifically, as shown in FIG. 2, it includes a central processing unit (CPU) 13 as a processing unit, memory 11 as a temporary storage area, and a non-volatile storage medium 15 such as a hard disk or solid-state device. Any commercially available personal computer can be used without particular limitations. Mobile devices such as smartphones and tablet devices can also be used as computers, and these are also included in the "computer" of this invention. Furthermore, although not specifically shown, the computer in this embodiment preferably has a communication device and is capable of communication over a network. It can also be configured to connect to a server with a database located on the network through communication and obtain updated data from the database as needed. The computer is also equipped with input devices 20, such as an image input device such as a keyboard, mouse, or camera, a voice input device such as a microphone, or a communication input device using a communication device such as Bluetooth®, and is configured to input necessary data and information as needed. It is also equipped with output means 30, such as a display for displaying evaluation results or a printer for printing, to output the results in a desired format as needed. (Other components (devices)) The computer of this embodiment may include various devices other than the above-mentioned devices as necessary. [Information terminal] In this embodiment, the information terminal 50 may be, like the computer described above, a device that includes a central processing unit (CPU) as an arithmetic processing unit, memory as a temporary storage area, and a non-volatile storage medium such as a hard disk or solid state device, has a communication device, and is capable of communication via a network. In particular, tablet-type terminals are preferred in retail settings from the standpoint of usability, etc. 〔camera〕 In this embodiment, the camera 60 may be substituted by a camera function provided in an information terminal, or a separate photographing device may be provided. In either case, it is preferable that the camera used in this embodiment is provided with a communication function from the viewpoint of convenience in utilizing data. [Other devices] The target product certification device of this embodiment may include various devices other than the above-mentioned devices as necessary.

[0012] <Program> The program stored in the computer and causing the computer to execute the following steps includes a storage step S1, a shelf content identification step S2, a ratio calculation step S3, and an output preparation step S4. It also includes a pre-preparation step S0 and a post-processing step S5. The program of this embodiment also includes a machine learning program for executing the shelf content identification step S2 and the ratio calculation step S3 among the above steps. This machine learning program can be incorporated without any particular restrictions and can be any program capable of learning based on existing image data. Further explanation will be given below with reference to FIG.

[0013] [Preparation step S0] Before executing the image data storage step S1, image data of various products is stored in the database of the storage medium. Examples of various products include foods, beverages, alcoholic beverages, cosmetics, detergents, hair styling products, and other products sold on the shelves of various retail stores, as will be described later. In this embodiment, when storing image data of various products (hereinafter, this image data will be referred to as "preliminary image data"), the machine learning program is used to learn the product images and extract feature points (described later) of each product (product learning step). The resulting learning data is also stored in the database. Furthermore, for shelves that need to be identified separately, image data of the shelves (hereinafter, this image data will be referred to as "preliminary shelf data") is learned in advance and the learning data is stored. These preliminaries will be described later in the explanation of later steps, as this will facilitate a better understanding of the present invention. In addition, it is possible to register target products in advance as well as retail stores (store area, traffic flow, number of shelves), but usually only store information such as the store name and location is registered. In addition, it is also possible to register store information and detailed information about each store by linking with an external store management system.

[0014] [Image data storage step S1] In step S1, image data of store shelves obtained by various imaging means (cameras in this embodiment) is stored in storage medium 15. Examples of stores include convenience stores, drug stores, supermarkets, cosmetic stores, pharmacies, department stores, and other retail stores. The shelves may be the type typically found in the stores described above, with multiple shelves (see FIG. 4). In this step, as shown in FIG. 1, the user takes a photo of the shelf with the camera 60, transfers the obtained image data to a computer via the network, and stores the received image data in the storage medium 15.

[0015] [Shelf Content Grasping Step S2] This step S2 is a step of analyzing the image data to determine the number of shelves on the shelf and the products placed on those shelves. This step S2 is composed of the following four processes (modules). Module 1: Recognizing the location of each product in shelf image data Module 2: Recognizing shelf position in shelf image data Module 3: Recognizing price tag locations in shelf image data Module 4: Recognizing missing parts in shelf image data This is explained below. (Module 1) This module recognizes product positions and counts the number of all products in the shelf image data. Therefore, it identifies the coordinates of the products from the shelf image data (for example, p in Figure 4, and this is done for all products recognized as products in the image data). The inventors have noticed that no matter how diverse products are, humans can recognize them. This is because each product has its own unique characteristics (brand (product) name, shape, color, text printed on packaging, etc.) that remain constant regardless of the photographing conditions, and because different products from the same manufacturer often look very similar, but have many differences in color, printed text, illustrations, etc. Therefore, the reasons why humans can distinguish these patterns are incorporated into the machine learning program, and the following points are extracted as feature points and stored in the database. Specifically, in the advance preparation step, a plurality of advance image data taken under different lighting conditions and from different angles are prepared as advance image data and stored in a database. A machine learning program is trained on these advance image data for one product to identify the characteristics (color, shape, size, logo, text) of each product. Differences in the characteristics of each product between different products are then organized and identified, and this identification data and data on differences are stored in the database together with each product information (advance preparation step S0). Here, the product information includes the product name and manufacturer name of each product, as well as the category and subcategory to which the product belongs. A category is a classification that comprehensively describes a product, such as daily necessities or cosmetics, while a subcategory is an expression that belongs to a category and is used to further classify the category into multiple groups, such as oral hygiene products, soaps, sanitary paper products and tools, nursing care products and tools, childcare products and tools, sanitary medical products and tools, laundry detergents, kitchen and dish detergents, household detergents, basic cosmetics, makeup cosmetics, body care cosmetics, fragrances, in-bath hair care, hair makeup, hair coloring, and men's cosmetics. In addition, for multiple products that are particularly similar in appearance, a preparatory step was carried out to enable more accurate product identification by pairing similar products together, specifying that there are differences between them, and having the system learn the differences between them. When the image data is acquired, it is compared with the database, and data with matching features is extracted to identify the product. If there is no perfectly matching data, the product is identified based on the most similar product. This identification will be described in detail in the target product identification step S3. (Module 2) This module recognizes the shelf position (coordinates of the shelf board) using shelf image data and counts the number of shelves. Typically, shelves are made up of common colors and shapes (e.g., wood or rectangular). In many stores, price tags t are attached to shelf b, as shown in Figure 4. Therefore, in the pre-preparation step, a machine learning program is trained on not only the products but also the pre-preparation data. The feature points of the shelf are also stored in a database. This data is then compared with the acquired image data to determine the shelf location. Here, the pre-preparation step uses shelves with data labels (equivalent to price tags) attached to the product placement locations. The trained model predicts product placement using the color, shape, and features of surrounding objects around the data labels. In particular, to detect product locations, we use oriented object detection (ORD) technology instead of the traditional rectangular bounding box method, which minimizes errors and improves recognition performance, especially in special cases. For example, because shelf layouts are usually very long, if a traditional rectangular bounding box is used, the bounding box will also include the product, which could result in the product being identified when the shelf level (shelf board) should be identified, resulting in inaccurate identification of the shelf level's position.However, the inventors have discovered that by using a rotated bounding box, such problems can be avoided and only the shelf levels can be accurately learned, and this is the method they have adopted. Furthermore, if there are multiple products on the same shelf, and Module 1 above identifies the products, and this module is unable to recognize the shelf level, it will recognize that a single product is on display. "Unable to recognize shelf level" means that when multiple products are stacked, the distance between the stacked products (sections recognized as products) is extremely small (0 or a distance of 5mm or less, close to 0). Normally, the presence of shelf levels creates a distance of 2cm or more, so if the above extremely small distance is recognized, the program is configured to determine that a single product is stacked on two or more levels, that there is no shelf level between the stacked products, and that there is no impact on the product's occupancy rate. (Module 3) This module focuses on the fact that price tags are often placed on shelves, and recognizes the position of price tags in the shelf image data. This price tag typically includes the price, product type (category, subcategory), product name, and other information. Therefore, not only is the price tag position recognized, but also the parts without a price tag are determined to be shelves or products that do not require recognition. Specifically, after the shelf position is identified from the image data, the presence of a price tag on the shelf is confirmed. This is because the machine learning program has been trained in a preparation step to detect cases where a price tag is installed on a shelf, allowing it to recognize a price tag on the shelf. Then, the character information of the part recognized as a price tag is read in the usual way and stored as price tag information. Furthermore, parts identified as not having a price tag are determined to be areas that do not require recognition, even if some product is placed there. Module 3, described later, recognizes price tags and associates them with products. This module must distinguish between cases where products are placed on top of a shelf and cases where products are hanging below the shelf, and accurately associate the price tags with the products. This module addresses this issue by pre-training the "shelf position recognition" function. Specifically, as shown in Figure 4 (g), a product group is identified in which items recognized as products are lined up horizontally. Next, the module determines the position of this product group g relative to the line h on the shelf (in the case of a hanging type, this is the shelf for placing price tags, and the products are suspended from a hanging shaft (not shown)). If g is located above h (the distance between h and g is close to zero), it is determined to be a placed case. If g is located below h (in this case, the distance between h and g is short, but not nearly zero, and is consistent and shorter than the distance between h and g above h), it is determined to be a hanging case. (Module 4) This module recognizes whether there are any missing items in the shelf image data. When the shelf position and price tag are identified, if the presence of a product cannot be recognized at the location where the price tag is recognized, it recognizes that the product corresponding to that price tag is missing (recognizes the missing product coordinates) and counts the number of missing items (the number of product types that are missing).

[0016] [Ratio calculation step S3] This step involves identifying pre-set target products from among the identified products, determining the number of such target products, and calculating the proportion of such target products among the total number of products on the shelf for each pre-set product category or subcategory that is a lower-level concept of that category. This step uses modules 5 and 6 below. Module 5: Extract product features as an embedding vector. Module 6: Search and extract from the database the embedding vector that is most similar to the embedding vector extracted from the image data. (Module 5) This module is used to extract product features. The feature points of the products that were recognized as products in the shelf content recognition step S2 are extracted. In this embodiment, the feature points are extracted by vectorizing the shape, color, logo, text, etc., which can be the feature points of the product in the image, using an embedding vector, and then expanding them into coordinates. Specifically, all of the shape, color, logo, and text that can be the feature points of one product, for example, product A, from the image data are vectorized and plotted on coordinates. Then, in the next module, the product is identified by comparing it with data in a database of pre-learning data that has been constructed by machine learning of the pre-image data and plotting the vector data of each product on coordinates. (Module 6) This module searches for embedding vectors and finds the vector most similar to the target product to identify the product. Specifically, as shown in Figure 5, embedding vectors are interpreted as three-dimensional vectors, and are plotted on three-dimensional coordinates. For example, Figure 5 shows embedding vectors 201-207 plotted for each of seven products in the database. The reason multiple embedding vector values ​​form clusters is because multiple image data for the same product are registered with different lighting and angles, resulting in slight deviations in the values. However, for the same product, individual product islands are formed, as shown in Figure 5, and each island is recognized as the embedding vector for a single product. Then, module 5 plots the embedding vector 210 of one product in the image data extracted by module 5 on coordinates. Then, module 5 extracts the embedding vector 205 of the product in the database that is closest to this embedding vector 210, and identifies the product with this embedding vector 205 as the product for which the embedding vector 210 was identified, i.e., the product recognized as the product in the image data. Then, whether or not this product is a target product is simply checked against the target product list, and if it is determined to be a target product, it is determined that the target product is present on the product shelf and is counted. This operation is performed for all products recognized in the image data, and the number of target products, non-target products, and total products are counted. The position of the target products on the shelf is also determined. This can be determined by linking them to the image data and performing this operation for all products recognized as target products, determining the position of each product recognized in the image data, and whether it is a target product or not. Then, the ratio (occupancy rate) of the target products is calculated by dividing the number of target products by the total number of products. While all products should be registered in the database in advance, there is a possibility that unregistered products may exist. In such cases, the manufacturer is identified based on the registered product with the closest embedding vector. The method for determining whether an island is formed as the same product is to compare the vector values ​​of the target product with those of existing products registered in the database (using common methods such as cosine similarity). If the distance between these vector values ​​is smaller than a predefined threshold, the two products are recognized as the same product. This threshold is calculated based on the actual dataset and remains unchanged during execution to ensure accuracy, but is updated as data accumulates. If a product is determined to be unsimilar to any existing products, it is identified as an unregistered product. The above threshold value will be specifically described. In this embodiment, the threshold value is set by executing the following modules i) to ii). i) Based on the image data acquired in the product learning step in the advance preparation step S0, chunks of similar products are created by collecting 10 products with similar embedding vectors obtained in module 5. Therefore, a large number of chunks are generated. ii) For each chunk, the distance between each product (the difference between the embedding vector values ​​of any two products belonging to that chunk) is calculated from the variation in the embedding vector, and the average distance between each product in that chunk (since there are 45 possible product combinations in one chunk, the sum of the 45 distances is divided by 45) is calculated and set as the threshold. Then, by executing the following module iii), it is determined which of the products registered in the database the target product to be detected is. iii) Extract the product (hereafter referred to as "Product X") registered in the database that has the closest embedding vector value to the target product. Calculate the distance from this product X (the difference between the embedding vector values ​​of both) and compare it with the threshold value for the chunk to which Product X belongs. As a result of the comparison, if the distance between the target product and product X is smaller than the threshold, the target product and product X are determined to be the same product. On the other hand, if the distance is larger than the threshold, the target product and product X are determined to be different products (usually, in this case, the product is recognized as an unregistered product).

[0017] [Output preparation step S4] Step S4 is a step of organizing the number of shelves, the number of products, the number of target products, and the ratio so that they can be output for each category or subcategory. In this step, the data obtained in steps S1 to S3 above is reorganized into preset categories or subcategories, and the information is transferred from the server to the information terminal so that it can be displayed on the information terminal. These arrangements, transfers, and displays can be performed in accordance with conventional methods.

[0018] [Post-processing step S5] If necessary, post-processing steps such as display, various outputs, data storage, and recording of changes over time can be configured.

[0019] [Usage and effects] To use the target product certification device of the present invention, first, a photograph of the product display shelf is taken with a camera to obtain image data of the shelf. The obtained image data is then transferred to a computer (server). Based on the transferred image data, a target product certification program is executed to calculate the ratio (occupancy rate), etc. The calculated data is then transferred to an information terminal and displayed on the information terminal's display, allowing the user to confirm it. The target product certification device and target product certification program of the present invention, used in this manner, can analyze product data on shelves for use by companies manufacturing consumer products. This allows them to determine the display share of target products in a store's sales floor compared to rival products, thereby enabling them to understand sales trends in the store. This is expected to be useful for corporate marketing activities. Manufacturers and other companies can check the share of their products, and by checking over time, they can also confirm the best positions for sales and trends in sales. While conventional methods have not been able to accurately identify target products, the present invention identifies products in relation to shelves and provides multiple product features, allowing them to be accurately identified by identifying them. Furthermore, by incorporating machine learning and taking into account price tag information such as price tag position, accuracy can be further improved.

[0020] The present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the spirit of the present invention. For example, a module could be added to check whether the number of products is in line with the predetermined shelf layout (the arrangement of products displayed in a retail store), or a module could be added to look at the difference with market share, understand the status of the target product's development, and provide guidelines for action. It can be designed to grasp the changes, discontinuations, market share, and shelf location of new products, and it can also automatically grasp out-of-stock items in retail stores to improve the efficiency of ordering operations. Furthermore, it can grasp the optimal SKU for retail shelves and can be linked to POS systems.

Claims

1. A program that causes a computer having a storage medium and an arithmetic processing unit to perform the following steps: Each of the above steps is an image data storage step of storing image data of store shelves obtained by various photographing means in the storage medium; a shelf content recognition step of analyzing the image data to recognize the number of shelves on the shelf and the products placed on the shelves; A target product certification program comprising a proportion calculation step of identifying predetermined target products from the identified products, determining the number of the target products, and calculating the proportion of the target products among the total number of products on the shelf for each predetermined product category or subcategory that is a lower concept of the category.

2. 2. The target product certification program according to claim 1, wherein the shelf content grasping step comprises the following four steps (modules): Module 1: Recognizing the position of each product in shelf image data Module 2: Recognizing shelf positions in shelf image data Module 3: Recognizing price tag locations in shelf image data Module 4: Recognizing missing parts in shelf image data

3. 2. The target product certification program according to claim 1, further comprising an output preparation step of organizing the number of shelves, the number of products, the number of target products, and the ratio so that they can be output for each category or subcategory.

4. a computer unit storing the program according to claim 1; and an information terminal connected to the computer unit directly or via a network, On the display unit of the information terminal, The number of shelves, the number of products, the number of target products, and the ratio are configured to be displayed for each category or subcategory. Approved product equipment.

Images