Information Processing System

Abstract

To provide an information processing system for executing processing related to the following effect by detecting a POP attached to a display shelf installed in a store and merchandise displayed on the display shelf.SOLUTION: An information processing system estimates the effect of a POP. The information processing system includes a POP detection processing part for detecting a POP from image data capturing a display shelf, and an effect estimation processing part for estimating the effect of the POP, and the effect estimation processing part estimates the effect of the POP by using sales information of merchandise corresponding to the POP.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system that detects POPs attached to display shelves installed in stores and to products displayed thereon, and executes processing related to the effects of such detection. [Background technology]

[0002] In various stores such as convenience stores and supermarkets, it is common to sell products by placing them on display shelves. Therefore, by displaying multiple products on a display shelf, even if one product is purchased, another person can purchase the same product. Therefore, managing where and how many products are displayed on the display shelf is important in terms of product sales strategy.

[0003] In addition, POPs may be attached to display shelves or products to promote or advertise the products on display. POPs are promotional materials such as advertisements, testers, and samples used to sell the products on display.

[0004] POPs are important items for product manufacturers and retailers to sell products. For this reason, a system for printing POPs according to the sales status of a product has been disclosed, for example, in Patent Document 1 listed below. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2022-20094 A [Patent Document 2] Patent No. 6877806 Summary of the Invention [Problem to be solved by the invention]

[0006] In the invention of Patent Document 1, since POP can be printed according to the sales situation, for example, it is possible to print POP for advertising a product whose sales situation is unfavorable.

[0007] On the other hand, since POPs are important items for selling products, it is desirable to understand the effects of POPs, such as what effect they have, which products they should be attached to and where they should be attached to be effective, but the above-mentioned Patent Document 1 does not allow such understanding.

[0008] Therefore, Patent Document 2 discloses a system for identifying POP advertisements that have an effect on store sales.

[0009] However, although the patented invention of Patent Document 2 describes how to identify whether image data contains a POP advertisement, it does not describe how to identify the POP advertisement.

[0010] It is not easy to identify products displayed from image data of display shelves, and because POPs are required to be conspicuous, various shapes and designs are adopted. Therefore, it is not easy to stably determine POPs from images, and the above-mentioned Patent Document 2 does not state at all how to identify them, and the system is not feasible.

[0011] In order to understand the effectiveness of POP, it is necessary to automatically and stably detect POP from image data of display shelves, but there was no such system in the past. Also, it was not possible to estimate the effectiveness of POP detected from image data. [Means for solving the problem]

[0012] In view of the above problems, the present inventors have invented an information processing system that detects POPs from image data obtained by photographing a display shelf and executes processing related to the effects of the POPs.

[0024] No.1 The invention is an information processing system that displays the effects of POPs, the information processing system having a POP detection processing unit that detects POPs from image data showing a display shelf, a linking processing unit that links the POP with the product to which the POP corresponds, and an effect display processing unit that displays the effect of the POP, wherein the POP detection processing unit determines the type of object detected from the image data using the area of ​​the detected object and the type of the detected object to determine a POP or a POP candidate, and the effect display processing unit identifies a POP to be processed and calculates the effect of the POP by performing a calculation using an index value based on the type of the identified POP.

[0025] With the configuration of the present invention, it is possible to link the POP with the product and then calculate the effect of the POP.

[0026] In the above-mentioned invention, the effect display processing unit specifies a POP corresponding to the product selected, and displays the specified POP Calculation using index values ​​based on the type of By doing so, it can be configured like an information processing system, where data is classified into predetermined ranks.

[0027] In the above-mentioned invention, the effect display processing unit displays a part or all of the area of ​​the product or the POP corresponding to the product in a display method according to the rank. , Display method according to rank The present invention can be configured as an information processing system that performs display output using the above-mentioned method.

[0028] By executing the processes of these inventions, it is possible to display and output the effects of the POP.

[0029] In the above-mentioned invention, the effect display processing unit is , customer testimonials, awareness or salesThe present invention can be configured as an information processing system that divides one or more items into the ranks and displays and outputs the product or a part or all of the area of ​​the product corresponding to the product in a manner that corresponds to the rank, in a switchable manner for each of the items.

[0030] In the above-mentioned invention, the effect display processing unit is , customer testimonials, awareness or sales The present invention can be configured as an information processing system that divides one or more items into the ranks and displays and outputs two or more items simultaneously in part or all of the area of ​​the product or its corresponding POP in a display method corresponding to the rank.

[0031] By executing the processing of these inventions, the effects of POP and other items can be displayed in a switchable or simultaneous manner.

[0033] No. 1 The information processing system of the present invention can be realized by loading and executing the program of the present invention into a computer. That is, the information processing program causes a computer to function as a POP detection processing unit that detects POPs from image data showing a display shelf, a linking processing unit that links the POPs with the products corresponding to the POPs, and an effect display processing unit that displays the effects of the POPs, in which the POP detection processing unit determines the type of an object detected from the image data using the area of ​​the detected object and the type of the detected object to determine a POP or a POP candidate, and the effect display processing unit identifies a POP corresponding to a product whose selection has been accepted, and calculates the effect of the POP by performing a calculation using an index value based on the type of the identified POP. Effect of the Invention

[0034] By using the information processing system of the present invention, it is possible to detect POPs from image data obtained by photographing a display shelf and execute processing related to the effects of the POP. [Brief description of the drawings]

[0035] [Figure 1] 1 is a block diagram illustrating an example of a configuration of an information processing system according to the present invention. [Diagram 2] 2 is a block diagram illustrating an example of a configuration of a POP detection processing unit in the information processing system of the present invention. FIG. [Diagram 3] 4 is a block diagram illustrating an example of a configuration of an effect estimation processing unit in the information processing system of the present invention. FIG. [Figure 4] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer used in the information processing system of the present invention. [Diagram 5] 4 is a flowchart showing an example of the entire process in the information processing system of the present invention. [Figure 6] 10 is a flowchart showing an example of a POP detection process in the information processing system of the present invention. [Figure 7] 11 is a flowchart showing an example of an effect estimation process in the information processing system of the present invention. [Figure 8] FIG. 2 is a diagram showing an example of photographed image data. [Figure 9] FIG. 13 is a diagram showing an example of a screen for accepting POP and product selection. [Figure 10] FIG. 13 is a diagram showing an example of the relationship when there are multiple POPs. [Figure 11] FIG. 13 is a diagram showing an example of a display of the contribution of POP for each product. [Figure 12] FIG. 11 is a diagram showing an example of a result of executing an object detection process from image data. [Figure 13] 13A and 13B are diagrams illustrating an example of a process for determining a POP area by determining whether a predetermined condition is satisfied for an object area detected from image data. [Figure 14]This figure shows an example of each area after the areas of objects detected as products, product tags, or shelf components in the object detection results of Figure 12 are determined to be POP areas if they satisfy certain conditions. [Figure 15] FIG. 11 is a block diagram illustrating an example of a configuration of a POP detection processing unit in a second embodiment. [Figure 16] 13 is a flowchart illustrating an example of a POP detection process according to the second embodiment. [Figure 17] FIG. 11 is a diagram illustrating an example of an outline of a process in a second embodiment. [Figure 18] FIG. 11 is a diagram illustrating an example of processing by an individual discrimination processing unit in the second embodiment. [Figure 19] 13 is a diagram showing a schematic diagram of a state in which input of designation of a product display area, a product tag area, and a shelf top area has been received for image data. FIG. [Figure 20] 13 is a flowchart illustrating an example of a POP detection process according to the third embodiment. [Figure 21] FIG. 13 is a block diagram illustrating an example of a configuration of an information processing system according to a fourth embodiment. [Figure 22] FIG. 13 is a block diagram illustrating an example of the configuration of a linking processing unit according to a fourth embodiment. [Figure 23] FIG. 13 is a block diagram illustrating an example of a configuration of an effect display processing unit in a fourth embodiment. [Figure 24] 13 is a flowchart illustrating an example of an entire process in an information processing system according to a fourth embodiment. [Diagram 25] 13 is a flowchart showing an example of a linking process in the fourth embodiment. [Figure 26] 13 is a flowchart showing an example of an effect display process in the fourth embodiment. [Figure 27] FIG. 13 is a diagram showing an example of a screen for accepting selection of a product for which a POP effect is to be estimated in the fourth embodiment. [Figure 28] FIG. 10 is a diagram illustrating an example of index values ​​according to the type of POP. [Figure 29]FIG. 10 is a diagram illustrating an example of colors displayed according to ranks. [Diagram 30] FIG. 13 is a diagram showing an example in which the area of ​​a product is output based on the color determined by the display control processing unit in the fourth embodiment. [Diagram 31] FIG. 13 is a diagram showing an example in which a POP area is output based on a color determined by a display control processing unit in the fourth embodiment. [Diagram 32] FIG. 13 is a diagram showing an example in which the product area is divided into categories and a color according to the rank of each category is output in the fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] An example of the overall processing function of information processing system 1 of the present invention is shown in a block diagram in Fig. 1, an example of the processing function of POP detection processing unit 20 is shown in Fig. 2, and an example of the processing function of effect estimation processing unit 22 is shown in Fig. 3. Information processing system 1 uses a management terminal 2 and an image data input terminal 3. Management terminal 2 is a computer used by an organization such as a company that operates information processing system 1. Furthermore, image data input terminal 3 is a terminal for inputting image data obtained by photographing display shelves in a store.

[0037] The management terminal 2 and the image data input terminal 3 in the information processing system 1 are realized by using a computer. An example of the hardware configuration of a computer is shown in Fig. 4. The computer has a calculation device 70 such as a CPU that executes the calculation processing of a program, a storage device 71 such as a RAM or a hard disk that stores information, a display device 72 such as a display that displays information, an input device 73 such as a keyboard or a mouse that can input information, and a communication device 74 that transmits and receives the processing results of the calculation device 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN.

[0038] In the case where the computer is equipped with a touch panel display, the display device 72 and the input device 73 may be integrally configured. Touch panel displays are often used in portable communication terminals such as tablet computers and smartphones, but are not limited to these.

[0039] The touch panel display is a device that integrates the functions of the display device 72 and the input device 73 in that input can be made directly on the display by a predetermined input device (such as a touch panel pen) or a finger.

[0040] The image data input terminal 3 may include a photographing device such as a camera in addition to the above devices. A portable communication terminal such as a mobile phone, a smartphone, or a tablet computer may also be used as the image data input terminal 3. The image data input terminal 3 photographs the display shelves in the store and inputs image data of the display shelves (photographed image data) to the management terminal 2. Products are displayed on the display shelves, and POPs are attached to the display shelves, their vicinity, or the products. Therefore, the image data includes images of the displayed products such as the products displayed on the display shelves, product tags, shelf components, and POPs. It is preferable that unnecessary objects such as people are not photographed.

[0041] The functions of each means in the present invention are only logically distinct, and may be physically or practically the same area. The order of processing in each means of the present invention may be changed as appropriate. Also, some of the processing may be omitted. For example, the normalization processing described below may be omitted. In that case, processing can be performed on image data that has not been normalized.

[0042] The information processing system 1 includes a POP detection processor 20, a data storage unit 21, and an effect estimation processor 22.

[0043] The POP detection processing unit 20 executes a process for detecting POPs or candidates thereof that appear in the photographed image data or image data that has been subjected to a normalization process (normally-oriented image data) described later, based on the photographed image data. The data of the image to be processed, including the photographed image data and normal-oriented image data, will be collectively referred to simply as "image data."

[0044] The data storage unit 21 stores image data received from the image data input terminal 3, the shooting date and time, the store identification information, the display shelf identification information for identifying the display shelf, the image data identification information, and the like, in association with each other. The image data may be image data to be processed by the present invention. In general, when simply shooting, it is difficult to shoot the object in a state in which it is facing the camera, so it is preferable to execute a correction process to correct the state to a state in which it is facing the camera, such as a keystone correction process. When multiple images of one display shelf are shot, image data in which the images are combined into one image data is also included. In addition, image data after a distortion correction process is executed is also included in the image data. In addition, the data storage unit 21 stores various data (including information) used in the processing of the information processing system 1 of the present invention.

[0045] The effect estimation processing unit 22 estimates the effect of the POP by using the POP detected in the POP detection processing.

[0046] The POP detection processing unit 20 includes an image data input reception processing unit 200 , an image data normalization processing unit 201 , an object detection processing unit 202 , and an individual discrimination processing unit 203 .

[0047] The image data input reception processing unit 200 receives input of image data (photographed image data) of a display shelf in a store photographed by the image data input terminal 3, and stores the image data in the data storage unit 21. It is preferable to receive input from the image data input terminal 3 together with store identification information such as the photographing date and time and the store name, display shelf identification information for identifying the display shelf, and image data identification information for identifying the photographed image data, in addition to the photographed image data. FIG. 8 shows an example of photographed image data. Note that FIG. 8 shows a case where one display shelf is photographed in the photographed image data, but multiple display shelves may be photographed, or a part of a display shelf may be photographed. In addition, although the processing is not particularly specified in the present invention, display shelves and shelf levels may be long in the horizontal direction. Therefore, in the processing, the image may be divided into a certain width, and each processing target may be within the divided range.

[0048] The image data orthogonalization processing unit 201 generates orthogonalized image data by performing a process (orthogonalization process) for correcting the photographed image data stored in the data storage unit 21 so that the photographed object faces the right way, for example, by performing a keystone correction process. The keystone correction process is a correction process performed so that the shelves of the display shelves shown in the photographed image data are horizontal and the products displayed there and the product tags (for example, price tags) for those products are vertical. Orthogonalization is a process of transforming the photographed image data so that the optical axis of the lens of the photographing device is aligned along the perpendicular direction of the plane of the photographed object, so that the image data is the same as when photographed from a sufficiently far away, for example, a keystone correction process. The image data that has been orthogonalized (orthogonalized image data) may be stored in the data storage unit 21 described above.

[0049] The keystone correction process executed by the image data normalization processor 201 accepts input of designation of four vertices in the captured image data, and executes the keystone correction process using each of the vertices. The four vertices to be designated may be the four vertices of a shelf in a display shelf, or the four vertices of the shelf position in a display shelf. Alternatively, they may be the four vertices of a group of two or three shelf levels. Any four points can be designated as the four vertices. The image data normalization processor 201 can use various known processing methods.

[0050] The object detection processing unit 202 executes a process of detecting an object appearing in the captured image data stored in the data storage unit 21 or the image data rectified by the image data rectification processing unit 201. The object detection process is preferably performed using so-called deep learning. In this case, the image data is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the type (class) of the object appearing in the input image data is output as an output value.

[0051] As the learning model, the above-mentioned learning model is learned by performing a predetermined learning process using annotation data for products, product tags (price tags), shelf components, etc. that may appear in the image data. For example, data that associates image data of a product with the fact that it is a product or the product name or product code (JAN code, etc.) is used as annotation data, and a learning process for the learning model is performed. Image data is image data in the annotation data, and is a class indicating that it is a product, or that it is a product and the product identification information such as the product name or product code is "product type". Similarly, image data of a product tag is image data in the annotation data, and is a class indicating that it is a product tag, or that it is a product tag and the product name or product identification information is "product tags". Furthermore, image data of a shelf component is image data in the annotation data, and is a class indicating that it is a shelf component or the part name of the shelf component is "shelf component". The types (classes) of objects used as annotation data by the object detection processing unit 202 include things that are generally displayed on display shelves, such as merchandise, product tags, and shelf components, but are not limited to these and the object detection processing unit 202 may be trained to detect any object.

[0052] The individual discrimination processing unit 203 discriminates that the object types (classes) detected from the image data by the object detection processing unit 202 as types other than merchandise, product tags, and shelf components are POPs or POP candidates. Furthermore, when an object detected as a type (class) of merchandise, product tags, or shelf components satisfies a predetermined condition, the individual discrimination processing unit 203 discriminates that the object type (class) is POP or a POP candidate.

[0053] For example, for an object detected as a type of commodity, it is determined whether or not there is a product-specific advertisement (POP) in that area, and if it is determined that there is a POP, the type of object in that area is determined to be a POP or a POP candidate. In this case, there is a method of detecting a POP attached to a displayed product from the area of ​​an object detected as a type of commodity, and a method of detecting a POP by comparing the image data of the area of ​​an object detected as a type of commodity with the image data of a product without a POP.

[0054] In the former method, a large number of image data including POPs attached to displayed products are collected in advance, and the POPs are annotated for each image data including a POP. A learning model that has been trained in advance using image data of normally displayed products without POPs as annotation data is then trained using the image data annotated with the POPs to generate a new learning model. Image data of the area of ​​an object detected as a type of product can be input to this new learning model, and in addition to outputting product identification information during product recognition processing, it can also additionally output whether or not a POP is attached to the displayed product in the image data.

[0055] In the latter method, after performing product recognition processing on image data of the area of ​​an object detected as a commodity, standard image data (e.g., sample data) without a POP of the recognized commodity is compared with the image data of the area of ​​the object, and the similarity is compared. If the similarity is equal to or greater than a predetermined value, it is determined that the object detected as a commodity does not have a POP, and if the similarity is less than the predetermined value, it is determined that the object detected as a commodity has a POP or a POP candidate.

[0056] In the above-mentioned method, by comparing image data of an area of ​​an object detected as a commodity with standard image data of the commodity recognized by the commodity recognition process without a POP, it is possible to determine whether or not it is a POP based on the location of the different area, which can result in more accurate determination. For example, if it is determined that the different area between the two image data is localized in a specific position, for example, near the top, shoulder, or tail of the commodity, it is possible to determine that it is a POP or a candidate for a POP.

[0057] As the commodity recognition process, various known processing methods can be used. For example, a process of identifying commodity identification information can be executed for image data of the object region using deep learning. In this case, the image data of the object region can be input to a learning model in which the weighting coefficient between neurons in each layer of a neural network consisting of many intermediate layers is optimized, and the commodity identification information can be identified based on the output value. As the learning model, a learning model in which commodity identification information is given as correct answer data to various image data in which the commodity is shown can be used. In addition, the commodity sample data (image data of the commodity shown from one or more directions or data such as feature amounts based thereon) can be compared with the image data of the object region, and the commodity identification information can be identified using the similarity. The commodity recognition process is not limited to these.

[0058] Furthermore, for objects detected as being of a product tag type, it is determined whether the image data of the area of ​​the object meets the requirements of a product tag type. For example, OCR recognition processing is performed on the area of ​​an object detected as being of a product tag type, and if it contains characters that are typically included in product tags, such as a price or product name, it is determined to be a product tag, and if not, it is determined to be a POP or a POP candidate. Alternatively, if the result of the OCR recognition processing contains a predetermined word, such as a word typically used in POP, such as "member discount," "tester," "sample," or "sample," it may be determined to be a POP or a POP candidate. Also, if handwritten characters are included, it may be determined to be a POP or a POP candidate.

[0059] For objects detected as being classified as shelf components, the system determines whether the image data of the object's area meets the requirements for a shelf component. For example, the system performs OCR recognition processing on the area of ​​an object detected as being classified as a shelf component, and if it contains a specific word, such as a member discount or other word that is typically used in POPs, it is determined to be a POP or a POP candidate; if not, it is determined to be a shelf component.

[0060] In addition, if the individual discrimination processing unit 203 determines that the type (class) of image information of an object area is a candidate for POP, the image information of the object area may be displayed on the display device 72 of the management terminal 2, etc., and input as to whether it is a POP or not may be accepted, and if the input is a POP, the type (class) of the image information of the object area may be set to POP.

[0061] When the individual discrimination processing unit 203 discriminates the area of ​​a POP, it assigns POP identification information for identifying the POP, and stores the information in association with the image data of the POP area in the data storage unit 21. It is also preferable that the image data showing the POP be associated with the shooting date and time, store identification information, and display shelf identification information.

[0062] The effect estimation processing unit 22 includes a POP selection reception processing unit 220 , a product selection reception processing unit 221 , an analysis processing unit 222 , and an effect output processing unit 223 .

[0063] The POP selection reception processing unit 220 extracts image data of the areas of the POPs detected by the POP detection processing unit 20 from the data storage unit 21, and receives the selection of the POPs to be the targets of effect estimation.

[0064] The product selection reception processing unit 221 extracts products identified from the image data from the data storage unit 21 and receives the selection of products to be subjected to effect estimation. Various known techniques can be used for the process of identifying products from image data and storing them in the data storage unit 21. Also, image data of products may be stored in advance in the data storage unit 21 from the image data.

[0065] For example, the area of ​​the product is identified by identifying thin, narrow shadows that occur between products from the image data, identifying repeating patterns in the image, identifying steps on the top edge of the package, identifying dividing positions based on constraints such as the product width being the same, etc. Then, an image matching process is performed on the image information of the product area with pre-stored image data of a reference product (sample data), and the product identification information (a specified code such as the product name, JAN code, etc.) of the sample data that has the highest similarity is identified as the product identification information of the product in the area. Through this process, the product shown in the image data can be identified, and the product image data and product identification information can be associated and stored in the data storage unit 21.

[0066] As another method, the product area may be specified and the product identification information may be identified using machine learning such as deep learning. In this case, the area to be processed, for example, image data, may be input to a learning model in which the weighting coefficients between the neurons in each layer of a neural network consisting of many intermediate layers are optimized, and the product area may be specified and the product identification information may be identified based on the output value. As a learning model, image data of various areas to be processed, for example, display shelves and shelf areas, may be used in which the product area and its product identification information are given as correct answer data.

[0067] The product identification process may be performed by the object detection processing unit 202, the individual discrimination processing unit 203, etc. Image data of the area of ​​the identified product is associated with the product identification information and stored in the data storage unit 21. It is also preferable that the image data showing the product is associated with the shooting date and time, the store identification information, and the display shelf identification information.

[0068] 9 shows an example of a screen that accepts the selection of a POP by the POP selection reception processing unit 220 and the selection of a product by the product selection reception processing unit 221. The effect estimation processing unit 22 accepts the selection of some or all of the store, display shelf, date, etc. for which the effect of the POP is to be estimated, using a predetermined method. Upon accepting the selection, the effect estimation processing unit 22 extracts the corresponding POP image data and product image data from the data storage unit 21 based on the store identification information of the selected store, the display shelf identification information of the selected display shelf, and the shooting date corresponding to the selected date, and displays them as a list on the screen shown in FIG. 9 described above.

[0069] The analysis processing unit 222 executes a process for estimating the effect of a POP. For example, when an "Analysis" button on the screen shown in Fig. 9 is pressed, the estimation processing unit receives the POP selected by the POP selection reception processing unit 220 and the product selected by the product selection reception processing unit 221, and extracts POP identification information corresponding to the image data of the selected POP, the shooting date and time of the image data in which the POP appears, store identification information, and display shelf identification information from the data storage unit 21. In addition, the estimation processing unit extracts product identification information corresponding to the image data of the selected product, product identification information corresponding to the image data in which the product appears, and the shooting date and time of the image data in which the product appears, store identification information, and display shelf identification information from the data storage unit 21.

[0070] Then, the analysis processing unit 222 acquires sales information of the product having the product identification information at the store on the date (for example, various information related to the sales of the product, such as the number of sales, sales amount, number of sales, reservation amount, number of reservations, etc.) from a product sales management system such as a POS system based on the date, store identification information, and product identification information selected by the effect estimation processing unit 22. Note that this date and time is not limited to the date, but may include multiple dates before and after the date. In other words, it is sufficient to acquire sales information of the product having the product identification information at the store on the date on which the POP was installed. Also, input of a date on which the POP was not installed is accepted, and sales information of the product having the product identification information is acquired. Note that the number of days on the date on which the POP was installed and the date on which the POP was not installed should be the same for comparison. It may be one day, one week, or any appropriate number of days.

[0071] The analysis processing unit 222 compares the sales information of the product on a date with the POP and the sales information of the product on a date without the POP as described above, and calculates the contribution of the POP (an index showing the improvement in the number of sales of the product due to the POP, for example, the rate of increase). For example, (Number of items sold on days with POP) ÷ (Number of items sold on days without POP) The contribution degree is calculated by calculating

[0072] As another calculation formula, the contribution of the POP may be calculated by taking into account the number of faces (the number of image data of the product area). The number of faces is the number of products displayed on the display shelf shown in the image data, and can be calculated for each product. Since the object detection processing unit 202 detects image data of the area having the type of product, the area is regarded as a face. Then, in the above-mentioned product identification process, the number of image data of the product area having the same identification information as the product identification information of the selected product in the same image data in which the selected product is shown is specified. This number is the number of faces. Information on the number of faces may be stored in the data storage unit 21 in association with the image data, the product identification information, etc. Therefore, the number of faces of the product on the display shelf on the date on which the selected POP is installed and the number of faces of the product on the same display shelf (the display shelf with the same display shelf identification information) on the date on which the POP is not installed are extracted from the data storage unit 21.

[0073] Then, the analysis processing unit 222 performs, for example, (Number of items sold on days with POP / Number of faces of items on days with POP) ÷ (Number of items sold on days without POP / Number of faces of items on days without POP) The contribution degree is calculated by calculating

[0074] The formula for calculating the contribution rate is not limited to the above, and any formula may be used as long as it allows comparison using sales information such as the number of sales on days with POP and the number of sales on days without POP.

[0075] Furthermore, when there are multiple POPs, the contribution degree may be calculated taking into account the influence of the multiple POPs. For example, when two POPs (POPa, POPb) are selected, there are combinations of the presence or absence of each POP as shown in FIG. 10. The number of faces in each case is normalized under the assumption that the number of faces is the same for all four cases according to the presence or absence of POPa and POPb, and the contribution degree of POPa can be calculated by dividing C by D in FIG. 10, and the contribution degree of POPb can be calculated by dividing B by D. In addition, the contribution degree when POPa and POPb are present can be calculated by dividing A by D, and the value may be used to correct the contribution degree of POPa and the contribution degree of POPb. When the number of POPs is three or more, the number of faces in each case can be normalized under the assumption that the number of faces is the same for each case, and the calculation can be performed in the same manner.

[0076] The effect output processing unit 223 outputs the contribution degree calculated by the analysis processing unit 222. For example, an example of outputting and displaying the contribution degree of POP for each product is shown in FIG.

[0077] As described above, it is possible to detect POP from image data and estimate the effect of POP on product sales. EXAMPLES

[0078] Next, an example of a processing process using the information processing system 1 of the present invention will be described with reference to the flowcharts of FIGS.

[0079] First, the process (S100) of detecting a POP contained in image data will be described with reference to the flowcharts of FIGS.

[0080] Image data of a store's display shelves is input from the image data input terminal 3, and the input is accepted by the image data input acceptance processing unit 200 of the management terminal 2 (S200). The image data input acceptance processing unit 200 also accepts input of the shooting date and time, store identification information, display shelf identification information, and image data identification information of the image data. The image data input acceptance processing unit 200 then associates the input image data, shooting date and time, store identification information, display shelf identification information, and image data identification information of the image data, and stores them in the data storage unit 21. Note that the store identification information, display shelf identification information, and image data identification information may be assigned automatically.

[0081] When a specified operation input is received in the management terminal 2, the image data alignment processing unit 201 extracts the image data to be stored in the data storage unit 21, receives input of four shelf positions (positions of display shelves) which are the vertices for performing alignment processing such as keystone correction processing, and executes the alignment processing (S210).

[0082] Then, by receiving a predetermined operation input in the management terminal 2, the object detection processing unit 202 executes an object detection process on the image data that has been normalized (S220) and classifies the type of object depicted in the image data (S230). FIG. 12 shows an example of a result of executing an object detection process on image data. In FIG. 12, "a" means a product, "b" means a product tag, "c" means a shelf, and "d" means a type of object other than a to c that has been detected. Note that the type of object detected from the image data only needs to be associated with each region, and is not written on each image. The region of the detected object may be cut out from the image data, but it does not have to be actually cut out as long as the image data of the region is in a processable state. Note that cutting out the image data of a region includes not only actually cutting out the image data of the region from the image data, but also specifying the image data of the region as a processable state without cutting out the image data of the region.

[0083] The individual discrimination processing unit 203 discriminates and cuts out the areas of objects of types other than products, product tags, and shelf components (areas of objects "d") from the object areas of the image data detected by the object detection processing unit 202 as POP areas (S240, S250).

[0084] In addition, the individual discrimination processing unit 203 determines whether an area of ​​an object in the image data detected by the object detection processing unit 202 (S240) that is detected as a type of commodity satisfies a predetermined condition, and determines whether the area is a POP area (S260).

[0085] For example, for the area of ​​an object detected as a type of commodity, it is determined whether or not the area has a product-specific advertisement (POP), and if it is determined that a POP is present, the type of the area is determined to be a POP. For the area of ​​an object detected as a type of commodity, a process is executed to detect POPs attached to displayed products, or a POP is detected by comparing the image data of the area of ​​the object detected as a type of commodity with image data of products without POPs (sample image data of the products).

[0086] Similarly, for an object region detected as a product tag, the system determines whether the image data of the object region meets the requirements for a product tag, and for an object detected as a shelf member, the system determines whether the image data of the object region meets the requirements for a shelf member.

[0087] As described above, for the area of ​​an object detected as a type of merchandise, product tags, or shelf components (S240), it is determined whether the area satisfies a predetermined condition for each object (S260), and if the condition is satisfied, all or part of the area is determined to be a POP (S270). An example of this process is shown in Figure 13. Also, for the object detection results in Figure 12, an example of each area after areas of objects detected as a type of merchandise, product tags, or shelf components that satisfy a predetermined condition are determined to be POP areas is shown in Figure 14.

[0088] By executing the above-mentioned processing in the individual discrimination processing unit 203, it is possible to detect POP areas in the image data.

[0089] As described above, when the POP detection processing unit 20 detects image data of a POP area, it assigns POP identification information and stores the data in the data storage unit 21 in association with the image data of the POP area. At this time, the photographing date and time of the image data in which the POP was detected, the store identification information, the display shelf identification information, and the image data identification information may also be stored in association with the image data. Note that the photographing date and time of the image data detected other than the image data of the POP area, the store identification information, the display shelf identification information, and the image data identification information may also be stored in association with the image data. In addition, for image data of an object area detected as a commodity or a commodity tag, the commodity identification information of the commodity shown in the image data of the area, the commodity identification information shown on the commodity tag, the price, the manufacturer, and other information may be detected using known technology, and further stored in association with the image data storage unit 21.

[0090] Then, when the POP detection processing unit 20 detects a POP, the effect estimation processing unit 22 executes a POP effect estimation process using the POP detected by the POP detection processing unit 20 (S110).

[0091] First, the effect estimation processing unit 22 receives a selection in a predetermined manner of the store, display shelf, and date for which the effect estimation of the POP is desired. Upon receiving the selection, the effect estimation processing unit 22 extracts the corresponding POP image data and product image data from the data storage unit 21 based on the store identification information of the selected store, the display shelf identification information of the selected display shelf, and the shooting date corresponding to the selected date, and displays them as a list on a screen such as that shown in FIG.

[0092] The selection of a POP for effect estimation from this screen is received by the POP selection reception processing unit 220 (S300), and the selection of a product is received by the product selection reception processing unit 221 (S310).

[0093] The analysis processing unit 222 extracts POP identification information, etc. corresponding to the POP based on the POP selected by the POP selection reception processing unit 220 from the data storage unit 21, and extracts product identification information, etc. corresponding to the product based on the product selected by the product selection reception processing unit 221 from the data storage unit 21.

[0094] The analysis processing unit 222 then acquires sales information of the product corresponding to the product identification information from a product sales management system such as a POS system based on the information on the date, product identification information, store identification information, etc., on which the selection was accepted by the effect estimation processing unit 22 (S320). Note that if the selection of date, store identification information, or display shelf identification information is not accepted in the effect estimation process, the date, store identification information, and display shelf identification information corresponding to the POP selected by the POP selection acceptance processing unit 220 and the product selected by the product selection acceptance processing unit 221 may be extracted.

[0095] As described above, the analysis processing unit 222 compares the sales information of the product on a date with the POP with the sales information of the product on a date without the POP, and calculates the contribution of the POP (an index showing the improvement in the sales volume of the product due to the POP, for example, the rate of increase) (S330). for example, (Number of items sold on days with POP) ÷ (Number of items sold on days without POP) The contribution degree is calculated by calculating

[0096] The calculated contribution degree is output by the effect output processing section 223 (S340).

[0097] By checking the output result output by the effect output processing section 223 in the above-described processing, for example the output result in FIG. 11, the estimated POP effect can be recognized. EXAMPLES

[0098] Next, another embodiment of the detection process in the POP detection processor 20 will be described. The detection processor of this embodiment further includes an area detection processor 204 in addition to the configuration of the first embodiment. An example of the configuration of the POP detection processor 20 in this embodiment is shown in Fig. 15, and an example of the POP detection process is shown in the flowchart of Fig. 16. Also, an example of the outline of the process is shown in Fig. 17, and an example of the individual discrimination process is shown in Fig. 18.

[0099] The area detection processing unit 204 detects the product display area where the products are displayed, the product tag area where the product tags are placed, and the upper shelf area which is the area above the shelves, from the image data (including normalized image data) stored in the data storage unit 21. The product display area, product tag area, and upper shelf area may be detected by the operator of the management terminal 2 manually specifying the product display area, product tag area, and upper shelf area, which are then accepted by the area detection processing unit 204, or the product display area, product tag area, and upper shelf area may be automatically detected from the second time onwards, based on information on the product display area, product tag area, and upper shelf area that was manually accepted the first time. The area detection processing unit 204 may also detect areas other than these and use them as processing targets.

[0100] FIG. 19 shows a schematic diagram of the state in which input of designation of the product display area, product tag area, and shelf top area has been received for image data that has been normalized from image data captured of a display shelf on which products are displayed.

[0101] When detecting the product display area, product tag area, and shelf top area, the area detection processing unit 204 may use deep learning to identify the product display area, product tag area, and shelf top area. In this case, the above-mentioned orthogonalized image data may be input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the product display area, product tag area, and shelf top area may be detected based on the output value. In addition, as the learning model, various image data may be given the product display area, product tag area, and shelf top area as correct answer data.

[0102] The area detection processing unit 204 may detect areas using various known methods in addition to using deep learning as described above.

[0103] The area detection processing unit 204 may actually extract the detected area as image data, or may virtually extract the area by specifying the range of the area without actually extracting the area as image data. In this case, the range of the area can be specified by coordinates.

[0104] The object detection processing unit 202 in this embodiment may perform object detection processing within the range of image data as in the first embodiment, or may perform the processing for each region detected by the region detection processing unit 204.

[0105] The individual discrimination processing unit 203 in this embodiment discriminates the type of each object detected by the object detection processing unit 202 using the type (class) of the object and the area in which the object is located. For example, the discrimination is performed according to predetermined individual discrimination conditions such as the discrimination tables shown in Figs. 17 and 18.

[0106] “a1” to “a4”, “b1” to “b10”, and “d1” to “d5” in FIG. 17 correspond to “a1” to “a4”, “b1” to “b10”, and “d1” to “d5” in FIG.

[0107] 17 and 18 are merely examples, and the object types are also merely examples. Also, these are merely examples of POP types, and the present invention is not limited to these, and any type can be set.

[0108] Next, an example of the POP detection process in the POP detection processor 20 of this embodiment will be described with reference to the flowchart of FIG.

[0109] Image data of a store's display shelves is input from the image data input terminal 3, and the input is accepted by the image data input acceptance processing unit 200 of the management terminal 2 (S400). The image data input acceptance processing unit 200 also accepts input of the shooting date and time, store identification information, display shelf identification information, and image data identification information of the image data. The image data input acceptance processing unit 200 then stores the accepted input image data, shooting date and time, store identification information, display shelf identification information, and image data identification information in the data storage unit 21 in association with each other.

[0110] When a specified operation input is received in the management terminal 2, the image data alignment processing unit 201 extracts the image data to be stored in the data storage unit 21, receives input of four shelf positions (positions of display shelves) which are the vertices for performing alignment processing such as keystone correction processing, and executes the alignment processing (S410).

[0111] Then, by receiving a specified operational input at the management terminal 2, the object detection processing unit 202 performs area detection processing on the aligned image data to detect the product display area, product tag area, and shelf top area (S420), and determines the range of each area.

[0112] Furthermore, by receiving a predetermined operation input in the management terminal 2, the object detection processing unit 202 executes an object detection process on the normalized image data (S430) and classifies the type of the object appearing in the image data (S440). An example of the result of executing the object detection process on the image data is classified as shown in FIG. 12, similar to the first embodiment.

[0113] Then, for each object in the image data detected by the object detection processing unit 202, the individual discrimination processing unit 203 classifies the detected object in accordance with predetermined discrimination conditions using the type of the object and the area in which the object is located (the area determined in S420).

[0114] Furthermore, individual discrimination processing section 203 uses the type of object in the image data detected by object detection processing section 202 and the area in which the object is located to determine whether a predetermined condition is satisfied (S450), and determines whether the type of the object is a POP or something other than POP (products, product tags, shelf components, etc.) (S460, S470). Note that which area each object is located in may be determined by any appropriate method, for example, by determining whether the center of gravity of the object's area is included in the area of ​​products, product tags, or the top of the shelf.

[0115] For example, when an object is in a product display area and the type of the object is a commodity, the individual discrimination processing unit 203 judges whether or not there is a commodity-adjacent POP in the area of ​​the object. For example, as in the first embodiment, discrimination is performed by a method of detecting a POP attached to a displayed commodity from the area of ​​the object detected as a commodity, and a method of detecting a POP by comparing image data of the area of ​​the object detected as a commodity with image data of a commodity without a POP. If a commodity-adjacent POP is detected, the area is discriminated as a POP (a2), and other areas are discriminated as a commodity (a1). If a commodity-adjacent POP is not detected, an OCR recognition process is performed on the area to discriminate whether a specific word used in POP such as "tester", "sample", or "sample" is included, and if not, the object in the area is discriminated as a commodity (a1). If it is included, the object in the area is discriminated as a POP such as a tester or sample (a3).

[0116] Furthermore, when an object in the product display area is a product tag, the individual discrimination processing unit 203 determines that the object in that area is a POP (d1).

[0117] Also, when an object in a product display area is classified as a shelf member, the individual discrimination processing unit 203 discriminates that the object in that area is a shelf member (c).

[0118] Moreover, if an object is in the product display area and the type of the object is other than products, product tags, or shelf members, it is determined to be a POP (d2).

[0119] When the object is in the product tag area and the type of the object is a product, the individual discrimination processing unit 203 discriminates that the object is a POP such as a tester or a sample (d4).

[0120] If the object is in the product tag area and the type of the object is a product tag, OCR recognition processing is performed on the object area, and if it is determined that there is handwriting in the area, it is determined to be a POP (retail POP) created by a store (retailer) (b4). If it is determined that there is no handwriting in the area as a result of OCR recognition processing on the object area, and if it is determined that the area contains a price and product name display, and if it contains a word indicating a condition such as "member", the type of the object is determined to be a product tag indicating a set product display (b2). On the other hand, if it is determined that the price and product name display is included, but does not contain a word indicating a condition such as "member", the type of the object is determined to be a product tag (b1). If it is determined that the price and product name display are not included, and if there is a point display, the type of the object is determined to be a POP with a "point display" (b3). Furthermore, if there is no point display in the above, it is determined to be a POP created by the product manufacturer (manufacturer POP) (b9). Note that here, a case has been shown in which manufacturer POP and retail POP are distinguished from each other and are determined to be the same, but it may simply be determined as a POP.

[0121] Furthermore, when an object is in the product tag area and the type of the object is a shelf member, the individual discrimination processing unit 203 discriminates that the object in that area is a shelf member (c).

[0122] Furthermore, if an object is in the product tag area and the type of the object is other than products, product tags, or shelf components, OCR recognition processing is performed on that area. Then, it is determined whether or not the object contains specific words used in POP such as "tester," "sample," or "sample," and if so, it is determined to be a POP such as a tester or sample (d4). On the other hand, if the object does not contain specific words used in POP such as "tester," "sample," or "sample," it is determined to be a POP such as a promotional item (d3).

[0123] When the object is in the upper shelf area and the type of the object is a commodity, the individual discrimination processing unit 203 discriminates that the object is a stock commodity (a4).

[0124] If an object is located in the upper shelf area and the type of the object is a product tag, OCR recognition processing is performed on the area of ​​the object, and if it is determined that there is handwriting in the area, it is determined to be a retail POP (b8). If the result of OCR recognition processing on the area of ​​the object is that it is determined that there is no handwriting in the area, and if it is determined that the area contains a display of the price and product name, and if it contains a word indicating a condition such as "member", the type of the object is determined to be a product tag indicating a set product display (b6). On the other hand, if it is determined that the area contains a display of the price and product name, but does not contain a word indicating a condition such as "member", the type of the object is determined to be a product tag (b5). If it is determined that the above-mentioned price and product name display is not included, and if there is a display of points, the type of the object is determined to be a POP with a "point display" (b7). Furthermore, if there is no display of points in the above, it is determined to be a manufacturer POP (b10).

[0125] Furthermore, when an object is in the shelf upper region and the type of the object is a shelf member, the individual discrimination processing unit 203 discriminates that the object in that region is a shelf member (c).

[0126] Furthermore, if an object is in the upper shelf area and the type of the object is other than merchandise, merchandise tags, or shelf components, it is determined to be a POP such as a large promotional item (d5).

[0127] As described above, the individual discrimination processing unit 203 can use the area detected by the area detection processing unit 204 and the type of object detected by the object detection processing unit 202 to determine whether the object is a POP or not, and detect the area of ​​the POP in the image data. EXAMPLES

[0128] As a modified example of the processing of the first and second embodiments, the processing shown in the flowchart of FIG. 20 may be performed. In this case, if the type of the object detected by the object detection processing unit 202 is other than merchandise, merchandise tags, or shelf members, the individual discrimination processing unit 203 determines that the object is a POP (S560). Then, for other objects, the same processing as the processing of S450 to S470 may be performed. That is, using the type of the object in the image data detected by the object detection processing unit 202 and the area in which the object is located, it is determined whether a predetermined condition is satisfied (S570), and it is determined whether the object is a POP or something other than a POP (such as merchandise, merchandise tags, or shelf members) (S580, S590). Note that the processing from S500 to S540 may be the same as the processing from S400 to S440. EXAMPLES

[0129] Using the processes in the second and third embodiments, the POP may be linked to the product to estimate the effect of the POP.

[0130] The information processing system 1 of this embodiment has a POP detection processing unit 20, a data storage unit 21, a linking processing unit 23, and an effect display processing unit 24. An example of the configuration of the information processing system 1 in this case is shown in Fig. 21, an example of the configuration of the linking processing unit 23 is shown in Fig. 22, and the configuration of the effect display processing unit 24 is shown in Fig. 23. The processing of the POP detection processing unit 20 may be similar to that of the second or third embodiment.

[0131] The linking processor 23 links a POP with a product corresponding to the POP (a product advertised by the POP). The linking processor 23 includes a POP determination processor 230 and a corresponding product determination processor 231.

[0132] The POP determination processor 230 determines the text information displayed in the POP area. The POP determination processor 230 extracts the image data of the POP area detected by the POP detection processor 20 from the data storage unit 21, and determines the text information of the image data displayed in that area using a known method such as OCR recognition processing or deep learning.

[0133] For example, the POP determination processor 230 binarizes the image data of the POP area and performs OCR recognition processing on the binarized image data. If necessary, it may also perform a determination process such as checking against a dictionary of product identification information stored in advance.

[0134] Another process of the POP determination processing unit 230 may be to determine character information in image data of a POP area using machine learning such as deep learning. In this case, the image data of the POP area may be input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of many intermediate layers are optimized, and the character information contained in the image data of the area may be determined based on the output value. In addition, the learning model may be one in which image data of various POP areas and the character information contained in the areas are given as correct answer data.

[0135] As another process of the POP determination processing unit 230, image data of a product contained therein may be determined instead of using character information. For example, it may be determined by image matching processing or deep learning that the image data displayed in the POP area contains sample data of a product that has been stored in advance.

[0136] The corresponding product determination processing unit 231 determines the product that the POP corresponds to. In other words, it identifies the product that the POP is trying to advertise and links the product to the POP. The corresponding product determination processing unit 231 uses a part or all of the text information displayed in the area of ​​the POP determined by the POP determination processing unit 230 to associate the POP with the product displayed on the display shelf where the POP is located.

[0137] The process of associating a POP with a product can be performed, for example, as follows: First, the store identification information, display shelf identification information, and date information associated with the image data of the POP area are extracted from the data storage unit 21. Then, from among the objects of products detected by the object detection processing unit 202 or the like and stored in the data storage unit 21, products that have the same store identification information, display shelf, and date information and whose product identification information matches part or all of the character information determined by the POP determination processing unit 230, in particular the product identification information indicating the product written on the POP, are identified.

[0138] This allows the POP to be associated with the product by associating the POP identification information of the POP with the product identification information of the product.

[0139] In order to identify products that correspond to a POP, rather than processing the entire display shelf, the type of POP determined by the individual discrimination processing unit 203 (Figures 17 and 18) may be used to change the range in which to search for corresponding products.

[0140] For example, if the product corresponds to the POP "a2" in FIG. 18, it is a POP attached to the product, so it is sufficient to search for the product in the range adjacent to the POP. Also, if the product corresponds to the POP "a3" in FIG. 18, it is a POP for "sample / tester", so it is sufficient to search for the product in the vicinity of the POP, for example, within the range of faces 2 and 3 from the range where the POP is located. Also, if the product corresponds to the POP "b3", "b4", and "b9" in FIG. 18, it is a POP located in the product tag area, so it is normal for the product to be located in the vicinity of the POP, so it is sufficient to search for the product in the range of faces 2 and 3 from the range where the POP is located. "b7", "b8", "b10", and "d5" in FIG. 18 are POPs installed at the top of the display shelf, so the entire product displayed on the display shelf is the search range. "d1" and "d2" in FIG. 18 are POPs installed in the product display area, so the product is located in the vicinity of the POP, so it is sufficient to search for the product in the vicinity of the POP, for example, within the range of faces 2 and 3 from the range where the POP is located. Furthermore, "d3" in Figure 18 is a POP in the product tag area, and "d4" is a "sample / tester" POP in the product tag area, so it is sufficient to search for products in the vicinity of the POP, for example, within a range of 2 or 3 faces from the range in which the POP is located.

[0141] Regarding the positions of the POP and the products, when an object is detected by the object detection processing unit 202, the position of the object can be treated as a face (a certain image area) and the positional relationship in the image data can be stored as coordinates, order, and position.

[0142] In addition, when the corresponding product determination processing unit 231 determines the corresponding product to which the POP corresponds, if the determination is made using image data in the POP determination processing unit 230, the POP can be linked to the product by associating the product identification information corresponding to the image data of the product contained in the image data displayed in the POP area with the POP identification information.

[0143] By carrying out the above-mentioned processing, the corresponding product determination processing unit 231 can associate the POP with the product. The association between the POP and the product may be, for example, by associating the POP identification information with the product identification information.

[0144] The effect display processing unit 24 estimates and displays the effect of a POP associated with (corresponding to) a product. The effect display processing unit 24 has a product selection reception processing unit 240, a POP identification processing unit 241, an index processing unit 242, a display control processing unit 243, and a display output processing unit 244.

[0145] The product selection reception processing unit 240 of this embodiment receives the selection of a product for which the effect of the corresponding POP is to be estimated. There are no limitations on the method of receiving the product selection, and for example, a selection of some or all of the store, display shelf, date, etc. for which the effect of the POP is to be estimated is received in a predetermined manner. Upon receiving the selection, the effect estimation processing unit 22 extracts image data of the corresponding product from the data storage unit 21 based on the store identification information of the selected store, the display shelf identification information of the selected display shelf, and the shooting date corresponding to the selected date, and displays it as a list as shown on the screen in FIG. 27. Then, from this list, the selection of a product for which the effect of the POP is to be estimated is received.

[0146] The product selection reception processing unit 240 may receive the selection of one product or multiple products. It may also receive the selection of products on a shelf of a display shelf or the entire display shelf. When the selection of multiple products is received, the effect display processing unit 24 performs processing for each selected product.

[0147] The POP identification processing unit 241 identifies a POP corresponding to a product whose selection has been accepted by the product selection reception processing unit 240. By identifying POP identification information corresponding to the product identification information of a product whose selection has been accepted by the product selection reception processing unit 240, it is possible to identify a POP corresponding to the selected product. Note that a product may be associated with multiple POPs, and in this case, each POP is identified. Note that the POP identification processing unit 241 executes its processing by identifying a POP corresponding to a product whose selection has been accepted by the product selection reception processing unit 240, but the POP to be processed may also be identified by accepting a POP selection.

[0148] In this case, a selection of some or all of the store, display shelf, date, etc. for which a POP effect estimation is to be performed is accepted in a predetermined manner. When the selection is accepted, the effect estimation processing unit 22 extracts the image data of the corresponding POP from the data storage unit 21 based on the store identification information of the selected store, the display shelf identification information of the selected display shelf, and the shooting date corresponding to the selected date, and displays them as a list on the screen. Then, from this list, a selection of the POP for which a POP effect estimation is to be performed is accepted. At this time, the product corresponding to the POP whose selection has been accepted is specified. Through such processing, the same processing can be performed even if a POP selection is accepted instead of a product selection.

[0149] The index processing unit 242 performs calculations using index values ​​based on the type of POP, such as summing up index values ​​based on the type of POP corresponding to the selected product.

[0150] In the index processing unit 242, as shown in Fig. 28, index values ​​are set according to the type of POP, and calculations are performed by adding up the index values ​​according to the type of POP corresponding to the selected product, etc. Then, the calculated index values ​​are divided into a predetermined rank, for example, into 10 ranks.

[0151] The display control processing unit 243 determines a display method according to the ranks divided by the index processing unit 242. For example, it determines the color to be displayed. For rank 1 (low total index value), it uses "blue" (or "white" when there is no index value), and for rank 10 (high total index value), it uses "red", and the display is divided into 10 colors as shown in FIG. 29, for example. This color is the color to be displayed in the area of ​​the product or POP. By displaying according to such ranks, it is possible to display something like a heat map.

[0152] The display output processing unit 244 of this embodiment displays and outputs the area of ​​the selected product or POP based on the color determined by the display control processing unit 243. An example of this display is shown in Fig. 30 and Fig. 31. Fig. 30(a) shows a case where the display of the product area is changed when one product is selected, and Fig. 30(b) shows a case where the display of the area of ​​each product is changed when multiple products are selected. Fig. 31(a) shows a case where the display of the POP area corresponding to the product is changed when a product is selected, and Fig. 31(b) shows a case where the display of the POP area corresponding to each product is changed when multiple products are selected. In addition to displaying the color of the entire product or POP area, a part or all of the product or POP area may be divided into a specified rectangle and the color may be changed within the range of the rectangle.

[0153] For convenience of the drawings, Figures 30 and 31 in this specification show the change in the display method by pattern, not by color. In addition to color, various display methods such as color shading, color brightness, different patterns, different highlighting, and the presence or absence of blinking patterns can be used. Also, instead of displaying on the image data, the rank itself may be displayed as text.

[0154] Next, an example of the processing in this embodiment will be described with reference to the flowcharts of FIGS.

[0155] First, similarly to the second or third embodiment, the POP detection processing unit 20 executes a POP detection process to detect POPs in the image data read from the image data input terminal 3 (S100). The POP detection process can be the same as that of each embodiment. The POP detected here is stored in the data storage unit 21. Similarly, since the product in the image data is also detected by the object detection processing unit 202, it is also stored in the data storage unit 21.

[0156] After the POP detection process is performed, the linking process is performed in the linking processor 23 (S120).

[0157] The POP determination processing unit 230 in the linking processing unit 23 extracts image data of the POP area stored in the data storage unit 21 at any timing, and determines the text information displayed in that area using known methods such as OCR recognition processing and deep learning (S600).

[0158] Then, using a part or all of the text information determined by the POP determination processing unit 230, the corresponding product determination processing unit 231 determines the product that the POP corresponds to (S610).

[0159] The corresponding product determination processing unit 231 extracts store identification information, display shelf identification information, and date information that are associated with the image data of the POP area from the data storage unit 21. Then, among the objects of merchandise detected by the object detection processing unit 202 or the like and stored in the data storage unit 21, the corresponding product is identified as a product that has the same store identification information, display shelf, and date information, and whose product identification information matches part or all of the character information determined by the POP determination processing unit 230, in particular the product identification information indicating the product written on the POP, thereby associating the POP with the product. The correspondence between the POP and the product can be achieved by storing the POP identification information and the product identification information in the data storage unit 21 in association with each other.

[0160] After the tying process in the tying processing unit 23 is completed as described above, the effect display processing unit 24 executes the effect display processing (S130).

[0161] The product selection reception processing unit 240 in the effect display processing unit 24 receives, in a predetermined manner, the selection of some or all of the store, display shelf, date, etc. for which POP effect estimation is desired. Upon receiving the selection, the effect estimation processing unit 22 extracts image data of the corresponding product from the data storage unit 21 based on the store identification information of the selected store, the display shelf identification information of the selected display shelf, and the shooting date corresponding to the selected date, and displays the image data as a list as shown on the screen in Fig. 27. Then, from this list, the selection of one or more products for which POP effect estimation is to be performed is received (S700).

[0162] The POP identification processing unit 241 identifies one or more POPs corresponding to the product selected and accepted by the product selection acceptance processing unit 240 (S710).

[0163] Then, the index processing unit 242 performs a calculation using the index value based on the type of POP identification information identified by the POP identification processing unit 241, and ranks the POP identification information according to a predetermined rank (S720).

[0164] Once the ranking has been performed by the index processing unit 242, the display control processing unit 243 determines the color to be displayed in the product or in the POP area corresponding to that product (S740), and the display output processing unit 244 displays the result on the display device 72 of the management terminal 2, as shown in Figures 30 and 31.

[0165] By executing the above-mentioned process, the effectiveness of the POP and the product can be determined. EXAMPLES

[0166] As a modification of the fourth embodiment, the effect display processing unit 24 may display any other items in addition to the effect estimation based on the association between the POP and the product. The other items may include, for example, customer feedback, recognition, sales, etc., but are not limited to these, and may be other items.

[0167] For example, as for customer feedback, the effect display processing unit 24 receives evaluations of a product selected by the product selection reception processing unit 240 through a customer questionnaire or the like, converts the evaluations into index values, calculates a total index value for the product based on the evaluations, and ranks the product according to predetermined ranks. The display control processing unit 243 then determines the color to be displayed in the area of ​​the product or the POP corresponding to the product, and the display output processing unit 244 displays the color on the display device 72 of the management terminal 2.

[0168] For a product whose selection has been accepted by the product selection acceptance processing unit 240, the effect display processing unit 24 accepts the recognition of the product through a recognition questionnaire or the like, converts the recognition into an index value, calculates a total index value for the product based on this, and ranks the product according to a predetermined rank. The display control processing unit 243 then determines the color to be displayed in the area of ​​the product or the POP corresponding to the product, and the display output processing unit 244 displays this on the display device 72 of the management terminal 2.

[0169] Furthermore, sales information for the product corresponding to the product identification information is obtained from a product sales management system such as a POS system based on at least two or more dates to be compared, the product identification information of the product selected and accepted by the product selection acceptance processing unit 240, store identification information, etc. Then, the sales increase rate is calculated. The increase rate is ranked among predetermined ranks. Then, the display control processing unit 243 determines the color to be displayed in the area of ​​the product or the POP corresponding to the product, and the display output processing unit 244 displays it on the display device 72 of the management terminal 2.

[0170] When the display output processing unit 244 displays the estimated effect of linking the POP with the product, customer feedback, recognition, sales, etc. in the area of ​​the product or the POP corresponding to the product, the display may be switchable for each item or may be displayed simultaneously.

[0171] For example, by performing a predetermined operation, colors corresponding to the ranks of each of the items, in the order of effectiveness estimation, customer feedback, recognition, and sales, may be displayed on the product or in an area of ​​a POP corresponding to the product.

[0172] Also, the product or the POP area corresponding to the product may be divided into items, and colors corresponding to the ranks of the items of effect estimation, customer opinions, recognition, and sales may be displayed simultaneously. For example, areas according to the number of items may be provided in the product area, and colors corresponding to the ranks of each item may be displayed simultaneously there. An example of this is shown in FIG. 32. FIG. 32(a) shows a case where rectangular areas corresponding to items are provided in the product area and the respective colors are displayed, and FIG. 32(b) shows a case where the product area is divided according to items and the respective colors are displayed in each divided area. Also, FIG. 32 shows an enlarged view of the area of ​​one product, and a similar display process may be performed in the POP area.

[0173] By executing the above-mentioned process, a plurality of items can be displayed at the same time. [Industrial Applicability]

[0174] By using the information processing system 1 of the present invention, it is possible to detect POPs from image data obtained by photographing a display shelf and estimate the effectiveness of the POP. [Explanation of symbols]

[0175] 1: Information processing system 2: Management terminal 3: Image data input terminal 20: POP detection processing unit 21: Data storage unit 22: Effect estimation processing unit 23: Linking processing unit 24: Effect display processing section 70: Arithmetic device 71:Storage device 72:Display device 73: Input device 74:Communication equipment 200: Image data input reception processing unit 201: Image data alignment processing unit 202: Object detection processing unit 203: Individual discrimination processing unit 204: Area detection processing unit 220: POP selection reception processing unit 221: Product selection reception processing unit 222: Analysis processing unit 223: Effect output processing section 230: POP determination processing unit 231: Compatible product determination processing unit 240: Product selection reception processing unit 241: POP specific processing unit 242: Index processing unit 243: Display control processing unit 244: Display output processing unit

Claims

1. An information processing system for displaying a POP effect, The information processing system includes: a POP detection processing unit that detects a POP from image data showing a display shelf; a linking processing unit that links the POP with a product corresponding to the POP; An effect display processing unit that displays an effect of the POP, The POP detection processing unit is determining a type of the object detected from the image data using an area of ​​the detected object and the type of the detected object, thereby determining whether the object is a POP or a candidate for a POP; The effect display processing unit: Identifying a POP to be processed, and calculating the effect of the POP by performing a calculation using an index value based on the type of the identified POP. An information processing system comprising:

2. The effect display processing unit: Identifying a POP corresponding to the product selected; Classifying the POPs into predetermined ranks by performing a calculation using index values ​​based on the identified POP types; 2. The information processing system according to claim 1 .

3. The effect display processing unit: displaying and outputting the product or a part or the whole of the area of ​​a POP corresponding to the product in a display method according to the rank; 3. The information processing system according to claim 2.

4. The effect display processing unit: The POP effect item and one or more items of customer feedback, recognition, or sales are classified into the ranks, displaying and outputting the product or a part or all of the area of ​​the POP corresponding to the product in a manner that allows switching between the items in a display method according to the rank; 4. The information processing system according to claim 3.

5. The effect display processing unit: The POP effect item and one or more items of customer feedback, recognition, or sales are classified into the ranks, displaying and outputting two or more items simultaneously in a part or all of an area of ​​the product or a POP corresponding to the product in a display method according to the rank; 5. The information processing system according to claim 4.

6. Computer, a POP detection processing unit that detects POPs from image data showing a display shelf; a linking processing unit for linking the POP with a product corresponding to the POP; An information processing program that causes an effect display processing unit to function as an effect display processing unit that displays the POP effect, The POP detection processing unit is determining a type of the object detected from the image data using an area of ​​the detected object and the type of the detected object, thereby determining whether the object is a POP or a candidate for a POP; The effect display processing unit: Identifying a POP corresponding to the product selected and calculating the effect of the POP by performing a calculation using an index value based on the type of the identified POP.

23. An information processing program comprising:

Images