Information processing system

Abstract

To provide an information processing system for executing processing related to the following results by detecting POPs attached to a display shelf installed in a shop and commodities displayed there.SOLUTION: An information processing system for estimating an effect of a POP includes a POP detection processing part for detecting a POP from image data showing a display shelf, a data storage part for storing various data to be used for processing, and an effect estimation processing part for estimating an effect of the POP. The POP detection processing part includes an object detection processing part for inputting the image data to a learning model that learns by using an object and its type, detecting an object shown in the image data, and outputting its type, and an individual discrimination processing part for discriminating that the detected type of the object is a POP or a candidate for the POP when the detected type of the object is a type other than a predetermined type. The effect estimation processing part includes an analysis processing for estimating an effect of the POP by using sales information of a commodity corresponding to the POP.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing system that detects POPs attached to display shelves installed in stores and products displayed thereon, and executes processing related to their effects.

Background Art

[0002] In various stores such as convenience stores and supermarkets, it is common to sell products by placing them on display shelves. Therefore, by displaying multiple products on the display shelf, even if one product is purchased, the same product can be purchased by others. And managing where and how many products are displayed on the display shelf is important in terms of product sales strategy.

[0003] In addition, POPs may be attached to display shelves or products for promoting the sales and advertising of the displayed products. POPs are advertisements, sales promotion items such as testers and sample products, used for selling the displayed products.

[0004] POPs are important items for product manufacturers and retailers for product sales. Therefore, for example, as disclosed in Patent Document 1 below, a system for printing POPs according to the product sales situation is disclosed.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] In the invention of Patent Document 1, since POP can be printed according to the sales situation, for example, POP can be printed and advertised for products with unfavorable sales situations.

[0007] On the other hand, since POP is an important item for product sales, it is desired to understand the effects of POP, such as what effects POP has and which products and positions POP should be attached to for effectiveness. However, in the above-mentioned Patent Document 1, such understanding cannot be achieved.

[0008] Therefore, Patent Document 2 discloses a system for identifying POP advertisements that affect the sales of a store.

[0009] However, in the patented invention of Patent Document 2, although there is a description of identifying that POP advertisements are included in the image data, there is no description of how to identify POP advertisements.

[0010] Although it is not easy to identify the products displayed from the image data of the display shelf, since POP is required to be eye-catching in some aspects, various shapes and designs may be adopted. Therefore, it is not easy to stably determine POP from the image, and in the above-mentioned Patent Document 2, there is no description at all on how to identify it, and it is not a realizable system.

[0011] In this way, in order to understand the effects of POP, POP must be automatically and stably detected from the image data of the display shelf, but there was no such system in the past. Also, it was not possible to estimate the effects of POP detected from the image data.

Means for Solving the Problem

[0012] In view of the above problems, the inventor has invented an information processing system that detects POP from the image data of the display shelf and executes processing related to the effects of POP.

[0013] The first invention is an information processing system for estimating the effect of POP. The information processing system includes a POP detection processing unit that detects POP from image data showing a display shelf, and an effect estimation processing unit that estimates the effect of the POP. The POP detection processing unit includes an object detection processing unit that inputs the image data into a learning model trained using an object and its type, detects the object shown in the image data, and outputs its type, and an individual determination processing unit that determines that the detected object type is POP or a candidate for POP when the detected object type is other than a predetermined type. The effect estimation processing unit is an information processing system that estimates the effect of the POP using the sales information of the product corresponding to the POP.

[0014] By executing the processing of the present invention, it is possible to detect POP from image data showing a display shelf and estimate the effect of the POP. To detect POP from image data, it can be done by executing the processing of the present invention.

[0015] In the above invention, the POP detection processing unit may include a region detection processing unit that detects a region of the image data, and the individual determination processing unit may be configured as an information processing system that determines the type of the detected object using the detected region and the type of the detected object.

[0016] When detecting POP, the type may be determined in consideration of the region where the object is detected.

[0017] In the above invention, the effect estimation processing unit may be configured as an information processing system including a POP selection reception processing unit that receives a selection of POP for which the effect is to be estimated, and an analysis processing unit that estimates the effect of the POP using the sales information of the product corresponding to the selected POP and the sales information of the product when the POP is not present.

[0018] In the above invention, the effect estimation processing unit can be configured as an information processing system having a POP selection reception processing unit that receives a selection of a POP for which an effect is to be estimated, a product selection reception processing unit that receives a selection of a product, obtains sales information when there is a POP for the product for which the selection has been received and sales information when there is no POP for the product for which the selection has been received from a predetermined sales management system, and an analysis processing unit that estimates the effect of the POP by comparing the sales information when there is a POP with the sales information when there is no POP.

[0019] By executing the processing of these inventions, the effect of the POP can be estimated.

[0020] In the above invention, the analysis processing unit can be configured as an information processing system that obtains sales information when there is a POP for the product for which the selection has been received and sales information when there is no POP for the product for which the selection has been received from a predetermined sales management system, and estimates the effect of the POP by comparing the sales information when there is a POP, the number of faces of the product, the sales information when there is no POP, and the number of faces of the product.

[0021] When estimating the effect of a POP as in the present invention, the estimation accuracy can be improved by taking into account the number of faces of the product in the estimation.

[0022] The information processing system of the first invention can be realized by causing a computer to read and execute the program of the present invention. That is, an information processing program that causes a computer to function as a POP detection processing unit that detects a POP from image data in which a display shelf appears, and an effect estimation processing unit that estimates the effect of the POP, wherein the POP detection processing unit inputs the image data into a learning model learned using an object and its type, detects an object appearing in the image data, and outputs an object detection processing unit of its type, and when the type of the detected object is a type other than a predetermined type, an individual determination processing unit that determines that the type of the detected object is a POP or a candidate for a POP, and the effect estimation processing unit estimates the effect of the POP using sales information of the corresponding product of the POP.

Effect of the Invention

[0023] By using the information processing system of the present invention, it becomes possible to detect a POP from image data obtained by photographing a display shelf and execute processing related to the effect of the POP.

Brief Description of the Drawings

[0024]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Mode for Carrying Out the Invention

[0025] An example of the overall processing function of the information processing system 1 of the present invention is shown in FIG. 1 as a block diagram, an example of the processing function of the POP detection processing unit 20 is shown in FIG. 2, and an example of the processing function of the effect estimation processing unit 22 is shown in FIG. 3. The information processing system 1 uses the management terminal 2 and the image data input terminal 3. The management terminal 2 is a computer used by an organization such as a company that operates the information processing system 1. Also, the image data input terminal 3 is a terminal for inputting image data obtained by photographing a store display shelf.

[0026] The management terminal 2 and the image data input terminal 3 in the information processing system 1 are realized using a computer. FIG. 4 schematically shows an example of the hardware configuration of the computer. The computer includes an arithmetic device 70 such as a CPU that executes arithmetic processing of programs, a storage device 71 such as a RAM or a hard disk that stores information, a display device 72 such as a display that displays information, an input device 73 such as a keyboard or a mouse through which information can be input, and a communication device 74 that transmits and receives the processing results of the arithmetic device 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN.

[0027] When the computer is equipped with a touch panel display, the display device 72 and the input device 73 may be integrally configured. The touch panel display is often used in portable communication terminals such as tablet computers and smartphones, but is not limited thereto.

[0028] The touch panel display is a device in which the functions of the display device 72 and the input device 73 are integrated in that input can be directly performed on the display with a predetermined input device (such as a pen for a touch panel) or a finger.

[0029] In addition to the above-described devices, the image data input terminal 3 may be provided with a photographing device such as a camera. As the image data input terminal 3, a portable communication terminal such as a mobile phone, a smartphone, or a tablet computer can also be used. The image data input terminal 3 photographs the display shelves in the store and inputs the image data (photographed image data) of the photographed display shelves to the management terminal 2. Products are displayed on the display shelves, and POPs are attached to the display shelves, their vicinity, or the products. Therefore, the image data shows displayed products such as products displayed on the display shelves, product tags, shelf members, and POPs. Preferably, it is preferable that unnecessary things such as people are not captured.

[0030] Each means in the present invention only has its functions logically distinguished, and may physically or de facto form the same area. The processing in each means of the present invention can also have its processing order appropriately changed. Also, a part of the processing may be omitted. For example, the orthorectification processing described later can also be omitted. In that case, processing can be executed on the image data without orthorectification processing.

[0031] The information processing system 1 includes a POP detection processing unit 20, a data storage unit 21, and an effect estimation processing unit 22.

[0032] The POP detection processing unit 20 executes processing to detect a POP or its candidate shown in the captured image data or the image data (orthorectified image data) after the orthorectification processing described later based on the captured image data. The data of the image to be processed, including the captured image data and the orthorectified image data, is collectively referred to simply as "image data".

[0033] The data storage unit 21 stores, in association with each other, the image data received from the image data input terminal 3, the shooting date and time, the store identification information, the display shelf identification information for identifying the display shelf, the image data identification information, and the like. The image data may be any image data that is the processing target of the present invention. Generally, when simply shooting, since it is difficult to shoot the object to be photographed in a state facing it directly, it is advisable to execute correction processing, such as trapezoidal correction processing, to correct it to a state facing it directly. When shooting a plurality of images of one display shelf, the image data synthesized as one image data is also included. Also, the image data after the distortion correction processing is also included in the image data. Further, the data storage unit 21 stores various data (including information) used in the processing of the information processing system 1 of the present invention.

[0034] The effect estimation processing unit 22 estimates the effect of the POP using the POP detected by the POP detection processing.

[0035] The POP detection processing unit 20 includes an image data input reception processing unit 200, an image data alignment processing unit 201, an object detection processing unit 202, and an individual discrimination processing unit 203.

[0036] The image data input reception processing unit 200 receives the input of image data (captured image data) of the display shelves in the store captured by the image data input terminal 3 and stores it in the data storage unit 21. From the image data input terminal 3, in addition to the captured image data, store identification information such as the capture date and time, store name, shelf identification information for identifying the display shelf, and image data identification information for identifying the captured image data may also be received for input. An example of the captured image data is shown in FIG. 8. Note that FIG. 8 shows a case where one display shelf is shown in the captured image data, but a plurality of display shelves may be shown, or even a part of the display shelf may be shown. Also, in the present invention, although the processing is not particularly specified, the display shelf and the shelf tiers may be long in the horizontal direction. Therefore, in that processing, it may be divided at a certain width, and the divided range may be used as the processing target for each process.

[0037] The image data alignment processing unit 201 generates aligned image data by performing a process (alignment process), for example, a trapezoidal correction process, to correct the captured image data stored in the data storage unit 21 so that the object to be captured is in a facing-forward state. The trapezoidal correction process is a correction process that makes the shelf tiers of the display shelf shown in the captured image data horizontal and makes the products displayed thereon and the product tags (for example, price tags) for those products vertical. Alignment means deforming the captured image data so that it becomes the same as when shooting from a sufficiently far distance along the perpendicular direction of the optical axis of the lens of the shooting device to the plane of the object to be shot. For example, there is a trapezoidal correction process. The aligned image data (aligned image data) may be stored in the above-mentioned data storage unit 21.

[0038] The trapezoid correction process executed by the image data normalization processing unit 201 receives the input of the designation of four vertices in the captured image data, and executes the trapezoid correction process using each of the vertices. The four vertices for receiving the designation may be the four vertices of the shelf level of the display shelf, or may be the four vertices of the shelf position of the display shelf. Further, they may be the four vertices of a group of two-tier or three-tier shelf levels. Any four points can be designated as the four vertices. The image data normalization processing unit 201 can use various known processing methods.

[0039] The object detection processing unit 202 executes a process of detecting an object shown in the image data on the captured image data stored in the data storage unit 21 or the image data normalized by the image data normalization processing unit 201. As the object detection process, it is preferable to perform detection using so-called deep learning. In this case, for a learning model in which the weighting coefficients between neurons in each layer of a neural network having a large number of intermediate layers are optimized, the above image data is input, and the type (class) of the object shown in the input image data is output as an output value.

[0040] As the learning model, products, product tags (price tags), shelf members, etc. that may appear in the image data are used as annotation data, and the above learning model is learned by performing predetermined learning processing using the annotation data. For example, data associating product image data with the fact that it is a product or its product name or product code (such as JAN code) is used as annotation data to execute learning processing for the learning model. The image data is the image data in the annotation data, and the class indicating that it is a product, or that it is a product and product identification information such as the product name or product code is a "product category". Similarly, the image data of the product tag is the image data in the annotation data, and the class indicating that it is a product tag, or that it is a product tag and its product name or product identification information is a "product tag category". Furthermore, the image data of the shelf member is the image data in the annotation data, and the class indicating that it is a shelf member or the part name of the shelf member is a "shelf member". Note that as the type (class) of the object used as annotation data by the object detection processing unit 202, there are objects that may generally be displayed on a display shelf, such as product categories such as products, product tag categories such as product tags, and shelf members, but it is not limited to these, and it may be learned to detect any object.

[0041] The individual discrimination processing unit 203 discriminates that, among the types (classes) of objects detected from the image data by the object detection processing unit 202, the types of objects detected as types other than product categories, product tag categories, and shelf members are POPs or candidates for POPs. Also, for objects detected as types (classes) of product categories, product tag categories, and shelf members, if they meet predetermined conditions, their types (classes) are discriminated as POPs or candidates for POPs.

[0042] For example, for an object whose type is detected as a product category, it is determined whether a product-attached advertisement (POP) is attached to the area. If it is determined that there is a POP, the type of the object in that area is determined as a POP or a POP candidate. In this case, there are a method of detecting a POP attached to a displayed product from the area of the object whose type is detected as a product category, and a method of detecting a POP by comparing the image data of the area of the object whose type is detected as a product category with the image data of a product without a POP.

[0043] In the case of the former method, first, a large number of image data including the POP part attached to the displayed product are collected in advance, and each image data including the POP part is annotated with the POP part. Then, for a learning model that has been learned in advance using the image data of a normal displayed product without a POP as annotation data, additional learning is performed using the image data annotated with the above POP part to generate a new learning model. For the newly generated learning model, the image data of the area of the object whose type is detected as a product category is input. In addition to outputting product identification information during product recognition processing, it is also possible to additionally output whether a POP is attached to the displayed product in the image data.

[0044] In the case of the latter method, after performing product recognition processing on the image data of the area of the object detected as a product category, the standard image data (for example, specimen data) without a POP of the recognized product is compared with the image data of the area of the object, and the similarity is compared. If the similarity is equal to or greater than a predetermined value, it is determined that there is no POP for the object detected as a product category. If the similarity is less than the predetermined value, it is determined that there is a POP or a POP candidate for the object detected as a product category.

[0045] In the above method, in the comparison between the image data of the region of the object detected as goods and the standard image data of the goods recognized in the goods recognition process without POP, by determining whether it is POP based on the position of the different region, it is possible to make a more accurate determination. For example, when it is determined that the different region in the two image data is localized in a specific position, for example, near the upper part, shoulder, or tail of the goods, it may be determined that it is POP or a candidate for POP.

[0046] As the goods recognition process, various known processing methods can be used. For example, a process of identifying goods identification information can be executed using deep learning (deep neural network) for the image data of the region of the object. In this case, for a learning model in which the weighting coefficients between neurons in each layer of a neural network with many intermediate layers are optimized, the image data of the region of the object can be input, and based on the output value, the goods identification information can be identified. As the learning model, one in which goods identification information is given as correct answer data for various image data in which goods are shown can be used. In addition to this, sample data of goods (image data obtained by photographing the goods from one or more directions or data such as feature amounts based thereon) and the image data of the region of the object can be compared, and the goods identification information can be identified using the similarity. The goods recognition process is not limited to these.

[0047] Also, for an object whose type is detected as goods tags, it is determined whether the image data of the region of the object satisfies the requirements of the goods tags. For example, OCR recognition processing is performed on the region of the object whose type is detected as goods tags. If characters usually included as goods tags, such as price and product name, are included, it is determined as goods tags. Otherwise, it is determined as POP or a candidate for POP. Alternatively, as a result of the OCR recognition processing, if a predetermined word, such as "member discount", "tester", "sample", "specimen", etc., a word usually used for POP is included, it may be determined as POP or a candidate for POP. Also, when handwritten characters are included, it may be determined as POP or a candidate for POP.

[0048] In addition, for an object detected as a shelf member, it is determined whether the image data of the region of the object satisfies the requirements of the shelf member. For example, OCR recognition processing is performed on the region of the object detected as a shelf member. If a predetermined word, such as a membership discount, a word commonly used in POP, is included, it is determined as POP or a candidate for POP. Otherwise, it is determined as a shelf member.

[0049] When the individual discrimination processing unit 203 determines that the type (class) of the image information of the region of the object is a candidate for POP, the image information of the region of the object is displayed on the display device 72 of the management terminal 2 or the like, and an input as to whether it is POP is received. If the input is POP, the type (class) of the image information of the region of the object may be set as POP.

[0050] When the individual discrimination processing unit 203 determines the region of POP, POP identification information for identifying the POP is assigned and stored in the data storage unit 21 in association with the image data of the region of POP. Further, it is preferable that the shooting date and time of the image data in which the POP appears, the store identification information, and the display shelf identification information are also associated.

[0051] The effect estimation processing unit 22 includes a POP selection reception processing unit 220, a product selection reception processing unit 221, an analysis processing unit 222, and an effect output processing unit 223.

[0052] The POP selection reception processing unit 220 extracts the image data of the region of POP detected by the POP detection processing unit 20 from the data storage unit 21 and receives the selection of the POP to be the target of effect estimation.

[0053] The product selection reception processing unit 221 extracts the product identified from the image data from the data storage unit 21 and receives the selection of the product to be the target of effect estimation. For the processing of identifying the product from the image data and storing it in the data storage unit 21, various known techniques can be used. Also, the image data of the product may be stored in the data storage unit 21 in advance from the image data.

[0054] For example, the area of a product is identified by identifying a thin and narrow shadow generated between products from image data, identifying a repeating pattern of an image, identifying a step on the upper side of a package, identifying a cutting position based on constraints such as the same product width, etc. Then, the image information of the product area is subjected to image matching processing with the image data of a reference product (specimen data) stored in advance, and the identification information (predetermined codes such as product name, JAN code, etc.) of the product with the highest similarity in the specimen data is identified as the product identification information of the product in the area. Through such processing, the product shown in the image data can be identified, and the image data of the product and the product identification information can be associated and stored in the data storage unit 21.

[0055] As another method, machine learning such as deep learning may be used to identify the area of a product and the product identification information. In this case, for a learning model in which the weighting coefficients between neurons in each layer of a neural network with many intermediate layers are optimized, an area to be processed, for example, image data, is input, and based on the output value, the area of the product and the identification of the product identification information may be performed. As the learning model, one in which the area of the product and its product identification information are given as correct data for image data of various areas to be processed, for example, an image of a display shelf or a shelf section area, can be used.

[0056] The product identification process may be performed by an object detection processing unit 202, an individual discrimination processing unit 203, etc. The image data of the identified product area and the product identification information are associated and stored in the data storage unit 21. Also, it is preferable to associate the shooting date and time of the image data showing the product, the store identification information, and the display shelf identification information.

[0057] Figure 9 shows an example of a screen for receiving the selection of a POP by the POP selection reception processing unit 220 and the selection of a product by the product selection reception processing unit 221. The effect estimation processing unit 22 receives the selection of some or all of the store, display shelf, date, etc. for which the effect of the POP is to be estimated in a predetermined method. When receiving the selection, the effect estimation processing unit 22 extracts the corresponding POP image data and product image data from the data storage unit 21 based on the store identification information of the selected store, the display shelf identification information of the selected display shelf, and the shooting date corresponding to the selected date, and displays them as a list on the screen shown in FIG. 9 above.

[0058] The analysis processing unit 222 executes a process for estimating the effect of the POP. For example, when the "Analysis" button on the screen shown in FIG. 9 is pressed, the estimation processing unit receives the POP selected by the POP selection reception processing unit 220 and the product selected by the product selection reception processing unit 221, and extracts from the data storage unit 21 the POP identification information corresponding to the image data of the selected POP, the shooting date and time of the image data in which the POP appears, the store identification information, and the display shelf identification information. In addition, the product identification information corresponding to the image data of the selected product, the product identification information corresponding to the image data in which the product appears, the shooting date and time of the image data in which the product appears, the store identification information, and the display shelf identification information are extracted from the data storage unit 21.

[0059] Then, based on the date received for selection by the effect estimation processing unit 22, the store identification information, and the product identification information, the analysis processing unit 222 acquires sales information of the product having the product identification information at the store on the date from a product sales management system such as a POS system (for example, various types of information related to product sales, such as the number of sales, sales amount, number of sold items, reservation amount, number of reservations, etc.). Note that this date is not limited to the first day of the date, and may include a plurality of dates before and after the date. That is, it is sufficient to be able to acquire the sales information of the product having the product identification information at the store on the date when the POP was installed. Also, an input of a date when the POP was not installed is received, and the sales information of the product having the product identification information is acquired. Note that the number of days for each of the date when the POP was installed and the date when the POP was not installed may be the same for comparison. It may be one day, one week, or any appropriate number of days.

[0060] The analysis processing unit 222 compares the sales information of the product on the date with POP as described above and the sales information of the product on the date without POP, and calculates the contribution degree of the POP (an index indicating the improvement in the number of product sales due to the POP, for example, the increase rate). For example, (Number of product sales on the day with POP)÷(Number of product sales on the day without POP) is calculated to obtain the contribution degree.

[0061] As another arithmetic expression, the contribution degree of the POP may be calculated taking into account the number of faces (the number of image data of the product areas). The number of faces is the number of products displayed on the display shelf shown in the image data and can be calculated for each product. Since the object detection processing unit 202 detects the image data of the areas having the types of products, those areas are regarded as faces. Then, in the above-described product identification process, in the image data identical to the image data in which the selected product is shown, the number of image data of the product areas having the same identification information as the product identification information of the selected product is specified. This number is the number of faces. The information on the number of faces may be stored in the data storage unit 21 in association with the image data, product identification information, etc. Therefore, the number of faces of the product on the display shelf on the date when the selected POP is installed and the number of faces of the same product on the same display shelf (the display shelf with the same display shelf identification information) on the date when the POP is not installed are extracted from the data storage unit 21.

[0062] And the analysis processing unit 222, for example, (Sales volume of the product on the day with POP / Number of faces of the product on the day with POP) ÷ (Sales volume of the product on the day without POP / Number of faces of the product on the day without POP) is calculated to obtain the contribution degree.

[0063] The arithmetic expression for calculating the contribution degree is not limited to the above. As long as comparison using sales information such as the sales volume on the day with POP and the sales volume on the day without POP is possible, any arithmetic expression may be used for the calculation.

[0064] Furthermore, when there are multiple POPs, the contribution degree considering the influence of multiple POPs may be calculated. For example, when two POPs (POPa, POPb) are selected, there are combinations of the presence or absence of each POP shown in FIG. 10. Assuming that the number of faces in all four cases is the same according to the presence or absence of POPa and POPb, the number of faces in each case is normalized, and the contribution degree of POPa can be calculated by calculating C÷D in FIG. 10, and the contribution degree of POPb can be calculated by B÷D. Also, since the contribution degree when there are both POPa and POPb can be calculated by A÷D, that value may be used for correcting the contribution degrees of POPa and POPb. Even when the number of POPs is three or more, similarly, assuming that the number of faces in each case is the same for each case, the number of faces in each case is normalized and calculated in the same way.

[0065] The effect output processing unit 223 outputs the contribution degree calculated by the analysis processing unit 222. For example, FIG. 11 shows an example of outputting and displaying the contribution degree of POPs for each product.

[0066] As described above, it is possible to detect POPs detected from image data and estimate the effect of POPs on product sales.

Example

[0067] Next, an example of a processing process using the information processing system 1 of the present invention will be described with reference to the flowcharts of FIGS. 5 to 7.

[0068] First, the process of detecting POPs included in image data (S100) will be described with reference to the flowcharts of FIGS. 5 and 6.

[0069] The image data captured of the store's display shelves is input from the image data input terminal 3 and received by the image data input reception processing unit 200 of the management terminal 2 (S200). Also, the input of the shooting date and time, store identification information, display shelf identification information, and image data identification information of the image data is received. Then, the image data input reception processing unit 200 stores the received image data, shooting date and time, store identification information, display shelf identification information, and image data identification information of the image data in the data storage unit 21 in association with each other. Note that the store identification information, display shelf identification information, and image data identification information may be automatically assigned.

[0070] When the management terminal 2 receives a predetermined operation input, the image data normalization processing unit 201 extracts the image data stored in the data storage unit 21, receives the input of four points of the shelf position (the position of the display shelf), which is the vertex for performing normalization processing such as trapezoidal correction processing, and executes the normalization processing (S210).

[0071] Then, by receiving a predetermined operation input on the management terminal 2, the object detection processing unit 202 executes object detection processing on the normalized image data (S220) and classifies the types of objects shown in the image data (S230). An example of the result of executing object detection processing from the image data is schematically shown in FIG. 12. In FIG. 12, "a" means merchandise, "b" means product tags, "c" means shelf members, and "d" means the types of objects detected as other than a to c. Note that the types of objects detected from the image data only need to be associated with their respective regions and do not need to be written on each image. Also, the regions of the detected objects may be cut out from the image data, but they do not necessarily need to be actually cut out as long as the image data of the regions is in a processable state. Note that cutting out the image data of the region includes not only actually cutting out the image data of the region from the image data but also specifying it as a processable state without cutting out the image data of the region.

[0072] The individual discrimination processing unit 203 discriminates and extracts, as the POP area, the areas of objects of types other than merchandise, product tags, and shelf members (the areas of the objects of "d") in the object areas of the image data detected by the object detection processing unit 202 (S240, S250).

[0073] In addition, the individual discrimination processing unit 203 discriminates whether a predetermined condition is satisfied for the areas detected as merchandise among the object areas of the image data detected by the object detection processing unit 202 (S240), and discriminates whether it is the POP area (S260).

[0074] For example, for the area of an object detected as merchandise, it is discriminated whether a product-adjacent type of advertisement (POP) is attached to that area. If it is discriminated that there is a POP, the type of that area is discriminated as POP. For the area of an object detected as merchandise, a process of detecting the POP attached to the displayed merchandise is executed, or by comparing the image data of the area of the object detected as merchandise with the image data of a product without a POP (product specimen image data), the POP is detected.

[0075] Similarly, for the area of an object detected as product tags, it is discriminated whether the image data of that object area satisfies the requirements of product tags. Also, for an object detected as a shelf member, it is discriminated whether the image data of that object area satisfies the requirements of the shelf member.

[0076] As described above, for the areas of objects detected as any of merchandise, product tags, and shelf members (S240), it is discriminated whether the predetermined conditions for each object are satisfied (S260). If the conditions are satisfied, all or part of that area is discriminated as POP (S270). An example of this process is shown in FIG. 13. Also, an example of each area after discriminating, as the POP area, the areas that satisfy the predetermined conditions in the areas of objects detected as any of merchandise, product tags, and shelf members with respect to the object detection result in FIG. 12 is shown in FIG. 14.

[0077] By executing the above-described processing in the individual discrimination processing unit 203, it is possible to detect the POP area in the image data.

[0078] When the POP detection processing unit 20 detects the image data of the POP area as described above, POP identification information is assigned and stored in the data storage unit 21 in association with the image data of the POP area. At this time, the shooting date and time of the image data in which the POP was detected, the store identification information, the display shelf identification information, and the image data identification information may also be stored in association. In addition, for the image data other than the image data of the POP area, the shooting date and time of the detected image data, the store identification information, the display shelf identification information, and the image data identification information may be stored in association. Further, for the image data of the area of the object detected as products and product tags, information such as the product identification information of the product shown in the image data of the area, the product identification information shown on the product tag, the price, and the manufacturer may be detected using known techniques and further stored in the image data storage unit 21 in association.

[0079] When the POP detection processing unit 20 detects a POP, the effect estimation processing unit 22 executes POP effect estimation processing using the POP detected by the POP detection processing unit 20 (S110).

[0080] First, the effect estimation processing unit 22 receives a selection of a store, a display shelf, and a date for which POP effect estimation is to be performed in a predetermined method. When receiving the selection, the effect estimation processing unit 22 extracts the corresponding POP image data and product image data from the data storage unit 21 based on the store identification information of the selected store, the display shelf identification information of the selected display shelf, and the shooting date corresponding to the selected date, and displays them as a list on a screen as shown in FIG. 9.

[0081] The POP selection reception processing unit 220 receives a selection of the POP for which effect estimation is to be performed from this screen (S300), and the product selection reception processing unit 221 receives a selection of a product (S310).

[0082] Based on the POP selected by the POP selection reception processing unit 220, the analysis processing unit 222 extracts POP identification information corresponding to the POP from the data storage unit 21, and based on the product selected by the product selection reception processing unit 221, extracts product identification information corresponding to the product from the data storage unit 21.

[0083] Then, based on the date information, product identification information, store identification information, etc. selected by the effect estimation processing unit 22, the analysis processing unit 222 obtains the sales information of the product corresponding to the product identification information from a product sales management system such as a POS system (S320). If the selection of the date, store identification information, and display shelf identification information is not received in the effect estimation processing, the date, store identification information, and display shelf identification information corresponding to the POP selected by the POP selection reception processing unit 220 and the product selected by the product selection reception processing unit 221 may be extracted.

[0084] The analysis processing unit 222 compares the sales information of the product on the date with POP as described above with the sales information of the product on the date without POP, and calculates the contribution degree of the POP (an index indicating the improvement in the sales volume of the product due to the POP, for example, the increase rate) (S330). For example, (Sales volume of the product on the day with POP)÷(Sales volume of the product on the day without POP) is calculated to obtain the contribution degree.

[0085] The calculated contribution degree is output by the effect output processing unit 223 (S340).

[0086] By checking the output result output by the effect output processing unit 223 through the above processing, for example, the output result in FIG. 11, etc., the estimated effect of the POP can be recognized.

Example

[0087] Next, another example of the detection process in the POP detection processing unit 20 will be described. In the detection processing unit of this example, in addition to the configuration of the first example, it further has a region detection processing unit 204. An example of the configuration of the POP detection processing unit 20 in this example is shown in FIG. 15, and an example of the POP detection process is shown in the flowchart of FIG. 16. Also, an example of the outline of the process is shown in FIG. 17, and an example of the individual discrimination process is shown in FIG. 18.

[0088] The region detection processing unit 204 detects a product display region, which is a region for displaying products, a product tag region, which is a region where product tags are placed, and an upper shelf region, which is a region above the shelf, from the image data stored in the data storage unit 21 (including the normalized image data). As the detection of the product display region, the product tag region, and the upper shelf region, the operator of the management terminal 2 may manually specify the product display region, the product tag region, and the upper shelf region, and the region detection processing unit 204 may accept it. Alternatively, based on the information of the product display region, the product tag region, and the upper shelf region that were manually input for the first time, the product display region, the product tag region, and the upper shelf region may be automatically detected after the second time. Also, the region detection processing unit 204 may detect other regions and use them as processing targets.

[0089] FIG. 19 schematically shows a state in which an input of the designation of the product display region, the product tag region, and the upper shelf region is accepted for the normalized image data obtained by photographing the display shelf on which products are displayed.

[0090] Note that when detecting the product display region, the product tag region, and the upper shelf region, the region detection processing unit 204 may use deep learning (deep neural network) to identify the product display region, the product tag region, and the upper shelf region. In this case, the normalized image data is input to a learning model in which the weighting coefficients between the neurons of each layer of a neural network having a large number of intermediate layers are optimized, and based on the output value, the product display region, the product tag region, and the upper shelf region may be detected. Also, as the learning model, one in which the product display region, the product tag region, and the upper shelf region are given as correct data to various image data can be used.

[0091] In addition to using deep learning as described above, the area detection processing unit 204 may detect an area using various known methods.

[0092] For the detected area, the area detection processing unit 204 may actually cut it out as image data, or may virtually cut it out by specifying the range of the area without actually cutting it out as image data. In this case, the range of the area can be specified by coordinates.

[0093] The object detection processing unit 202 in this embodiment may perform object detection processing from the range of the image data as in the first embodiment, or may perform it for each area detected by the area detection processing unit 204.

[0094] The individual discrimination processing unit 203 in this embodiment discriminates the type of each object detected by the object detection processing unit 202 using the type (class) of the object and the area where the object is located. For example, discrimination is performed according to predetermined individual discrimination conditions such as the discrimination tables shown in FIGS. 17 and 18.

[0095] "a1" to "a4", "b1" to "b10", "d1" to "d5" in FIG. 17 correspond to "a1" to "a4", "b1" to "b10", "d1" to "d5" in FIG. 18.

[0096] Note that FIGS. 17 and 18 are examples, and the type of the object is also an example thereof. Also, it is an example of the type of POP, and is not limited thereto and can be arbitrarily set.

[0097] Next, an example of the POP detection processing in the POP detection processing unit 20 of this embodiment will be described using the flowchart of FIG. 16.

[0098] The image data of the store display shelf is input from the image data input terminal 3 and received by the image data input reception processing unit 200 of the management terminal 2 (S400). In addition, the input of the shooting date and time, store identification information, display shelf identification information, and image data identification information of the image data is received. Then, the image data input reception processing unit 200 stores the received image data, shooting date and time, store identification information, display shelf identification information, and image data identification information in the data storage unit 21 in association with each other.

[0099] When the management terminal 2 receives a predetermined operation input, the image data rectification processing unit 201 extracts the image data stored in the data storage unit 21, receives the input of four points of the shelf position (the position of the display shelf), which is the vertex for performing rectification processing such as trapezoidal correction processing, and executes the rectification processing (S410).

[0100] Then, by receiving a predetermined operation input on the management terminal 2, the object detection processing unit 202 performs an area detection process on the rectified image data, detects a product display area, a product tag area, and an upper shelf area (S420), and determines the range of each area.

[0101] Also, by receiving a predetermined operation input on the management terminal 2, the object detection processing unit 202 executes an object detection process on the rectified image data (S430) and classifies the type of the object shown in the image data (S440). An example of the result of executing the object detection process from the image data is classified as shown in FIG. 12 as in the first embodiment.

[0102] Then, the individual discrimination processing unit 203 classifies the detected objects according to the predetermined discrimination conditions using the type of each object in the image data detected by the object detection processing unit 202 and the area where the object is located (the area determined in S420).

[0103] In addition, the individual discrimination processing unit 203 determines whether a predetermined condition is satisfied (S450) using the type of the object in the image data detected by the object detection processing unit 202 and the region where the object is located, and determines whether the type of the object is a POP or other than a POP (merchandise, product tags, shelf members, etc.) (S460, S470). Note that for which region each object is located in, for example, it may be determined by an appropriate method such as which region of the merchandise, product tags, or upper part of the shelf the center of gravity of the region of the object is included in.

[0104] For example, for an object in the product display area and the type of the object is merchandise, the individual discrimination processing unit 203 determines whether there is a product-adhering type of POP for the region of the object. For example, similar to Example 1, the discrimination is made by a method of detecting the POP attached to the displayed product from the region of the object detected as merchandise and a method of detecting the POP by comparing the image data of the region of the object detected as merchandise with the image data of the product without a POP. If a product-adhering type of POP can be detected, the region is discriminated as a POP (a2), and the other regions are discriminated as merchandise (a1). Also, if a product-adhering type of POP cannot be detected, OCR recognition processing is executed for that region to determine whether a predetermined word used for POPs such as "tester", "sample", "specimen" is included, and if not, the object in that region is discriminated as merchandise (a1). If it is included, the object in that region is discriminated as a POP such as a tester or a sample (a3).

[0105] In addition, for an object in the product display area and the type of the object is product tags, the individual discrimination processing unit 203 discriminates that the object in that region is a POP (d1).

[0106] In addition, for an object in the product display area and the type of the object is a shelf member, the individual discrimination processing unit 203 discriminates that the object in that region is a shelf member (c).

[0107] Also, for an object in the product display area, if the type of the object is other than products, product tags, and shelf members, it is determined to be a POP (d2).

[0108] For an object in the product tag area, if the type of the object is a product, the individual discrimination processing unit 203 determines that it is a POP such as a tester or a sample (d4).

[0109] Also, for an object in the product tag area, if the type of the object is a product tag, OCR recognition processing is performed on the area of the object. If it is determined that there is handwriting in the area, it is determined to be a POP (retail POP) created by the store (retail) (b4). If, as a result of the OCR recognition processing on the area of the object, it is determined that there is no handwriting in the area, and it is determined that the display includes the price and product name, and the word indicating a condition such as "member" is included, the type of the object is determined to be a product tag indicating a set product display (b2). On the other hand, if it is determined that the display includes the price and product name, and the word indicating a condition such as "member" is not included, the type of the object is determined to be a product tag (b1). If it is determined that the above display does not include the price and product name, and there is a point display, the type of the object is determined to be a POP of "point display" (b3). Further, if there is no point display in the above, it is determined to be a POP (manufacturer POP) created by the product manufacturer (b9). Here, the case of distinguishing and determining between the manufacturer POP and the retail POP is shown, but it may also be simply determined as a POP.

[0110] Also, for an object in the product tag area, if the type of the object is a shelf member, the individual discrimination processing unit 203 determines that the object in the area is a shelf member (c).

[0111] Also, for an object in the product tag area, if the type of the object is other than products, product tags, and shelf members, OCR recognition processing is executed for that area. Then, it is determined whether a predetermined word used in POP such as "tester", "sample", or "specimen" is included. If it is included, it is determined that it is a POP such as a tester or a sample (d4). On the other hand, if a predetermined word used in POP such as "tester", "sample", or "specimen" is not included, it is determined that it is a POP such as a promotional item (d3).

[0112] For an object in the upper shelf area, if the type of the object is products, the individual discrimination processing unit 203 determines that it is a stocked product (a4).

[0113] Also, for an object in the upper shelf area, if the type of the object is product tags, OCR recognition processing is performed on the area of the object. If it is determined that there is handwriting in that area, it is determined that it is a retail POP (b8). If, as a result of the OCR recognition processing on the area of the object, it is determined that there is no handwriting in that area, and it is determined that the display of the price and product name is included, and a word indicating a condition such as "member" is included, the type of the object is determined to be a product tag indicating a set product display (b6). On the other hand, if it is determined that the display of the price and product name is included, and a word indicating a condition such as "member" is not included, the type of the object is determined to be a product tag (b5). If it is determined that the above display of the price and product name is not included, and there is a display of points, the type of the object is determined to be a POP with a "point display" (b7). Furthermore, if there is no display of points in the above case, it is determined that it is a manufacturer POP (b10).

[0114] Also, for an object in the upper shelf area, if the type of the object is a shelf member, the individual discrimination processing unit 203 determines that the object in that area is a shelf member (c).

[0115] Also, for an object in the upper shelf area, if the type of the object is other than merchandise, merchandise tags, and shelf members, it is determined to be a POP such as a large promotional item (d5).

[0116] As described above, the individual discrimination processing unit 203 uses the area detected by the area detection processing unit 204 and the type of the object detected by the object detection processing unit 202 to determine whether the object is a POP or not, and can detect the area of the POP in the image data.

Example

[0117] As a modification of the processing of the first and second embodiments, the processing shown in the flowchart of FIG. 20 may be performed. In this case, when the type of the object detected by the object detection processing unit 202 is other than merchandise, merchandise tags, and shelf members, the individual discrimination processing unit 203 determines that the object is a POP (S560). Then, for other objects, the same processing as the processing from S450 to S470 may be executed. That is, using the type of the object in the image data detected by the object detection processing unit 202 and the area where the object is located, it is determined whether a predetermined condition is satisfied (S570), and it is determined whether the object is a POP or other than a POP (merchandise, merchandise tags, shelf members, etc.) (S580, S590). Note that the processing from S500 to S540 may be the same as the processing from S400 to S440.

Example

[0118] Using the processing of the second and third embodiments, a POP and a product may be associated with each other to estimate the effect of the POP.

[0119] The information processing system 1 of the present embodiment includes a POP detection processing unit 20, a data storage unit 21, an association processing unit 23, and an effect display processing unit 24. An example of the configuration of the information processing system 1 in this case is shown in FIG. 21, an example of the configuration of the association processing unit 23 is shown in FIG. 22, and the configuration of the effect display processing unit 24 is shown in FIG. 23. The processing of the POP detection processing unit 20 may be the same as that of the second or third embodiment.

[0120] The linking processing unit 23 links a POP with the product corresponding to that POP (the product advertised by the POP). The linking processing unit 23 includes a POP determination processing unit 230 and a corresponding product determination processing unit 231.

[0121] The POP determination processing unit 230 determines the character information displayed in the area of the POP. The POP determination processing unit 230 extracts the image data of the area of the POP detected by the POP detection processing unit 20 from the data storage unit 21, and determines the character information of the image data displayed in that area by a known method such as OCR recognition processing or deep learning.

[0122] For example, the POP determination processing unit 230 binarizes the image data of the area of the POP and executes OCR recognition processing on the binarized image data. Additionally, if necessary, determination processing such as collation with a dictionary of product identification information stored in advance may be performed.

[0123] As another process of the POP determination processing unit 230, machine learning such as deep learning may be used to determine the character information of the image data in the area of the POP. In this case, for a learning model in which the weighting coefficients between the neurons of each layer of a neural network with a large number of intermediate layers are optimized, the image data of the area of the POP is input, and based on the output value, the character information contained in the image data of that area may be determined. Also, as the learning model, one given with various image data of the area of the POP and the character information contained in that area as correct answer data can be used.

[0124] As yet another process of the POP determination processing unit 230, instead of using character information, the image data of the product contained therein may be determined. For example, it may be determined by image matching processing or deep learning (deep learning) whether the image data of the product stored in advance is included in the image data displayed in the area of the POP.

[0125] The corresponding product determination processing unit 231 determines the products corresponding to the POP. That is, it identifies the products that the POP is trying to advertise and associates the POP with those products. The corresponding product determination processing unit 231 uses part or all of the character information displayed in the area of the POP determined by the POP determination processing unit 230 to associate it with the products displayed on the display shelf where the POP is located.

[0126] As the process of associating the POP with the products, for example, the following process can be performed. First, the store identification information, display shelf identification information, and date information associated with the image data of the area of the POP are extracted from the data storage unit 21. Then, among the product objects detected by the object detection processing unit 202 and stored in the data storage unit 21, products having the same store identification information, display shelf, and date information, and whose product identification information matches part or all of the character information determined by the POP determination processing unit 230, particularly the product identification information indicating the products described on the POP, are identified.

[0127] In this way, the POP can be associated with the products. At this time, as the association, the POP identification information of the POP and the product identification information of the products may be associated.

[0128] Note that in order to identify the products corresponding to the POP, instead of processing the entire display shelf, the range for searching for the corresponding products may be changed using the type of the POP determined by the individual discrimination processing unit 203 (FIGS. 17 and 18).

[0129] For example, for a product corresponding to the POP of "a2" in Fig. 18, since it is the POP attached to the product, the product may be searched within the range adjacent to the POP. Also, for a product corresponding to the POP of "a3" in Fig. 18, since it is the POP of "sample / tester", the product in the vicinity of the POP, for example, within the range of 2 or 3 faces from the range where the POP is located as the product, may be searched. Also, for a product corresponding to the POP of "b3", "b4", or "b9" in Fig. 18, since it is the POP located in the product tag area, it is normal for the product to be located in the vicinity of the POP. Therefore, the product within the range of 2 or 3 faces from the range where the POP is located may be searched. Since "b7", "b8", "b10", and "d5" in Fig. 18 are POPs provided at the upper part of the display shelf, the entire product displayed on the display shelf becomes the search range. Since "d1" and "d2" in Fig. 18 are POPs installed in the product display area, and the product is located in the vicinity thereof, the product in the vicinity of the POP, for example, within the range of 2 or 3 faces from the range where the POP is located as the product, may be searched. Furthermore, since "d3" in Fig. 18 is the POP in the product tag area and "d4" is the POP of "sample / tester" in the product tag area, the product in the vicinity of the POP, for example, within the range of 2 or 3 faces from the range where the POP is located as the product, may be searched.

[0130] Regarding the positions of the POP and the product, when the object detection processing unit 202 detects an object, the position may be stored in the image data as a face (a certain image area) in terms of the coordinate, order, and position relationship in the image data.

[0131] In addition, when the corresponding product determination processing unit 231 determines the product corresponding to the POP, if the POP determination processing unit 230 determines using the image data, the product identification information corresponding to the image data of the product included in the image data displayed in the area of the POP is associated with the POP identification information, thereby enabling the association between the POP and the product.

[0132] By performing the above processing, the corresponding product determination processing unit 231 can associate the POP with the product. The association between the POP and the product may be achieved, for example, by associating the POP identification information with the product identification information.

[0133] The effect display processing unit 24 estimates and displays the effect of the POP associated with (corresponding to) the product. The effect display processing unit 24 includes a product selection reception processing unit 240, a POP identification processing unit 241, an index processing unit 242, a display control processing unit 243, and a display output processing unit 244.

[0134] The product selection reception processing unit 240 of this embodiment receives the selection of a product for which the effect of the POP corresponding to the product is to be estimated. There is no restriction on the method of receiving the selection of the product. For example, it receives the selection of some or all of the store, display shelf, date, etc. for which the effect estimation of the POP is desired in a predetermined method. When receiving the selection, the effect estimation processing unit 22 extracts the image data of the corresponding product from the data storage unit 21 based on the store identification information of the selected store, the display shelf identification information of the selected display shelf, and the corresponding shooting date of the selected date, and displays it as a list as shown in the screen of FIG. 27. Then, from this list, it receives the selection of the product for which the effect of the POP is to be estimated.

[0135] Note that the product selection reception processing unit 240 may receive the selection of one product or the selection of a plurality of products. It may also receive the selection of the products on a shelf level of the display shelf or all the products on the display shelf. When receiving the selection of a plurality of products, the processing in the effect display processing unit 24 is performed for each selected product.

[0136] The POP specific processing unit 241 identifies the POP corresponding to the product for which selection has been received by the product selection reception processing unit 240. By identifying the POP identification information corresponding to the product identification information of the product for which selection has been received by the product selection reception processing unit 240, the POP corresponding to the selected product can be identified. Note that since there may be a plurality of POPs associated with a product, when a plurality of POPs are associated, each POP is identified. Note that the POP specific processing unit 241 executes the processing by identifying the POP corresponding to the product for which selection has been received by the product selection reception processing unit 240, but the POP to be processed may be identified by receiving the selection of the POP.

[0137] In this case, some or all of the store, display shelf, date, etc. for which it is desired to estimate the effect of the POP are received for selection by a predetermined method. When the selection is received, the effect estimation processing unit 22 extracts the image data of the corresponding POP from the data storage unit 21 based on the store identification information of the selected store, the display shelf identification information of the selected display shelf, and the corresponding shooting date of the selected date, and displays it as a list on the screen. Then, from this list, a selection of the POP for which the effect of the POP is to be estimated is received. At this time, the product corresponding to the POP for which the selection has been received is identified. By such processing, the same processing can be performed by receiving the selection of the POP instead of the selection of the product.

[0138] The index processing unit 242 performs an operation using the index value based on the type of the POP, such as summing the index values based on the type of the POP corresponding to the selected product.

[0139] In the index processing unit 242, as shown in FIG. 28, the index value is set according to the type of the POP, and an operation is performed such as summing the index values corresponding to the type of the POP corresponding to the selected product. Then, the calculated index value is divided into ranks divided into a predetermined number of stages, for example, 10 stages.

[0140] The display control processing unit 243 determines a display method according to the rank classified by the index processing unit 242. For example, it determines the color to be displayed. When the rank is 1 (the total index value is low), it is "blue" (when there is no index value, it is "white"), and when the rank is 10 (the total index value is high), it is "red". For example, as shown in FIG. 29, the color is divided into 10 levels for display. This color is the color to be displayed in the area of the relevant product or POP. By such display according to the rank, display can be performed like a so-called heat map.

[0141] The display output processing unit 244 of this embodiment displays and outputs the area of the selected product or POP based on the color determined by the display control processing unit 243. Examples of this display are shown in FIGS. 30 and 31. In FIG. 30(a), it is the case where the display of the area of a single product is changed when one product is selected, and in FIG. 30(b), it is the case where the display of the area of each product is changed when multiple products are selected. In FIG. 31(a), it is the case where the display of the area of the POP corresponding to the product is changed when a product is selected, and in FIG. 31(b), it is the case where the display of the area of the POP corresponding to each product is changed when multiple products are selected. In addition to displaying the overall color of the product or POP area, a part or all of the product or POP area may be divided into predetermined rectangles, and the color may be changed within the range of those rectangles.

[0142] Note that FIGS. 30 and 31 in this specification show the change of the display method by pattern instead of color for convenience of the drawing. In addition to color, various display methods such as color shade, color brightness, pattern difference, emphasis display difference, and presence or absence of a blinking pattern can be used as the display method. Also, instead of performing the display on the image data, the rank itself may be displayed in text or the like.

[0143] Next, an example of the processing in this embodiment will be described using the flowcharts of FIGS. 24 to 26.

[0144] First, similar to Example 2 or Example 3, the POP detection processing unit 20 executes POP detection processing for detecting POP in the image data read from the image data input terminal 3 (S100). The POP detection processing can be the same as that in each example. The POP detected here is stored in the data storage unit 21. Similarly, since the product in the image data is also detected by the object detection processing unit 202, it is also stored in the data storage unit 21.

[0145] After executing the POP detection processing, the association processing in the association processing unit 23 is executed (S120).

[0146] The POP determination processing unit 230 in the association processing unit 23 extracts the image data of the area of the POP stored in the data storage unit 21 at an arbitrary timing, and determines the character information displayed in that area using a known method such as OCR recognition processing or deep learning (S600).

[0147] Then, using part or all of the character information determined by the POP determination processing unit 230, the corresponding product determination processing unit 231 determines the product corresponding to the POP (S610).

[0148] The corresponding product determination processing unit 231 extracts the store identification information, display shelf identification information, and date information associated with the image data of the area of the POP from the data storage unit 21. Then, among the product objects detected by the object detection processing unit 202 and stored in the data storage unit 21, the products having the same store identification information, display shelf, and date information, and whose product identification information matches part or all of the character information determined by the POP determination processing unit 230, especially the product identification information indicating the product described in the POP, are identified to perform the association between the POP and the product. The association between the POP and the product can be achieved by associating the POP identification information and the product identification information and storing them in the data storage unit 21.

[0149] After the end of the association processing in the association processing unit 23 by the above processing, the effect display processing in the effect display processing unit 24 is executed (S130).

[0150] The product selection reception processing unit 240 in the effect display processing unit 24 receives selection of some or all of, for example, a store, a display shelf, a date, etc. for which it is desired to estimate the POP effect, in a predetermined manner. When receiving such selection, the effect estimation processing unit 22 extracts the image data of the corresponding product from the data storage unit 21 based on the store identification information of the selected store, the display shelf identification information of the selected display shelf, and the shooting date corresponding to the selected date, and displays it as a list as shown in the screen of FIG. 27. Then, selection of one or more products for which the POP effect is to be estimated is received from this list (S700).

[0151] The POP specific processing unit 241 specifies one or more POPs corresponding to the product for which selection has been received by the product selection reception processing unit 240 (S710).

[0152] Then, the index processing unit 242 performs an operation using the index value based on the type of the POP identification information specified by the POP specific processing unit 241, and ranks which rank among the predetermined ranks it is located in (S720).

[0153] When the index processing unit 242 performs ranking, the display control processing unit 243 determines the color to be displayed in the area of the product or the POP corresponding to the product (S740), and the display output processing unit 244 performs the display on the display device 72 of the management terminal 2 as shown in FIGS. 30 and 31.

[0154] By executing the above processing, the effect of the POP and the product can be determined.

Example

[0155] As a modification of Example 4, the effect display processing unit 24 may be able to display not only the effect estimation by associating the POP and the product, but also other arbitrary items. Examples of other items include, but are not limited to, customer voices, recognition rates, sales, etc., and other items may also be acceptable.

[0156] For example, as customer feedback, for the product selected by the product selection acceptance processing unit 240, the evaluation of the product is received by the effect display processing unit 24 through a customer questionnaire or the like, the evaluation is converted into an index value, and based on this, the total index value for the product is calculated, and ranking is performed to determine which rank among the predetermined ranks the product belongs to. Then, the display control processing unit 243 determines the color to be displayed in the area of the product or the POP corresponding to the product, and the display output processing unit 244 performs the display on the display device 72 of the management terminal 2.

[0157] Also, for the awareness, for the product selected by the product selection acceptance processing unit 240, the awareness of the product is received by the effect display processing unit 24 through an awareness questionnaire or the like, the awareness is converted into an index value, and based on this, the total index value for the product is calculated, and ranking is performed to determine which rank among the predetermined ranks the product belongs to. Then, the display control processing unit 243 determines the color to be displayed in the area of the product or the POP corresponding to the product, and the display output processing unit 244 performs the display on the display device 72 of the management terminal 2.

[0158] Furthermore, for sales, based on at least two or more dates for comparison, product identification information of the product selected by the product selection acceptance processing unit 240, store identification information, etc., sales information of the product corresponding to the product identification information is obtained from a product sales management system such as a POS system. Then, the growth rate of the sales is calculated. Ranking is performed to determine which rank among the predetermined ranks the growth rate belongs to. Then, the display control processing unit 243 determines the color to be displayed in the area of the product or the POP corresponding to the product, and the display output processing unit 244 performs the display on the display device 72 of the management terminal 2.

[0159] When the display output processing unit 244 displays the effect estimation based on the association between the POP and the product, customer feedback, awareness, sales, etc. in the area of the product or the POP corresponding to the product, it may be displayed in a switchable manner for each item or displayed simultaneously.

[0160] For example, by performing a predetermined operation, colors corresponding to the ranks of each item of effect estimation, voice of customers, recognition, and sales may be displayed in the area of the product or the POP corresponding to the product in the order of these items.

[0161] Also, the area of the product or the POP corresponding to the product may be divided for each item, and the colors corresponding to the respective ranks may be simultaneously displayed for the items of effect estimation, voice of customers, recognition, and sales. For example, an area corresponding to the number of items may be provided in the area of the product, and for each item, the colors corresponding to the respective ranks may be simultaneously displayed there. FIG. 32 schematically shows an example of this. FIG. 32(a) shows the case where rectangular areas corresponding to the items are provided in the area of the product and the respective colors are displayed, and FIG. 32(b) shows the case where the area of the product is divided according to the items and the respective colors are displayed in the divided areas. Also, FIG. 32 is an enlarged view of the area of one product, and similar display processing may be performed in the area of the POP.

[0162] By executing the above-described processing, it is possible to simultaneously display a plurality of items.

Industrial Applicability

[0163] By using the information processing system 1 of the present invention, it becomes possible to detect a POP from image data obtained by photographing a display shelf and estimate the effect of the POP.

Explanation of Signs

[0164] 1: Information processing system 2: Management terminal 3: Image data input terminal 20: POP detection processing unit 21: Data storage unit 22: Effect estimation processing unit 23: Association processing unit 24: Effect display processing unit 70: Arithmetic device 71: Storage device 72: Display device 73: Input device 74: Communication device 200: Image data input reception processing unit 201: Image data normalization processing unit 202: Object detection processing unit 203: Individual discrimination processing unit 204: Region detection processing unit 220: POP selection reception processing unit 221: Product selection reception processing unit 222: Analysis processing unit 223: Effect output processing unit 230: POP determination processing unit 231: Corresponding product determination processing unit 240: Product selection reception processing unit 241: POP identification processing unit 242: Index processing unit 243: Display control processing unit 244: Display output processing unit

Claims

1. An information processing system for estimating the effect of a POP, The information processing system includes: a POP detection processing unit that detects a POP from image data showing a display shelf; An effect estimation processing unit that estimates an effect of the POP, The POP detection processing unit is an object detection processing unit that inputs the image data into a learning model that is trained using objects and their types, detects objects appearing in the image data, and outputs the types of the objects; and an individual discrimination processing unit that discriminates that the type of the detected object is a POP or a candidate for a POP when the type of the detected object is a type other than a predetermined type, The effect estimation processing unit: Estimating the effect of the POP using sales information of the product corresponding to the POP; An information processing system comprising:

2. The POP detection processing unit is A region detection processing unit that detects a region of the image data, The individual discrimination processing unit is determining a type of the detected object using the detected area and the type of the detected object; 2. The information processing system according to claim 1 .

3. The effect estimation processing unit: a POP selection reception processing unit that receives a selection of a POP whose effect is to be estimated; an analysis processing unit that estimates an effect of the POP by using sales information of a product corresponding to the selected POP and sales information of the product when the POP is not present; 3. The information processing system according to claim 1, further comprising:

4. The effect estimation processing unit: a POP selection reception processing unit that receives a selection of a POP whose effect is to be estimated; a product selection reception processing unit that receives a product selection; an analysis processing unit that acquires sales information of the product selected when a POP is present and sales information of the product selected when a POP is not present from a predetermined sales management system, and estimates an effect of the POP by comparing the sales information of the product selected when a POP is present and the sales information of the product selected when a POP is not present; 3. The information processing system according to claim 1, further comprising:

5. The analysis processing unit includes: Acquire sales information when there is a POP for the product for which the selection has been accepted, and sales information when there is no POP for the product for which the selection has been accepted, from a predetermined sales management system; The effect of the POP is estimated by comparing the sales information and the number of faces of the product when the POP is present with the sales information and the number of faces of the product when the POP is not present.

5. The information processing system according to claim 4.

6. Computer, a POP detection processing unit that detects POPs from image data showing a display shelf; An information processing program that functions as an effect estimation processing unit that estimates the effect of the POP, The POP detection processing unit is an object detection processing unit that inputs the image data into a learning model that has been trained using objects and their types, detects objects appearing in the image data, and outputs the types of the objects; and an individual discrimination processing unit that discriminates that the type of the detected object is a POP or a candidate for a POP when the type of the detected object is a type other than a predetermined type, The effect estimation processing unit: Estimating the effect of the POP using sales information of the product corresponding to the POP; 2. An information processing program comprising:

Images