Information processing device

Abstract

To provide a technology of easily discerning appropriateness of products display.SOLUTION: An information processing device 2000 includes an image analysis unit 2020 that analyzes a shelf image obtained by capturing a products shelf, and generates actual display information which indicates the display state of products on the products shelf, a comparison unit 2040 that compares the actual display information with standard display information which indicates the standard of products display on the products shelf, and generates comparison information which indicates the comparison result, and a generation unit 2060 that, by using the comparison information, generates a result image indicating the product that does not conform to the standard indicated by the standard display information. In the result image, a product that does not conform to the standard indicated by the standard display information and a product conforming to the standard indicated by the standard display information are shown so as to be distinguishable from each other. In the result image, the generation unit 2060 superimposes, as superimposition data for indicating the comparison result, a heat map in which the color of a part corresponding to the product that does not conform to the standard display information is relatively dark while the color of a part corresponding to the product conforming to the standard display information is relatively light, on the shelf image.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present invention relates to the management of products.

Background Art

[0002] In stores such as convenience stores and supermarkets, products are displayed on product shelves. And information processing technologies for improving the efficiency of operations related to such product displays have been developed. For example, Patent Document 1 discloses a technology for automatically determining whether the relationship between a product placed on a product shelf and a shelf label attached to the product shelf is correct.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There may be an operation of prescribing in advance how each product should be displayed on the product shelf (such as the display position and the number of faces), and displaying the products according to the prescription. However, even when such an operation is carried out, in reality, the products may not be displayed according to the prescription. Therefore, an operation is performed in which a person in charge of the manufacturer visually checks the display status of the products and checks whether the products are correctly displayed. However, visual checks have problems such as a large workload for the operator.

[0005] The present invention of the application has been made in view of the above problems, and one of its purposes is to provide a technology that can easily grasp the correctness of product displays.

Means for Solving the Problems

[0006] The information processing apparatus of the present invention is Image analysis means that analyzes shelf images in which product shelves are captured and generates actual display information representing the product display situation on the product shelves, A comparison means that compares the actual display information with standard display information representing the standards for displaying products on the product shelves, and generates comparison information representing the result of the comparison, The system includes a generation means that generates a result image in which products that do not conform to the standards indicated by the standard display information and products that are out of stock can be identified using the comparison information.

[0007] The information processing system of the present invention comprises an information processing device and a portable terminal. The mobile device transmits the shelf image, which contains a photograph of the product shelf, to the information processing device. Information processing equipment, Image analysis means that receives the shelf image, performs image analysis on the shelf image, and generates actual display information representing the display status of products on the product shelf, A comparison means that compares the actual display information with standard display information representing the standards for displaying products on the product shelves, and generates comparison information representing the result of the comparison, The system includes a generation means that generates a result image in which products that do not conform to the standards indicated by the standard display information and products that are out of stock can be identified using the comparison information.

[0008] The control method of the present invention is performed by a computer. The control method is One or more computers, Image analysis is performed on the shelf image in which the product shelves are captured to generate actual display information representing the product display status on the product shelves. The actual display information is compared with the standard display information, which represents the criteria for displaying products on the product shelves, and comparison information representing the result of that comparison is generated. This includes generating a result image using the comparison information that displays products that do not meet the criteria indicated by the standard display information and products that are out of stock in a manner that allows for identification.

[0009] The program of the present invention causes a computer to execute each step of the control method of the present invention.

Advantages of the Invention

[0010] According to the present invention, a technique for easily grasping the correctness of product display is provided.

Brief Description of the Drawings

[0011] The above-described object, as well as other objects, features and advantages, will become more apparent from the following preferred embodiments and the accompanying drawings below.

[0012] [Figure 1] It is a diagram conceptually illustrating the operation of the information processing apparatus according to Embodiment 1. [Figure 2] It is a diagram illustrating the functional configuration of the information processing apparatus. [Figure 3] It is a diagram illustrating the configuration of a computer that realizes the information processing apparatus. [Figure 4] It is a diagram illustrating the usage form of the information processing apparatus. [Figure 5] It is a flowchart illustrating the flow of processing executed by the information processing apparatus of Embodiment 1. [Figure 6] It is a diagram illustrating a case where the same products are displayed separately. [Figure 7] It is a diagram illustrating the configuration of actual display information. [Figure 8] It is a diagram illustrating the configuration of reference display information. [Figure 9] It is a diagram illustrating comparison information. [Figure 10] It is a block diagram illustrating the functional configuration of the information processing apparatus of Embodiment 2. [Figure 11] It is a first diagram illustrating a result image. [Figure 12] It is a second diagram illustrating a result image. [Figure 13] It is a third diagram illustrating a result image. [Figure 14]It is a figure illustrating a result image including the shelf matching rate. [Figure 15] It is a figure illustrating a result image including a list of products. [Figure 16] It is a figure illustrating how the result image changes according to an operation on the product list. [Figure 17] It is a figure illustrating how the result image changes according to an operation on the product image. [Figure 18] It is a figure illustrating a result image including superimposed data representing unidentified products. [Figure 19] It is a figure illustrating correction information.

Embodiments for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same components are denoted by the same reference numerals, and the description thereof will be omitted as appropriate. Also, unless otherwise specified, in each block diagram, each block represents a configuration in terms of functional units, not in terms of hardware units.

[0014] [Embodiment 1] [Overview] FIG. 1 is a diagram conceptually illustrating the operation of an information processing apparatus (information processing apparatus 2000 whose configuration is illustrated in FIG. 2) according to Embodiment 1. Here, the operation of the information processing apparatus 2000 described using FIG. 1 is an illustration for facilitating the understanding of the information processing apparatus 2000 and does not limit the operation of the information processing apparatus 2000. Details and variations of the operation of the information processing apparatus 2000 will be described later.

[0015] The information processing device 2000 acquires a shelf image captured of the product shelf on which the products are displayed. In Figure 1, the product shelf 20 is captured by the camera 10, resulting in the generation of the shelf image 12. The shelf image 12 includes the product shelf 20 and the products displayed on the product shelf 20. Note that the product shelf targeted by the information processing device 2000 can be any place where products are displayed, and is not limited to those with multiple shelves as shown in Figure 1. For example, in cases where products are stacked flat on a table, this table is treated as a product shelf.

[0016] The information processing device 2000 acquires standard display information that represents the criteria for displaying products on the product shelf 20. For example, the standard display information indicates information such as the products that should be displayed on the product shelf 20 and the position of each product on the product shelf 20.

[0017] The information processing device 2000 compares the actual display situation represented by the shelf image 12 with the standard display information and generates information representing the comparison result. For example, in Figure 1, information is generated indicating that there is an error in the placement of two products.

[0018] To achieve the above processing, the information processing device 2000 performs image analysis on the shelf image 12 to generate information regarding the display status of products on the product shelf 20 (hereinafter referred to as actual display information). The actual display information includes, for example, the names of the products displayed on the product shelf 20 and the location of each product.

[0019] The information processing device 2000 compares the actual display information generated by image analysis of the shelf image 12 with the acquired standard display information and generates comparison information representing the result.

[0020] <Effects and Actions> According to the information processing device 2000 of this embodiment, the actual display information generated by image analysis of the shelf image 12 is compared with the standard display information, and comparison information representing the result is generated. By comparing the actual display information and the standard display information in this way, it is possible to determine whether the product display status on the product shelf 20 conforms to the standard. Therefore, compared to the case where a person visually checks whether the product display status conforms to the standard, it is possible to easily determine whether the product display status conforms to the standard.

[0021] Furthermore, in cases where the information is checked visually by a person, there is a risk of human error such as misinterpretation. According to the Information Processing Device 2000, such human errors can also be prevented.

[0022] <Example of functional configuration> Figure 2 illustrates the functional configuration of the information processing device 2000. The information processing device 2000 includes an image analysis unit 2020 and a comparison unit 2040. The image analysis unit 2020 generates actual display information by performing image analysis on the shelf image 12. The comparison unit 2040 generates comparison information by comparing the actual display information with reference display information.

[0023] <Example of hardware configuration of information processing device 2000> Each functional component of the information processing device 2000 may be implemented by hardware (e.g., hardwired electronic circuits) or by a combination of hardware and software (e.g., a combination of electronic circuits and programs that control them). The following will further explain the case where each functional component of the information processing device 2000 is implemented by a combination of hardware and software.

[0024] Computer 1000 can be various types of computers. For example, computer 1000 may be a personal computer (PC), a server machine, a tablet terminal, or a smartphone. Computer 1000 may be a dedicated computer designed to implement the information processing device 2000, or it may be a general-purpose computer.

[0025] Figure 3 illustrates the configuration of a computer 1000 that implements an information processing device 2000. The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, memory 1060, storage device 1080, input / output interface 1100, and network interface 1120 to send and receive data to and from each other. However, the method of connecting the processor 1040 and the other components is not limited to bus connection. The processor 1040 is a processor implemented as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The memory 1060 is a main memory device implemented as RAM (Random Access Memory). The storage device 1080 is an auxiliary storage device implemented as a hard disk, SSD (Solid State Drive), memory card, or ROM (Read Only Memory). The input / output interface 1100 is an interface for connecting the computer 1000 to input / output devices. For example, input devices such as keyboards and output devices such as display devices are connected to the input / output interface 1100. The network interface 1120 is an interface for connecting the computer 1000 to a network. This communication network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The method by which the network interface 1120 connects to the network may be wireless or wired.

[0026] The storage device 1080 stores program modules that implement each function of the information processing device 2000. The processor 1040 implements each of these program modules by executing them.

[0027] <About Camera 10> Camera 10 is any imaging device capable of capturing images and generating images representing the imaging results. For example, camera 10 is a camera installed in a smartphone or tablet device.

[0028] <Examples of usage scenarios for the Information Processing Device 2000> For example, a user of the information processing device 2000 uses the information processing device 2000 via a mobile terminal. The user of the information processing device 2000 is, for example, a person who checks the display status of products, such as a field representative. Figure 4 is a diagram illustrating an example of how the information processing device 2000 is used. In Figure 4, the information processing system 4000 is composed of the information processing device 2000 and the mobile terminal 3000. The mobile terminal 3000 is a mobile terminal used by the user. The information processing device 2000 is implemented, for example, as a server device.

[0029] The mobile terminal 3000 is equipped with a camera 10. The mobile terminal 3000 also has an application 100 pre-installed for using the information processing device 2000. The application 100 controls the camera 10 to take images. Furthermore, the application 100 sends a request to the information processing device 2000 containing the shelf image 12 generated by the camera 10. The information processing device 2000 receives this request from the application.

[0030] The information processing device 2000 performs image analysis on the shelf image 12 shown in the request to generate actual display information, and then generates comparison information by comparing this with reference display information. The information processing device 2000 then transmits the generated comparison information to the mobile terminal 3000.

[0031] Note that the comparison information does not necessarily need to be transmitted to the mobile terminal 3000. The handling of the generated comparison information will be described later. Also, the information processing device 2000 does not necessarily need to be used via the mobile terminal 3000.

[0032] <Processing flow> Figure 5 is a flowchart illustrating the processing flow performed by the information processing device 2000 of Embodiment 1. The image analysis unit 2020 acquires the shelf image 12 (S102). The image analysis unit 2020 generates actual display information by performing image analysis on the shelf image 12 (S104). The comparison unit 2040 acquires the reference display information (S106). The comparison unit 2040 generates comparison information by comparing the actual display information with the reference display information (S108).

[0033] The processing flow performed by the information processing device 2000 is not limited to the example described above. For example, the acquisition of reference display information may be performed before the generation of actual display information.

[0034] The timing at which the information processing device 2000 executes the series of processes shown in Figure 5 is arbitrary. For example, the information processing device 2000 executes the series of processes shown in Figure 5 in response to receiving a request from the application 100 mentioned above. Alternatively, for example, the information processing device 2000 may receive an input operation from a user operating an input device connected to the information processing device 2000, specifying the shelf image 12. In this case, the information processing device 2000 executes the series of processes shown in Figure 5 as a trigger for receiving the input operation from the user.

[0035] <How to obtain shelf image 12: S102> The method by which the image analysis unit 2020 obtains the shelf image 12 is arbitrary. For example, the image analysis unit 2020 obtains the shelf image 12 included in the request sent by the aforementioned mobile terminal 3000. Alternatively, for example, in response to receiving a user operation specifying a shelf image 12, the image analysis unit 2020 may access the storage device where the specified shelf image 12 is stored and obtain that shelf image 12.

[0036] Here, one product shelf 20 may be represented by multiple shelf images 12. For example, if the product shelf 20 is long horizontally, the entire product shelf 20 can be captured by capturing images of the product shelf 20 multiple times while shifting the imaging range of the camera 10 horizontally.

[0037] In this case, in order to ensure that all products displayed on the product shelf 20 are captured, it is preferable to capture the product shelf 20 in two shelf images 12 that include adjacent areas of the product shelf 20, such that products on the boundary between the two images (the edges of the shelf images 12) are included in both shelf images 12. For example, the information processing device 2000 can grasp the situation of one product shelf 20 from multiple shelf images 12 by treating the multiple shelf images 12 as being joined together at the parts that contain common products. This is equivalent to taking multiple photographs so that parts of each photograph overlap in order to obtain a panoramic photograph.

[0038] <How to identify product shelf 20 included in shelf image 12> The information processing device 2000 needs to identify which product shelf 20, located in a store or other location, is included in the acquired shelf image 12. For example, when acquiring the shelf image 12, the information processing device 2000 also acquires an identifier that identifies the product shelf 20 included in that shelf image 12.

[0039] For example, as mentioned above, suppose a request including a shelf image 12 is sent from an application 100 running on a mobile terminal 3000 to an information processing device 2000. In this case, the application 100 includes an identifier for the product shelf 20 contained in the shelf image 12 in the request. For example, the application 100 receives input from the user of the mobile terminal 3000 specifying the identifier for the product shelf 20, and includes the entered identifier in the request.

[0040] The method for identifying the product shelves 20 included in the shelf image 12 is not limited to obtaining the identifier of the product shelves 20 along with the shelf image 12. For example, the identifier of the product shelves 20 can be captured by the camera 10. Specifically, a mark representing the identifier of the product shelves 20 can be attached to the product shelves 20 in advance. In this way, the shelf image 12 generated as a result of the camera 10 capturing the product shelves 20 will include the mark representing the identifier of that product shelf 20. The information processing device 2000 can identify the product shelves 20 included in the shelf image 12 by performing image analysis on this mark. For example, this mark can be a string representing the identifier of the product shelves 20 or a QR code (registered trademark).

[0041] <Generating actual display information: S104> The image analysis unit 2020 generates actual display information by performing image analysis on the shelf image 12 (S104). The information to be included in the actual display information is described below.

[0042] <<Product Identification Information>> The actual display information includes identification information for each product displayed on shelf 20. For example, product identification information includes identifiers such as the product name and JAN (Japanese Article Number) code. In the following explanation, unless otherwise specified, it is assumed that the product name will be used as product identification information. The same applies to the standard display information described later.

[0043] For example, the image analysis unit 2020 extracts image regions representing each product from the shelf image 12. The image analysis unit 2020 compares the extracted image regions with the image features of products registered in the product database to obtain information about the product represented by that image region from the product database. The product database is assumed to contain at least one of the following pieces of information for each product: product name, identifier (such as JAN (Japanese Article Number) code), name of the group to which the product belongs (e.g., brand name), product category to which the product belongs, product volume (e.g., 350ml), product size, standard price, and manufacturer. The manufacturer of a product is any manufacturer related to that product (e.g., manufacturing company or distributor).

[0044] As described above, the image analysis unit 2020 can identify the product names of each product included in the shelf image 12 based on the information obtained from the product database.

[0045] <<Group Name>> In addition, for example, the actual display information indicates the name of the group to which the product belongs. The group to which a product belongs can be obtained from the product database, similar to the product name mentioned earlier. Alternatively, any information that identifies the manufacturer, such as the manufacturer's identifier, can be used instead of the group name. The same applies to the standard display information described later.

[0046] <<Manufacturer Name>> Other examples include display information, which indicates the product's manufacturer. The manufacturer's information can be obtained from the product database, similar to the product name mentioned earlier. Alternatively, any information that identifies the manufacturer, such as a manufacturer identifier, can be used instead of the manufacturer's name. The same applies to the standard display information described later.

[0047] <<Position>> In addition, for example, the actual display information indicates the position of the product on the product shelf 20. The position of a product is represented by a combination of, for example, its position in the row direction (e.g., which row it is on from the top) and its position in the column direction (e.g., which column it is in from the leftmost column). Here, existing technologies can be used to identify the position of each product from an image that includes the product shelf.

[0048] Furthermore, if the same type of product (products with the same product name) is placed in multiple locations, the actual display information will indicate multiple locations for a single product.

[0049] <<Number of faces>> Other information, such as the actual display information, indicates the number of product faces. It should be noted that identical products may be displayed separately on shelf 20. Figure 6 illustrates a case where identical products are displayed separately. In Figure 6, product A is arranged in two separate areas: area 70-1 and area 70-2. Area 70-1 has 2 product faces, and area 70-2 has 3 product faces.

[0050] When the same product is placed in locations that are far apart, the actual display information will show the number of faces for each of the multiple locations that are separated from each other. For example, in Figure 6, the actual display information will show 2 faces for product A in area 70-1 and 3 faces in area 70-2.

[0051] In addition, for example, the actual display information may be the sum of the number of faces of a product at multiple locations that are far apart from each other. For example, in Figure 6, the actual display information shows 5 as the number of faces of product A.

[0052] <<Example of display information structure>> Figure 7 illustrates the structure of actual display information. The display information in Figure 7 is referred to as Table 400. Table 400 has columns for Product 402, Group 404, Manufacturer 406, Position 408, and Face Count 410.

[0053] Here, the information that the actual display information should include, that is, the information that should be obtained by image analysis of the shelf image 12, may be set in advance in the image analysis unit 2020, or it may be determined dynamically. In the latter case, for example, the image analysis unit 2020 determines the information to be obtained from the shelf image 12 based on the acquired reference display information. For example, if the reference display information indicates the products that should be placed on the shelf and the position where each product should be placed, the information that the image analysis unit 2020 should obtain from the shelf image 12 will be the product name and the position of the product.

[0054] <Regarding standard display information> As mentioned above, the standard display information represents the standard for displaying products on the product shelf 20. The information shown in the standard display information is explained below.

[0055] <<Product Identification Information>> For example, standard display information indicates the identification information (such as product name or JAN code) for each product that should be displayed on shelf 20. In the following explanation, unless otherwise specified, the product identification information indicated by the standard display information will be the product name.

[0056] <<Position>> In addition, for example, the standard display information indicates the position where each product should be placed on the product shelf 20. The position of the product indicated by the standard display information is represented, for example, by a combination of the position in the shelf direction and the position in the column direction. In this case, the standard display information indicates that both the position in the shelf direction and the position in the column direction of the product should be the position indicated by the standard display information.

[0057] The product location indicated by the standard display information may be a range in the column direction or vertical direction. For example, the standard display information may indicate a range such as "2nd to 4th row" or "3rd to 5th column." Alternatively, the standard display information may indicate only the row direction location for a particular product. In this case, the product can be displayed in any column of the row indicated in the standard display information. Similarly, the standard display information may indicate only the column direction location for a particular product.

[0058] <<Number of faces>> In addition, for example, standard display information indicates the number of facets of a product. For instance, if the standard display information indicates that product A has 5 facets, it means that product A should be displayed so that it has 5 facets.

[0059] <<Parallel request>> In addition, for example, standard display information may indicate that certain products should be placed together (parallel requirements). In other words, products for which parallel requirements are indicated in the standard display information should not be placed in multiple areas, as illustrated in Figure 6.

[0060] Here, the unit to which a parallel request is specified is not limited to individual products. For example, a parallel request may be specified at the unit of a group of products. In this case, for example, the standard display information indicates a parallel request for products of a specific brand. Each product belonging to a brand for which a parallel request is indicated in the standard display information is required to be placed in a grouped position on the product shelf 20.

[0061] In addition, parallel requirements may also be specified on a manufacturer-by-manufacturer basis. In this case, for example, the standard display information would indicate parallel requirements for products from a specific manufacturer. Each product from a manufacturer for which parallel requirements are indicated in the standard display information is required to be placed in a grouped position on the product shelf 20.

[0062] In addition, parallel requirements may also be specified by the volume of the product. In this case, for example, the standard display information indicates parallel requirements for products of a specific volume. Each product of a specific volume for which parallel requirements are indicated in the standard display information is required to be placed in a grouped position on the product shelf 20. More specifically, for example, if the specific volume is 500ml, then products with a volume of 500ml, such as 500ml cans of beer sold by a manufacturer, will be placed in a grouped position on the product shelf 20.

[0063] <<Array Request>> In addition, the standard display information may also indicate requirements regarding the placement relationships between multiple products (hereinafter referred to as arrangement requirements). For example, an arrangement requirement might be "place product A and product B adjacent to each other." For example, suppose the standard display information indicates an arrangement requirement for product A and product B without specifying their positions. In this case, product A and product B can be placed at any position on the product shelf 20, as long as they are adjacent to each other.

[0064] Conversely, an array requirement may also indicate that multiple items should be placed separately. For example, one could define an array requirement such as "item A and item B should be placed on different rows."

[0065] Here, the unit in which an arrangement request is specified is not limited to products. For example, an arrangement request may be specified in terms of product groups. For example, a request such as "place group D and group E adjacent to each other" would be used. In this case, the area on the product shelf 20 where products belonging to group D are placed and the area where products belonging to group E are placed must be adjacent to each other. In addition, similar to product arrangement requests, group arrangement requests may also indicate that multiple groups should be placed apart.

[0066] In addition, for example, arrangement requirements may be specified on a manufacturer-by-manufacturer basis. In this case, for example, the requirement might be "place manufacturer X's products and manufacturer Y's products adjacent to each other." In this case, the area on the product shelf 20 where manufacturer X's products are placed and the area where manufacturer Y's products are placed must be adjacent to each other. Similar to product arrangement requirements, manufacturer arrangement requirements may also indicate that products from multiple manufacturers should be placed far apart from each other.

[0067] <<Applicable Period>> In addition, standard display information may also indicate the period during which the standard display information applies to each product. For example, if the standard display information for product A indicates an application period of "October 1, 2017 to October 31, 2017," it means that the information for product A indicated in the standard display information (such as location) must be observed within the above application period. Note that the application period may be defined not only for each product, but also for groups of products or for the standard display information as a whole.

[0068] <<Example of standard display information structure>> Figure 8 is a diagram illustrating the structure of standard display information. The table in Figure 8 is denoted as Table 500. Table 500 has the following columns: Product 502, Group 504, Manufacturer 506, Position 508, Number of Faces 510, Parallel Request 512, Array Request 514, and Application Period 516.

[0069] Product 502, group 504, and manufacturer 506 each have data displayed in one of them. Records in product 502 that show a product name indicate criteria for products with that name. Records in group 504 that show a group name indicate criteria for groups with that name. Records in manufacturer 506 that show a manufacturer name indicate criteria for manufacturers with that name. The products to be placed on the product shelf 20 are the products whose product names are displayed in product 502. Note that table 500, as described with reference to Figure 8, is not limited to the configuration described above. For example, table 500 may further include at least one of the following: identifier, product category, quantity, and product size.

[0070] The standard display information may be information about all products placed on the product shelf 20, or it may be information about some of the products. In the latter case, for example, the standard display information indicates the display standards for products from a specific manufacturer. In this case, standard display information for one product shelf 20 is prepared for each manufacturer. Therefore, the standard display information in Figure 8 does not need to include manufacturer 506.

[0071] If standard display information is provided for each manufacturer, the information processing device 2000 may compare only the standard display information for a specific manufacturer with the actual display information. In this case, for example, the information processing device 2000 obtains information that identifies the standard display information to be compared (for example, the manufacturer's name). This information is entered by the user of the information processing device 2000. For example, the user performs an input operation to the aforementioned mobile terminal 3000, specifying the manufacturer's name. The application 100 includes the manufacturer's name entered by the user in the aforementioned request and sends it. By receiving this request, the information processing device 2000 can understand the manufacturer's name specified by the user. Alternatively, for example, the user may input the manufacturer's name by operating an input device connected to the information processing device 2000.

[0072] <Method for obtaining standard display information: S106> There are various ways in which the information processing device 2000 can acquire standard display information. For example, standard display information can be pre-stored in a storage device accessible from the information processing device 2000. In this case, the information processing device 2000 acquires the standard display information by accessing this storage device. At this time, the information processing device 2000 uses the identifier of the product shelf 20 to be processed to acquire the standard display information for that product shelf 20. Also, when acquiring standard display information for a specific manufacturer, the information processing device 2000 uses the manufacturer name, etc., acquired by the method described above to acquire the standard display information for that specific manufacturer.

[0073] In addition, for example, the standard display information may be stored in a storage device owned by the management server. In this case, for example, the information processing device 2000 obtains the standard display information for the product shelf 20 identified by the identifier of the product shelf 20 by sending a request including the identifier of the product shelf 20 to the management server. When obtaining the standard display information for a specific manufacturer, the request should also include the identifier of that manufacturer (for example, the manufacturer's name).

[0074] <Comparison of actual display information and standard display information: S108> The comparison unit 2040 compares the actual display information with the reference display information and generates comparison information representing the comparison result (S108). To this end, the comparison unit 2040 determines whether each criterion (criterion for product, product group, or manufacturer) indicated in the reference display information is met in the actual display information.

[0075] For example, the comparison unit 2040 compares the product name (product identification information) shown in the actual display information with the product name (product identification information) shown in the standard display information to determine whether each product shown in the standard display information is also shown in the actual display information.

[0076] In addition, for example, if the reference display information indicates the position of a certain product, the comparison unit 2040 determines whether the position of that product in the actual display information matches the position indicated in the reference display information.

[0077] In addition, for example, if the standard display information indicates the number of faces of a certain product, the comparison unit 2040 determines whether the number of faces of that product indicated by the actual display information matches the number of faces indicated by the standard display information. Here, it is possible that the number of faces of a product indicated by the actual display information is greater than the number of faces of that product indicated by the standard display information. For example, a product that should have 3 faces may actually have 5 faces in its display. In such a case, where the number of faces of a product in its display exceeds the number of faces indicated by the standard display information, the product may be determined to have a number of faces that matches the standard, or it may be determined not to have a number of faces that matches the standard. Information that determines which result to make is included in the standard display information, for example.

[0078] If the reference display information indicates a parallel display request or an arrangement request, the comparison unit 2040 determines whether the parallel display request or the arrangement request is satisfied based on the position of each product shown in the actual display information. For example, suppose the reference display information indicates a parallel display request that products belonging to a certain product group should be placed together. In this case, the comparison unit 2040 determines whether the products of that product group are placed together based on the position of each product belonging to that product group in the actual display information. Specifically, if the positions of each product belonging to that product group are consecutive in the row direction or column direction, then those products are placed together, and the parallel display request is satisfied. Also, suppose the reference display information indicates an arrangement request that several products should be placed adjacent to each other. In this case, the comparison unit 2040 determines whether the arrangement request is satisfied by determining whether the positions of those several products are adjacent in the actual display information.

[0079] Furthermore, if the standard display information indicates an applicable period, the comparison unit 2040 will only compare the actual display information with the standard display information for products whose applicable period in the standard display information includes the current date.

[0080] Figure 9 is a diagram illustrating comparison information. The table in Figure 9 is called Table 600. Table 600 has columns for Product 602, Group 604, Manufacturer 606, Display 608, Position 610, Number of Faces 612, Parallel Request 614, and Array Request 616. Records in Product 602 that show a product name show the comparison result between the standard for that product name shown in the standard display information and the actual display information. Records in Group 604 that show a group name show the comparison result between the standard for that product group shown in the standard display information and the actual display information. Records in Manufacturer 606 that show a manufacturer name show the comparison result between the standard for that manufacturer's product shown in the standard display information and the actual display information.

[0081] Display 608, position 610, number of faces 612, parallel request 614, and arrangement request 616 each indicate whether the actual display information conforms to the criteria. For example, in table 600, if the display 608 of a certain product record indicates "mismatch," it means that the name of the product is shown in the standard display information, but not in the actual display information. In other words, it means that the product that should be displayed on the product shelf 20 is not displayed. For example, in table 600, if the position 610 of a certain product record indicates "mismatch," it means that the position of the product shown in the actual display information does not conform to the criteria shown in the standard display information.

[0082] There are various ways to handle comparison information. For example, the comparison unit 2040 stores the comparison information in a memory device. Alternatively, as will be explained in the embodiments described later, the comparison unit 2040 may use the comparison information to generate image data or the like representing the comparison result.

[0083] <<Limitation of Comparison Items>> Assume that the standard display information indicates standards for multiple items, such as the product name of the product to be displayed on the product shelf 20, the position of each product, and the number of faces of each product. In this case, the comparison unit 2040 may compare the actual display information with the standard display information for all items indicated in the standard display information, or it may compare the actual display information with the standard display information for some items. In the latter case, the items to be compared are specified, for example, by user input.

[0084] Furthermore, the comparison unit 2040 may perform the comparison between actual display information and standard display information only for certain products. For example, the comparison unit 2040 may compare actual display information and standard display information only for products from a specific manufacturer. In this case, the specific manufacturer may be specified, for example, through user input.

[0085] Furthermore, the method for comparing actual display information with standard display information for a specific manufacturer is not limited to the method described above. For example, as mentioned earlier, standard display information can be prepared for each manufacturer, and only the standard display information for a specific manufacturer can be acquired by the comparison unit 2040.

[0086] <<Calculation of Match Rate>> The comparison unit 2040 may calculate an index value that represents the degree to which the product display status shown by the actual display information matches the criteria shown in the standard display information. This index value is called the matching rate. For example, the matching rate can be defined as the value obtained by dividing "the number of products displayed on the product shelf 20 that match the criteria shown in the standard display information" by "the total number of products that should be displayed on that product shelf 20". The following formula (1) represents this definition. Here, "a product matches the criteria" means that the information of that product shown in the actual display information matches all the criteria (such as position and number of faces) shown in the standard display information.

number

[0087] Furthermore, the matching rate may be calculated not only for individual product shelves 20, but also for the entire store. In this case, the matching rate for the entire store is calculated using the comparison information generated for each of the multiple product shelves 20. Specifically, this matching rate can be defined as the value obtained by dividing "the number of products displayed in the store that meet the criteria shown in the standard display information" by "the total number of products that should be displayed in that store." The following formula (2) represents this definition.

number

[0088] Hereafter, the matching rate calculated for each shelf (formula (1)) will be called the "shelf-specific matching rate," and the matching rate calculated for the entire store (formula (2)) will be called the "store-specific matching rate." For example, if one company operates multiple stores, the store-specific matching rate for each store can be used as one of the indicators to understand the operational status of each store. Stores with a high store-specific matching rate can be said to have their products displayed appropriately according to the standards. On the other hand, stores with a low store-specific matching rate may not have their products displayed appropriately.

[0089] Here, the trigger for the information processing device 2000 to calculate the store-specific matching rate is arbitrary. For example, the information processing device 2000 calculates the store-specific matching rate for a store when comparison information has been generated for all product shelves 20 present in that store.

[0090] Furthermore, shelf-specific matching rates and store-specific matching rates may be calculated not for all displayed products, but only for products from a specific manufacturer. For example, suppose we want to calculate the shelf-specific matching rate for a certain manufacturer. In this case, the comparison unit 2040 calculates the shelf-specific matching rate for that manufacturer by dividing "the number of products from that manufacturer placed on shelf 20 that meet the criteria shown in the standard display information" by "the total number of products from that manufacturer that should be displayed on shelf 20". The comparison unit 2040 also calculates the store-specific matching rate for that manufacturer by dividing "the number of products from that manufacturer placed in a store that meet the criteria shown in the standard display information" by "the number of products from that manufacturer that should be displayed in the store".

[0091] Furthermore, when calculating shelf-specific matching rates or store-specific matching rates for a particular manufacturer, the information processing device 2000 obtains identification information (such as the manufacturer's name or manufacturer's identifier) ​​to identify that manufacturer. This manufacturer identification information is included, for example, in a request sent from the mobile terminal 3000.

[0092] <<Alerts using match rate>> As mentioned above, the information processing device 2000 obtains an identifier for a product shelf 20 in order to identify that product shelf 20 included in the shelf image 12. The information processing device 2000 then obtains standard display information for the product shelf 20 identified by that identifier. If the information processing device 2000 were to obtain an identifier for a product shelf 20 other than the product shelf 20 included in the shelf image 12, the information processing device 2000 would compare the display status of the product shelf 20 included in the shelf image 12 with the standard display information for the other product shelf 20. Therefore, the shelf-specific matching rate calculated in this case is highly likely to be low.

[0093] Therefore, the information processing device 2000 may be configured to determine whether the shelf-specific matching rate calculated by the comparison unit 2040 is below a predetermined value, and if it is below the predetermined value, to output an alert indicating that the identifier of the product shelf 20 may be incorrect.

[0094] For example, suppose the information processing device 2000 receives a request from a mobile terminal 3000 that includes a shelf image 12 and an identifier for product shelf 20. In this case, if the calculated shelf-specific matching rate is above a predetermined value, the information processing device 2000 sends a message representing the aforementioned alert as a response to the request. Upon receiving this message, the user of the mobile terminal 3000 can take action, for example, by specifying the correct identifier for product shelf 20 and resending the request.

[0095] The above example illustrates the output of an alert in a situation where the identifier of the product shelf 20 may be incorrect, but the situations in which an alert is output are not limited to such situations. For example, the information processing device 2000 may use at least one of the shelf-specific matching rates and store-specific matching rates calculated by the comparison unit 2040 to output an alert indicating that the display is not being carried out appropriately according to the standards if the matching rate does not meet a predetermined condition. For example, the predetermined condition is that the matching rate is equal to or greater than a predetermined value.

[0096] [Embodiment 2] Figure 10 is a block diagram illustrating the functional configuration of the information processing device 2000 of Embodiment 2. Except for the matters described below, the information processing device 2000 of Embodiment 2 has the same functions as the information processing device 2000 of Embodiment 1.

[0097] The information processing device 2000 of Embodiment 2 has a generation unit 2060. The generation unit 2060 generates a result image for the product shelf 20 using the comparison information generated for the product shelf 20. The result image for the product shelf 20 is image data that distinguishes between products that do not conform to the criteria indicated by the standard display information and products that conform to the criteria indicated by the standard display information. The result image may be stored in a storage device as an image file, or it may be displayed on a display device as screen data.

[0098] According to the information processing device 2000 of this embodiment, the result of comparing the product display status on the product shelf 20 with standard display information is graphically represented in the form of a result image. Therefore, the difference between the product display status on the product shelf 20 and the standard display can be easily grasped intuitively.

[0099] <About the display device> The display device on which the result image is displayed can be any display device. For example, the display device may be a display device connected to the information processing device 2000 or a display device of the mobile terminal 3000. Alternatively, the display device may be a display device connected to a terminal installed in the staff room of a store where the product shelves 20 are located, or a display device connected to a terminal installed in the office of the company that operates the store. For example, suppose a field representative for a certain manufacturer uses the information processing device 2000 to check whether the manufacturer's products are correctly displayed on the product shelves 20. In this case, the display device on which the result image is displayed may be a display device connected to a terminal installed in the manufacturer's office or elsewhere. The terminal installed in the staff room or office may be, for example, a PC.

[0100] If the display device that shows the result image is not connected to the information processing device 2000, the information processing device 2000 transmits the result image to the terminal to which that display device is connected. Therefore, the information processing device 2000 needs to be able to identify the destination of the result image.

[0101] There are various methods for identifying the destination of the resulting image. For example, information identifying the destination of the resulting image can be stored in advance in a storage device accessible from the information processing device 2000. In this case, the information processing device 2000 identifies the destination of the resulting image by accessing the storage device.

[0102] In addition, for example, if the information processing device 2000 processes a request sent from a mobile terminal 3000, the information processing device 2000 sends the result image to the source of the request. Alternatively, the destination for the result image may be included in the request. In this case, the information processing device 2000 sends the result image to the destination indicated in the request.

[0103] <About the result image> The specific content of the resulting images varies. Examples of variations in the resulting images are provided below.

[0104] For example, the generation unit 2060 generates a result image by superimposing superimposed data representing the comparison result between actual display information and standard display information onto the shelf image 12. Figure 11 is the first figure illustrating a result image. In the result image 80 of Figure 11, superimposed data showing the comparison result is superimposed on the shelf image 12. Superimposed data 90 (a rectangle with a dot pattern) is superimposed on the image area of ​​products that match the criteria indicated by the standard display information. Superimposed data 92 (a rectangle with a diagonal line pattern) is superimposed on the image area of ​​products that do not match the criteria indicated by the standard display information. Furthermore, among the image areas where no products are displayed, superimposed data 94 (a rectangle with a grid pattern) is superimposed on the image area that represents the position where the product should be displayed according to the standard display information (i.e., the image area that represents the position where the product is out of stock). For example, if the standard display information indicates that product A is located at the position of "second shelf, third column," but the product is not placed at this position, it can be said that the product is out of stock at this position.

[0105] Note that the superimposed data may use colors instead of patterns such as dots. For example, green, red, and yellow can be used instead of dots, diagonal lines, and grids, respectively.

[0106] Here, the superimposed data (superimposed data 92 or superimposed data 94 in Figure 11) that represents the portion that does not conform to the standards indicated by the standard display information may be highlighted by methods such as flashing.

[0107] Products that meet the criteria and those that do not may be identified not by the pattern or color of the superimposed data, but by the shape of the superimposed data. Figure 12 is a second figure illustrating the result image 80. In the result image 80 of Figure 12, superimposed data 96 (black circles) is superimposed on the image areas of products that do not meet the criteria indicated by the standard display information. In addition, superimposed data 97 (cross marks) is superimposed on the image areas that represent the locations where products are out of stock.

[0108] In addition, a heatmap may be used as superimposed data. Figure 13 is a third figure illustrating the resulting image 80. In Figure 13, one superimposed data 98 ​​is superimposed on the shelf image 12. The superimposed data 98 ​​is a heatmap in which the parts that do not match the standard display information are relatively dark in color, and the parts that do match the standard display information are relatively light in color.

[0109] The result image 80 may include not only the shelf image 12 on which superimposed data representing the comparison results is superimposed, but also strings representing shelf-specific matching rates. Figure 14 is an example of a result image 80 that includes shelf-specific matching rates. The result image 80 in Figure 14 includes strings 70 representing shelf-specific matching rates on top of the shelf image 12.

[0110] The result image 80 may include not only the shelf image 12 with superimposed data representing the comparison results, but also a list of the products displayed on the shelf image 12. Figure 15 is an example of a result image 80 that includes a list of products. In Figure 15, area 120 displays the shelf image 12 with superimposed data. On the other hand, area 130 displays the product list 140.

[0111] Here, when displaying the resulting image on a display device, the generation unit 2060 may change the display of area 120 in response to user input operations on the product list 140. For example, suppose the user selects a record in the product list 140. In this case, the comparison unit 2040 will highlight the image area in area 120 that corresponds to the product indicated by the selected record.

[0112] Figure 16 illustrates how the result image 80 changes in response to an operation on the product list 140. In Figure 16, it is assumed that the user has performed an input operation to select a record for product A on the result image 80 shown in Figure 15. In response to this input operation, superimposed data 160 is displayed on the result image 80. The superimposed data 160 is an enlarged version of the image area corresponding to product A extracted from the result image 80 in Figure 15, and is displayed in front of the shelf image 12. As a result, the image area corresponding to product A, which the user selected, appears to stand out and be emphasized. Note that the operation to select a record is not limited to touch operation, but may also be performed using a mouse or other means.

[0113] Conversely, the generation unit 2060 may change the display of the product list 140 in response to input operations performed by the user on the product images displayed in area 120. Specifically, when a user selects an image of a product, the record in the product list 140 that represents that product is highlighted.

[0114] Figure 17 illustrates how the resulting image 80 changes in response to an operation on the product image. In Figure 17, the background color of the record representing product A is changed when the user selects the image area 170 representing product A in area 120.

[0115] <How to update the display on your display device> If the display device that shows the result image 80 is connected to a terminal other than the information processing device 2000, input operations for the result image 80 are detected by that terminal. For example, if the result image 80 is displayed on the display device of a mobile terminal 3000, input operations for the result image 80 are detected by the mobile terminal 3000.

[0116] When an input operation on the result image 80 is detected by a terminal other than the information processing device 2000, various methods can be employed to update the display on the display device in response to that input operation. For example, the terminal that detects an input operation on the result image 80 transmits the content of that input operation (for example, information about the superimposed data that was the target of the input operation) to the information processing device 2000. The information processing device 2000 generates a result image 80 with updated content based on the received information and transmits the result image 80 to the terminal. The terminal then displays the received result image 80 on the display device. Through this series of steps, the result image 80 displayed on the display device is updated.

[0117] In addition, for example, a mobile terminal 3000 may be equipped with a function to update the display on the display device in response to input operations on the result image 80. In this case, when the terminal detects a user's input operation, it modifies the result image 80 itself without sending the content to the information processing device 2000. In this case, an application for viewing the result image 80 is pre-installed on the terminal. For example, in the case of the mobile terminal 3000, the aforementioned application 100 is equipped with a function for viewing the result image 80.

[0118] <Regarding items that could not be recognized from shelf image 12> Due to reasons such as some parts of shelf image 12 being unclear, it may be impossible to recognize some of the products included in shelf image 12. In this case, the actual display information may indicate the location where some unidentified product (unknown product) is placed.

[0119] If the location of an unknown product is indicated in the actual display information, it is preferable that the result image 80 shows the location of the unknown product. Specifically, the generation unit 2060 includes superimposed data representing the unknown product in the result image 80. Figure 18 is an example of a result image 80 that includes superimposed data representing an unknown product. In Figure 18, the superimposed data 110 represents an unknown product. That is, the result image 80 in Figure 18 indicates that the product at the location where the superimposed data 110 is superimposed has not been identified.

[0120] For unidentified items, it is preferable to re-image with camera 10. For example, the superimposed data 110 may be implemented as a button to invoke the camera function. In this case, for example, the resulting image 80 is displayed on the display device of the mobile terminal 3000 on which camera 10 is installed. In this case, application 100 detects that the user has pressed superimposed data 110 or otherwise, and accordingly activates a camera application to control camera 10. When the user operates the camera application and causes camera 10 to take an image, application 100 transmits the generated shelf image 12 to information processing device 2000. Information processing device 2000 then generates the actual display information again using the acquired shelf image 12.

[0121] Here, if the newly generated shelf image 12 clearly includes the previously unknown product, the newly generated actual display information will show the product name of the previously unknown product. The information processing device 2000 then compares the newly generated actual display information with the reference display information to generate comparison information again and updates the result image 80. In this case, the information processing device 2000 may generate comparison information only for the previously generated actual display information that was previously unknown, and merge it with the previously generated comparison information.

[0122] The method of using the superimposed data 110 representing the unknown product is not limited to allowing the camera 10 to re-image. For example, the superimposed data 110 may be configured to include information about the product that should be placed in the location of the unknown product. The display method of such product information may be configured similarly to the information shown in Figure 19 later (correction information 150-1 and correction information 150-2). In that case, the information processing device 2000 can achieve this by obtaining information about the product associated with the location of the unknown product from the standard display information. When the displayed product information is selected by the user, the information processing device 2000 may, in accordance with that selection, regenerate actual display information different from the actual display information based on the information about the selected product and the actual display information generated using the already acquired shelf image 12 (i.e., include information about the selected product in the actual display information).

[0123] <Displaying information to correct product displays> The resulting image 80 may include information representing the criteria indicated by the standard display information (i.e., information showing how to modify the display to meet the criteria) for parts that do not conform to the criteria indicated by the standard display information. Hereinafter, this information will be referred to as modification information.

[0124] Figure 19 is an example of correction information. The resulting image 80 in Figure 19 contains two pieces of correction information 150. By looking at correction information 150-1, it can be seen that the product placed in the location where superimposed data 92 is superimposed should be replaced with product A. Also, by looking at correction information 150-2, it can be seen that product B should be placed in the location where superimposed data 94 is superimposed.

[0125] Here, when the resulting image 80 is displayed on a display device, the generation unit 2060 may include the correction information 150 in the resulting image 80 from the beginning, or it may include the correction information 150 in the resulting image 80 at the time an input operation is made on the superimposed data (such as superimposed data 92 or superimposed data 94) that indicates that the standard is not met.

[0126] Furthermore, the generation unit 2060 may generate screen data showing detailed information about the product indicated in the correction information 150, in response to input operations performed on the correction information 150. Product information includes, for example, the product identifier, the group to which the product belongs, the manufacturer of the product, and the location where the product is stored (the location of the shelf in the warehouse and the position of the product on that shelf). For example, if a product is out of stock, it is necessary to bring the product from the storage location and replenish it. By displaying the storage location of the product, such product replenishment work can be made easier.

[0127] Here, the correction information 150 may be displayed for all products that do not meet the criteria indicated by the standard display information, or it may be displayed for some products. In the latter case, whether or not to display the correction information 150 for a product is determined based on the importance of that product. In this case, an importance level is set in advance for each product.

[0128] For example, the generation unit 2060 generates correction information 150 only for products whose importance is above a predetermined value and includes it in the resulting image 80. Alternatively, for example, the generation unit 2060 may generate correction information 150 only for a predetermined number of products that do not meet the criteria indicated by the standard display information and include it in the resulting image 80.

[0129] [Example hardware configuration] The hardware configuration of the computer implementing the information processing device 2000 of Embodiment 2 is shown, for example, in Figure 3, similar to Embodiment 1. However, the storage device 1080 of the computer 1000 implementing the information processing device 2000 of this embodiment further stores program modules that realize the functions of the information processing device 2000 of this embodiment.

[0130] The embodiments of the present invention have been described above with reference to the drawings, but these are merely examples of the present invention, and combinations of the above embodiments or various other configurations can also be adopted.

[0131] Some or all of the above embodiments may also be described as follows, but are not limited to the following: 1. Image analysis means that analyzes shelf images in which product shelves are captured and generates actual display information representing the display status of products on the product shelves, An information processing apparatus having comparison means for comparing the actual display information with standard display information representing the standards for displaying products on the product shelves, and generating comparison information representing the result of the comparison. 2. The standard display information indicates the identification information of each product to be displayed on the product shelf, The information processing device according to 1, wherein the image analysis means generates the actual display information indicating the identification information of each product displayed on the product shelf. 3. The information processing device according to 2., wherein the standard display information indicates at least one of the display position of the product on the product shelf, the number of faces, and the arrangement relationship with other products. 4. The information processing device according to any one of 1 to 3, wherein the comparison means calculates a conformance rate representing the degree to which the product display status conforms to the standards indicated by the standard display information and includes it in the comparison information. 5. The information processing apparatus according to any one of 1 to 3, wherein the comparison means obtains a manufacturer's identifier, calculates a matching rate representing the degree to which the display status of the manufacturer's products identified by the identifier conforms to the standards indicated by the standard display information, and includes it in the comparison information. 6. An information processing apparatus according to any one of 1 to 5, comprising a generation means for generating a result image representing a product that does not conform to the criteria indicated by the standard display information, using the comparison information. 7. The information processing device according to 6, wherein the resulting image is displayed in a manner that allows for the identification of products that do not conform to the standards indicated by the standard display information and products that conform to the standards indicated by the standard display information. 8. The information processing apparatus according to 6. or 7., wherein the generating means includes in the resulting image correction information indicating the location where the product should be displayed, or the location where the product is currently displayed, for products that do not conform to the standards indicated by the standard display information.

[0132] 9. An information processing system having an information processing device and a mobile terminal, The mobile terminal transmits the shelf image, in which the product shelf is captured, to the information processing device. The aforementioned information processing device is Image analysis means that receives the transmitted shelf image, performs image analysis on the shelf image, and generates actual display information representing the display status of products on the product shelf, A comparison means that compares the actual display information with standard display information representing the standards for displaying products on the product shelves, and generates comparison information representing the result of the comparison, The system includes a generation means that generates a result image representing products that do not conform to the standards indicated by the aforementioned standard display information, and transmits it to the mobile terminal. The aforementioned mobile terminal is an information processing system that receives the result image and displays it on a display device. 10. The information processing system according to 9, wherein the standard display information generates the actual display information which indicates at least one of the display position of the product on the product shelf, the number of faces, and the arrangement relationship with other products. 11. The information processing system described in 10., wherein the standard display information indicates the display position, number of faces, or arrangement relationship of the product with other products on the product shelf. 12. The information processing system according to any one of 9 to 11, wherein the comparison means calculates a conformity rate representing the degree to which the product display status conforms to the standards indicated by the standard display information and includes it in the comparison information. 13. The information processing system according to any one of 9 to 11, wherein the comparison means obtains a manufacturer's identifier, calculates a matching rate representing the degree to which the display status of the manufacturer's products identified by the identifier conforms to the standards indicated by the standard display information, and includes it in the comparison information. 14. The information processing system according to any one of 9. to 13., wherein the resulting image is displayed in a manner that allows for the identification of products that do not conform to the standards indicated by the standard display information and products that conform to the standards indicated by the standard display information. 15. The information processing system according to any one of 9 to 14, wherein the generation means includes in the resulting image corrective information indicating the location where the product should be displayed, or the location where the product is currently displayed, for products that do not conform to the standards indicated by the standard display information.

[0133] 16. A control method performed by a computer, Image analysis step: Image analysis of shelf images in which product shelves are captured to generate actual display information representing the product display situation on the product shelves. A control method comprising: a comparison step of comparing the actual display information with standard display information representing the standards for displaying products on the product shelf, and generating comparison information representing the result of the comparison. 17. The aforementioned standard display information indicates the identification information of each product to be displayed on the product shelf, The control method according to 16, wherein in the image analysis step, the actual display information indicating the identification information of each product displayed on the product shelf is generated. 18. The control method according to 17, wherein the reference display information indicates at least one of the display position of the product on the product shelf, the number of faces, and the arrangement relationship with other products. 19. The control method according to any one of paragraphs 16 to 18, wherein in the comparison step, a conformance rate representing the degree to which the product display status conforms to the criteria indicated by the standard display information is calculated and included in the comparison information. 20. The control method according to any one of paragraphs 16 to 18, wherein in the comparison step, an identifier of the manufacturer is obtained, and a matching rate representing the degree to which the display status of the manufacturer's products identified by the identifier conforms to the criteria indicated by the standard display information is calculated and included in the comparison information. 21. The control method according to any one of paragraphs 16 to 20, further comprising a generation step of generating a result image representing a product that does not conform to the criteria indicated by the standard display information, using the comparison information. 22. The control method according to 21, wherein the resulting image is displayed in a manner that allows for the identification of products that do not conform to the standards indicated by the standard display information and products that conform to the standards indicated by the standard display information. 23. The control method according to 21. or 22., wherein, in the generation step, for products that do not conform to the criteria indicated by the standard display information, correction information representing the location where the product should be displayed, or the product that should be displayed in the location where the product is currently displayed, is included in the resulting image.

[0134] 24. A program that causes a computer to perform each step of the control method described in any one of sections 16 to 23.

[0135] This application claims priority based on Japanese Patent Application No. 2017-191790, filed on 29 September 2017, and incorporates all of its disclosures herein.

Claims

1. Image analysis means that analyzes shelf images in which product shelves are captured and generates actual display information representing the product display situation on the product shelves, A comparison means that compares the actual display information with standard display information representing the standards for displaying products on the product shelves, and generates comparison information representing the result of the comparison, The system includes a generation means that generates a result image in which products that do not conform to the standards indicated by the standard display information and products that are out of stock can be identified using the comparison information, The generation means generates a result image by superimposing superimposed data of different shapes on the areas of the shelf image corresponding to products that do not conform to the criteria indicated by the standard display information and the areas corresponding to out-of-stock products. The generation means is an information processing device that generates the result image representing an unknown product that was not recognized by the image analysis performed by the image analysis means.

2. Image analysis means that analyzes shelf images in which product shelves are captured and generates actual display information representing the product display situation on the product shelves, A comparison means that compares the actual display information with standard display information representing the standards for displaying products on the product shelves, and generates comparison information representing the result of the comparison, The system includes a generation means that generates a result image in which products that do not conform to the standards indicated by the standard display information and products that are out of stock can be identified using the comparison information, The generation means generates a result image by superimposing superimposed data of different shapes on the areas of the shelf image corresponding to products that do not conform to the criteria indicated by the standard display information and the areas corresponding to out-of-stock products. The comparison means is an information processing device that calculates a matching rate, which represents the degree to which the display status of the products on the product shelf matches the standards for each product on the product shelf shown in the standard display information, and includes this rate in the comparison information.

3. The information processing apparatus according to claim 2, wherein the generation means generates the resulting image by superimposing superimposed data using a heatmap onto the shelf image, wherein the color of the parts that do not conform to the standards indicated by the standard display information is relatively dark, and the color of the parts that conform to the standard display information is relatively light.

4. The information processing apparatus according to claim 2 or 3, wherein the result image includes a string representing the matching rate for each product shelf and a shelf image on which superimposed data representing the comparison result is superimposed.

5. Image analysis means that analyzes shelf images in which product shelves are captured and generates actual display information representing the product display situation on the product shelves, A comparison means that compares the actual display information with standard display information representing the standards for displaying products on the product shelves, and generates comparison information representing the result of the comparison, The system includes a generation means that generates a result image in which products that do not conform to the standards indicated by the standard display information and products that are out of stock can be identified using the comparison information, The generation means generates a result image by superimposing superimposed data of different shapes on the areas of the shelf image corresponding to products that do not conform to the criteria indicated by the standard display information and the areas corresponding to out-of-stock products. The comparison means is an information processing device that obtains a manufacturer's identifier, calculates a second matching rate representing the degree to which the display status of the manufacturer's products identified by the identifier matches the standards for each product on the product shelf shown in the standard display information, and includes the second matching rate in the comparison information.

6. Image analysis means that analyzes shelf images in which product shelves are captured and generates actual display information representing the product display situation on the product shelves, A comparison means that compares the actual display information with standard display information representing the standards for displaying products on the product shelves, and generates comparison information representing the result of the comparison, The system includes a generation means that generates a result image in which products that do not conform to the standards indicated by the standard display information and products that are out of stock can be identified using the comparison information, The generation means generates a result image by superimposing superimposed data of different shapes on the areas of the shelf image corresponding to products that do not conform to the criteria indicated by the standard display information and the areas corresponding to out-of-stock products. The generation means is an information processing device that includes in the result image correction information representing the corrected position where the product should be displayed, or the product that should be displayed in the current position where the product is displayed, for products that do not conform to the standards indicated by the standard display information.

7. The information processing device according to any one of claims 1 to 6, wherein the superimposed data displayed over the area of ​​the missing product is an "X" mark, and the superimposed data displayed over the area of ​​the product that does not meet the criteria is a mark of a different shape from the "X" mark.

8. The information processing apparatus according to any one of claims 1 to 7, wherein the generation means generates a result image by superimposing superimposed data of different patterns or colors on areas of the shelf image corresponding to products that do not conform to the criteria indicated by the standard display information and products that are out of stock.

Images