Information processing device, information processing method, and program

Abstract

To provide a technique for easily and accurately determining whether shelf labels are correctly associated with commodities in commodity shelves of a store.SOLUTION: An information processor includes a commodity information acquisition unit, a shelf label information acquisition unit, a first determination unit and an output unit. The commodity information acquisition unit analyzes an image in which commodities and shelf labels are captured to acquire an order of arrangement of the commodities. The shelf label information acquisition unit analyzes the image to acquire the order of arrangement of the shelf labels. The first determination unit determines whether or not the acquired order of arrangement of the commodities coincides with the order of arrangement of the shelf labels. The output unit outputs information indicating a result of the determination.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for analyzing shelf allocation operations in stores and the like. [Background technology]

[0002] In a store, shelf tags are placed near products displayed on shelves, displaying information about the products, such as the product name and price. If the shelf tags are not correctly associated with the products, problems can occur in which customers of the store misinterpret the product's price, etc.

[0003] An example of a technology for checking whether products on a product shelf are correctly associated with shelf labels is disclosed in Patent Documents 1 and 2 below. Patent Document 1 below discloses a technology in which related information indicating the correspondence between product information and electronic shelf labels is prepared in advance, and whether the correspondence between products and electronic shelf labels determined based on the positions of products in an image and the positions of electronic shelf labels matches the correspondence in the related information. Furthermore, Patent Document 2 below discloses a technology for determining whether there is an error in the relationship between the number of product types recognized from an image and the number of shelf labels, or in the relationship between the positions of products for each type recognized from the image and the positions of the shelf labels recognized from the image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2016 / 052379 [Patent Document 2] International Publication No. 2016 / 052383 Summary of the Invention [Problem to be solved by the invention]

[0005] The technology of Patent Document 1 requires that correspondences between product information and electronic shelf labels must be prepared in advance for all products displayed on the shelves, which is time-consuming. Also, the technology of Patent Document 2 cannot detect errors in correspondences, such as when the correspondences of positions and numbers are correct but the product information written on the shelf labels is incorrect.

[0006] The present invention has been made in view of the above problems, and an object of the present invention is to provide a technology for easily and accurately determining whether or not a shelf tag is correctly associated with a product on a product shelf in a store. [Means for solving the problem]

[0007] The information processing device according to the present disclosure includes: a product information acquiring means for acquiring the arrangement order of the products by analyzing an image of the products and the shelf tags; a shelf label information acquisition means for acquiring the arrangement order of the shelf labels by analyzing the image; a first determination means for determining whether or not the arrangement order of the products in the image matches the arrangement order of the shelf tags based on the acquired arrangement order of the products and the arrangement order of the shelf tags; an output means for outputting information indicating a result of the determination; Equipped with.

[0008] The information processing method according to the present disclosure includes: The computer Acquire the arrangement order of the products by analyzing the captured image of the products and the shelf tags; The image is analyzed to obtain the arrangement order of the shelf labels; determining whether or not the arrangement order of the products in the image matches the arrangement order of the shelf tags based on the acquired arrangement order of the products and the arrangement order of the shelf tags; outputting information indicating the result of the determination; This includes:

[0009] The program in this disclosure is Computer, A product information acquisition means for acquiring the arrangement order of the products by analyzing an image of the products and the shelf tags; a shelf label information acquisition means for acquiring the arrangement order of the shelf labels by analyzing the image; a first determination means for determining whether or not the arrangement order of the products in the image matches the arrangement order of the shelf labels based on the acquired arrangement order of the products and the arrangement order of the shelf labels; an output means for outputting information indicating the result of the determination; Function as. Effect of the Invention

[0010] According to the present invention, it is possible to easily and accurately determine whether or not shelf tags are correctly associated with products on product shelves in a store. [Brief description of the drawings]

[0011] The above objects, as well as other objects, features and advantages, will become more apparent from the following preferred embodiments and the accompanying drawings.

[0012] [Figure 1] 1 is a block diagram conceptually showing a functional configuration of an information processing device in a first embodiment. [Diagram 2] FIG. 1 is a diagram illustrating a configuration of a computer that realizes an information processing device. [Diagram 3] 10 is a flowchart showing an example of the operation of the information processing device according to the second embodiment. [Figure 4] FIG. 2 is a diagram showing an example of an image to be processed. [Diagram 5] 13 is a flowchart showing an operation example of the information processing device according to the third embodiment. [Figure 6] FIG. 13 is a block diagram showing a functional configuration of an information processing device according to a fourth embodiment. [Figure 7] 13 is a flowchart showing an example of the operation of the information processing device according to the fourth embodiment. [Figure 8] 13 is a flowchart showing another example of the operation of the information processing device according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In all drawings, similar components are given similar reference numerals and their description will be omitted as appropriate. In addition, unless otherwise specified, in each block diagram, each block represents a functional configuration, not a hardware configuration.

[0014] [First embodiment] [Functional configuration] 1 is a block diagram conceptually illustrating a functional configuration of an information processing device 10 in the first embodiment. As shown in FIG. 1, the information processing device 10 in the present embodiment has a product information acquisition unit 110, a shelf label information acquisition unit 120, a first determination unit 130, and an output unit 140.

[0015] The product information acquisition unit 110 acquires the position of each product by analyzing an image (hereinafter, referred to as a "processing target image") in which one or more products and one or more shelf tags are captured. The product information acquisition unit 110 may further acquire product information associated with each product by analyzing the processing target image. The product information includes information for identifying the product (product identification information). The shelf label information acquisition unit 120 acquires the position of each shelf label by analyzing the processing target image. The shelf label information acquisition unit 120 may further acquire product information (shelf label product information) written on each shelf label by analyzing the processing target image. Here, the shelf label may be a shelf label made of an analog medium such as paper, or may be a so-called electronic shelf label made of an electronic medium. The first determination unit 130 determines the consistency of the correspondence between the product and the shelf label based on the relationship between the product position acquired by the product information acquisition unit 110 and the shelf label position acquired by the shelf label information acquisition unit 120. When the product information acquisition unit 110 further acquires product information associated with each product, and the shelf label information acquisition unit 120 further acquires shelf label product information, the first determination unit 130 may further determine the consistency of the correspondence between the products and the shelf labels based on the relationship between the product information acquired by the product information acquisition unit 110 and the shelf label product information acquired by the shelf label information acquisition unit 120. The output unit 140 outputs the determination result regarding the consistency of the correspondence between the products and the shelf labels.

[0016] [Hardware configuration] Each functional component of the information processing device 10 may be realized by hardware that realizes each functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a further description will be given of the case where each functional component of the information processing device 10 is realized by a combination of hardware and software.

[0017] The computer 1000 may be any of various computers. For example, the computer 1000 may be a PC (Personal Computer), a server machine, a tablet terminal, a smartphone, etc. The computer 1000 may be a dedicated computer designed to realize the information processing device 10, or may be a general-purpose computer.

[0018] FIG. 2 is a diagram illustrating a configuration of a computer 1000 that realizes the information processing device 10. The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. However, the method of connecting the processor 1040 and the like to each other is not limited to bus connection. The processor 1040 is a processor realized by a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), etc. The memory 1060 is a main storage device realized by a RAM (Random Access Memory), etc. The storage device 1080 is an auxiliary storage device realized by a HDD (Hard Disk Drive), an SSD (Solid State Drive), a memory card, or a ROM (Read Only Memory), etc.

[0019] The input / output interface 1100 is an interface for connecting the computer 1000 to an input / output device. For example, the input / output interface 1100 is connected to an input device such as a keyboard and an output device such as a display device.

[0020] The network interface 1120 is an interface for connecting the computer 1000 to a network. This network is, for example, a local area network (LAN) or a wide area network (WAN). The method for connecting the network interface 1120 to the network may be a wireless connection or a wired connection.

[0021] The storage device 1080 stores program modules that realize each function of the information processing device 10 (such as the product information acquisition unit 110, the shelf label information acquisition unit 120, the first determination unit 130, and the output unit 140). The processor 1040 executes each of these program modules to realize each function corresponding to the program module.

[0022] As described above, in this embodiment, when determining the consistency of the correspondence between a product and a shelf label, the first determination unit 130 uses the relationship between the product position and the shelf label position, or the relationship between the product position and the shelf label position, and the relationship between the product shown in the image and the product written on the shelf label shown in the image. With this configuration, it is possible to determine whether the product and the shelf label are correctly associated without performing a correspondence process in advance. Furthermore, with this configuration, it is possible to accurately determine whether the product and the shelf label are correctly associated.

[0023] [Second embodiment] In this embodiment, the information processing device 10 of the first embodiment will be described in more detail.

[0024] [Functional and hardware configuration] The information processing device 10 of this embodiment has the same functional configuration (e.g., FIG. 1) and hardware configuration (e.g., FIG. 2) as those of the first embodiment. The information processing device 10 of this embodiment acquires an image to be processed from an external device connected via an input / output interface 1100 or a network interface 1120. Although not particularly limited, the external device may be a tablet terminal or smartphone of a retail store staff member or manufacturer staff member, or a surveillance camera in a store. For example, the information processing device 10 communicates with a tablet terminal or smartphone of a retail store staff member or manufacturer staff member via the network interface 1120 to acquire an image to be processed.

[0025] [Example of operation] An operation example of the information processing device 10 will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an operation example of the information processing device 10 according to the second embodiment.

[0026] <First operation example> First, the information processing device 10 acquires a processing target image (e.g., FIG. 4) (S102). FIG. 4 is a diagram showing an example of the processing target image. In the processing target image shown in FIG. 4, the reference numeral 20 indicates a product displayed on a product shelf. In the processing target image shown in FIG. 4, the reference numeral 30 indicates a shelf tag placed on the product shelf. The processing target image shown in FIG. 4 is generated by, for example, a mobile terminal such as a tablet terminal or a smartphone, or a surveillance camera in a store. For example, a retail store staff member or a manufacturer staff member uses a camera function of a mobile terminal to capture an image of a desired area of ​​a product shelf, thereby generating the processing target image shown in FIG. 4. For example, the information processing device 10 can acquire the processing target image by communicating with a tablet terminal or a smartphone of the retail store staff member or the manufacturer staff member. However, if the information processing device 10 has an imaging function, the information processing device 10 may generate the processing target image. If the processing target image is stored in a storage device (external server, etc.) not shown, the information processing device 10 can acquire the processing target image by communicating with the storage device.

[0027] Then, the product information acquisition unit 110 analyzes the processing target image and acquires product information and product position information (hereinafter referred to as "product position information") of the product appearing in the processing target image (S104).

[0028] Specifically, the product information acquisition unit 110 analyzes the processing target image to extract an area representing a product in the processing target image (hereinafter also referred to as a "product area"). The product information acquisition unit 110 can extract the product area by utilizing various known object recognition techniques. Furthermore, when multiple products are shown in the processing target image, the product information acquisition unit 110 extracts a product area for each product. The product information acquisition unit 110 analyzes the extracted product area and acquires product information of the product represented by the product area and position information of the product.

[0029] For example, the product information acquisition unit 110 can recognize which product each product area represents by performing pattern matching based on the image feature amount extracted from each product area. In this case, a table in which the image feature amount of each product is associated with the product information of each product is prepared in the memory 1060 or the storage device 1080. As an example, the product information on the product may include at least any of the following information: the name (product name) of each product, an identifier for identifying each product (e.g., JAN (Japanese Article Number) code, etc.), the name of the group to which the product belongs (e.g., brand name), the product category to which the product belongs, the product content (e.g., 350 ml), the product size, the standard price of each product, and the manufacturer (manufacturing manufacturer or sales manufacturer). Here, the standard price means a standard price when determining the price of each product in a store, such as a suggested retail price, or a set price of a product in a nearby store or another affiliated store. The standard price may also be information with a range (information indicating a price range). For example, information indicating the price range for each product may be stored in the memory 1060 or the like, and the product information acquisition unit 110 may read out the information indicating the price range of the product. Information indicating a range for determining the price range may be stored in the memory 1060 or the like, and the product information acquisition unit 110 may calculate the price range of the product based on the price of the product obtained based on the result of image processing and the information indicating the range. This range may be changed depending on the price of the product. The product information acquisition unit 110 may also acquire multiple set prices of the product in the store or other affiliated stores, and determine the price range of the product based on the highest price and the lowest price. The product information acquisition unit 110 identifies an image feature that matches or is most similar to the image feature extracted from the product area from among the multiple image feature values ​​stored in the table. Then, the product information acquisition unit 110 acquires product information associated with the identified image feature value.

[0030] Furthermore, the product information acquisition unit 110 can acquire the position of the product on the product shelf as product position information. The position of the product is expressed, for example, by a combination of the position in the row direction (e.g., which row from the top row the product is located in) and the position in the column direction (e.g., which row from the leftmost row the product is located in). Here, existing technology can be used as a technology for identifying the position of each product from an image including a product shelf. Note that, when products of the same type (products with the same product name) are arranged in multiple positions, the product information acquisition unit 110 acquires multiple positions for one product. Note that the process of identifying the position is not limited to this. For example, the product information acquisition unit 110 can acquire the coordinate value of the product area identified for each product on the processing target image as product position information.

[0031] Here, there may be cases where multiple identical products are displayed horizontally. In such a case, the product information acquisition unit 110 identifies an area (hereinafter referred to as a "product group area") that represents adjacent identical products, and acquires the coordinates of the product group area on the processing target image as position information of one product. The product information acquisition unit 110 can determine "adjacent identical products" based on, for example, "whether or not the image feature amount extracted from each product area or the product information obtained using the image feature amount is identical or similar." The product information acquisition unit 110 can also obtain the number of products with identical product information, that is, the number of faces from the identified product group area.

[0032] In the above description, the product information acquisition unit 110 acquires the coordinates of the product group area on the processing target image as the position information of one product, but the configuration of the product information acquisition unit 110 is not limited to this. For example, the product information acquisition unit 110 may acquire the coordinates of each product included in the product group area on the processing target image.

[0033] Furthermore, multiple types of products (variation products) that are the same item but have different attributes (e.g., flavor, color, etc.) may be placed adjacent to each other. In such a case, the product information acquisition unit 110 may recognize these multiple types of products as different products from each other, or may recognize them as the same product. In the latter case, for example, feature quantities of multiple types of products to be recognized as the same product are stored in association with each other in one record of the above-mentioned table. Then, the product information acquisition unit 110 can specify, for example, an area (product group area) that represents a group of adjacent variation products by using this association. Then, the product information acquisition unit 110 acquires the coordinates of the product group area on the processing target image as position information of one product. However, the product information acquisition unit 110 may acquire position information of each product among multiple types of products that have different attributes.

[0034] Similarly, the shelf label information acquisition unit 120 analyzes the image to be processed and acquires product information (hereinafter referred to as "shelf label product information") and shelf label position information (hereinafter referred to as "shelf label position information") written on the shelf label appearing in the image to be processed (S106).

[0035] Specifically, the shelf label information acquisition unit 120 extracts an area representing a shelf label (hereinafter referred to as a "shelf label area") by analyzing the processing target image. If multiple shelf labels are shown in the processing target image, the shelf label information acquisition unit 120 extracts a shelf label area for each shelf label. The shelf label information acquisition unit 120 analyzes the extracted shelf label area and acquires product information written on the shelf label represented by the shelf label area and position information of the shelf label.

[0036] For example, the shelf label information acquisition unit 120 can acquire product information (such as product name, product description, and product price) written on the shelf label by analyzing the shelf label area using OCR (Optical Character Recognition) technology or the like. Furthermore, the shelf label may contain a barcode or two-dimensional code (such as a Japan Article Number (JAN) code or a Price Look Up (PLU) code) that encodes identification information (product ID) for identifying each product. In this case, the shelf label information acquisition unit 120 can analyze the area in which the barcode or two-dimensional code is displayed and acquire the encoded identification information as shelf label product information. Note that, if the barcode or two-dimensional code is a Non-PLU code, the shelf label information acquisition unit 120 can further acquire the price of the product by analyzing the area in which the barcode or two-dimensional code is displayed. Furthermore, the shelf label information acquisition unit 120 can acquire the coordinate value on the processing target image of the shelf label area identified for each shelf label as shelf label position information.

[0037] Then, the first determination unit 130 associates the product with the shelf label based on the product position information acquired by the product information acquisition unit 110 and the shelf label position information acquired by the shelf label information acquisition unit 120 (S108).

[0038] As an example, the first determination unit 130 can associate products with shelf labels as follows. First, the first determination unit 130 determines whether a product is located within a predetermined threshold d from a certain shelf label based on a distance calculated using product position information and shelf label position information. th Identify products located within a distance of a given threshold d th is an arbitrary value indicating the allowable range of positional deviation between a product and a shelf label, and is stored in advance in the memory 1060 or the like. Then, the first determination unit 130 associates the identified product with the shelf label. th When there are multiple products within this distance, the first determination unit 130 may associate the closest product with the shelf label.

[0039] Then, the first determination unit 130 determines whether the association between the product and the shelf label meets a criterion (S110). In other words, the first determination unit 130 determines the consistency of the association between the product and the shelf label. As an example, the first determination unit 130 determines that the criterion is not met (there is no consistency) when there is a product that is not associated with a shelf label (i.e., when there are not enough shelf labels). As another example, the first determination unit 130 determines that the criterion is not met (there is no consistency) when there is a shelf label that is not associated with a product (i.e., when there are excess shelf labels). As another example, the first determination unit 130 determines that the criterion is not met (there is no consistency) when multiple shelf labels with different shelf label product information (multiple shelf labels with different written content) are associated with one product. As another example, the first determination unit 130 determines that the criterion is not met (there is no consistency) when multiple products of different types are associated with one shelf label. When the association between the product and the shelf label in the process of S108 does not fall into any of the above-mentioned examples, the first determination unit 130 determines that the criteria are met (there is consistency, i.e., the product and the shelf label are correctly associated). Note that the operation of the first determination unit 130 is not limited to the examples given here.

[0040] In addition, in the process shown in S110, the first determination unit 130 may further determine whether or not the mutually associated product and the shelf tag have the same product information (i.e., whether or not the product information written on the shelf tag is incorrect). In this case, the first determination unit 130 determines that the criteria are not met when the mutually associated product and the shelf tag do not have the same product information. In this way, the first determination unit 130 can detect that an incorrect shelf tag is attached to a certain product. As a specific example, the first determination unit 130 determines whether or not the product price included in the shelf label product information acquired by the shelf label information acquisition unit 120 is included in the price range of the standard price of the product included in the product information acquired by the product information acquisition unit 110. For example, when the product price included in the shelf label product information significantly deviates from the standard price (price range) included in the product information, the first determination unit 130 can determine that an incorrect shelf tag is attached to a certain product (there is no consistency). In this case, the first determination unit 130 can also determine that an abnormal price is set for a certain product (there is no consistency).

[0041] If the first determination unit 130 determines that the criteria are not met (S110: NO), the output unit 140 outputs information indicating the determination content by the first determination unit 130 to a mobile terminal owned by a person in charge at the store's headquarters or a person in charge at the manufacturer, or to a terminal installed in the store (S112). For example, the output unit 140 displays a screen on the display of the output destination terminal in which the determination results for each combination of product and shelf tag associated in the process of S108 (for example, correct association, missing shelf tag, excess shelf tag, incorrect product information written on the shelf tag, etc.) are superimposed on the processing target image. That is, the output unit 140 displays the determination results on a presentation unit such as a display in a manner that is recognizable by the user.

[0042] <Second operation example> In the second operation example of the information processing device 10, steps S102 to S106 are the same as those in the first operation example. In the association based on the position information described in S108 of the first operation example, the first determination unit 130 associates products and shelf labels having the same product information. Then, the first determination unit 130 judges whether the association between the product and the shelf label satisfies a criterion (S110). In other words, the first determination unit 130 judges the consistency of the association between the product and the shelf label. As an example, when there is a product that is not associated with a shelf label (i.e., when there are not enough shelf labels), the first determination unit 130 judges that the criterion is not met (there is no consistency). As another example, when there is a shelf label that is not associated with a product (i.e., when there are excess shelf labels), the first determination unit 130 judges that the criterion is not met (there is no consistency). As another example, the first determination unit 130 determines that the criterion is not met (there is no consistency) when the difference between the position information of the associated product and shelf tag is equal to or greater than a criterion (when the shelf tag is misaligned). The first determination unit 130 determines that the criterion is met (there is consistency, i.e., the product and shelf tag are correctly associated) when the association between the product and shelf tag in the process of S108 does not fall into any of the above-mentioned examples. Note that the operation of the first determination unit 130 is not limited to the examples given here.

[0043] If the first determination unit 130 determines that the criteria are not met (S110: NO), the output unit 140 outputs information indicating the determination content by the first determination unit 130 to a mobile terminal owned by a person in charge at the store's headquarters or a person in charge at the manufacturer, or to a terminal installed in the store (S112). For example, the output unit 140 displays a screen on the display of the output destination terminal in which the determination results for each combination of products and shelf tags associated in the process of S108 (for example, correct association, missing shelf tags, excess shelf tags, misaligned shelf tags, etc.) are superimposed on the image to be processed.

[0044] As described above, according to this embodiment, the information processing device 10 determines the position of the product by image processing, and also determines the position of the shelf label. Then, if the relationship between these positions does not satisfy a criterion (for example, if the distance is greater than a certain amount), information indicating this is output. Therefore, errors in the correspondence between the product and the shelf label can be detected with high accuracy.

[0045] [Third embodiment] The functional configuration of the information processing device 10 according to this embodiment is similar to the functional configuration of the information processing device 10 according to the first or second embodiment.

[0046] [Example of operation] FIG. 5 is a flowchart showing an example of the operation of the information processing device 10 according to the third embodiment. Among the processes shown in this figure, the processes shown in S102 to S106 are the same as the processes shown in FIG. 3. Then, the first determination unit 130 identifies the order of the product information acquired by the product information acquisition unit 110 and the order of the shelf label product information acquired by the shelf label information acquisition unit 120 (S120). Then, the first determination unit 130 determines whether or not these two orders match (S122). Here, if the order of the product information and the order of the shelf label product information do not match, this means at least one of the following states: the order of the shelf labels is correct but the order of the products is incorrect, or the order of the products is correct but the order of the shelf labels is incorrect. Therefore, if the first judgment unit 130 judges that the two orders do not match (S122: No), the output unit 140 outputs information indicating that fact to a mobile terminal owned by a person in charge at the retail store's headquarters or a person in charge at the manufacturer, or to a terminal installed at the retail store (S124).

[0047] As described above, in this embodiment, an error in the correspondence between products and shelf labels can be accurately detected based on the difference between the arrangement order of product information and the arrangement order of shelf label product information.

[0048] [Fourth embodiment] [Functional configuration] 6 is a block diagram showing a functional configuration of an information processing device 10 according to the fourth embodiment. The information processing device 10 according to this embodiment has the same functional configuration as the information processing device 10 according to the first to third embodiments, except that it has a second determination unit 150 (second determination means and third determination means).

[0049] The second determination unit 150 determines whether or not the survey target product is present among the products recognized from the processing target image. The survey target product is, for example, a retail store's key product, a company's main product, or a competing product of another manufacturer. The second determination unit 150 can determine whether or not the survey target product is present in the processing target image by using information for identifying the survey target product, such as a product ID that uniquely identifies the survey target product or a name of the survey target product. The information for identifying the survey target product is stored in advance in the memory 1060 or the storage device 1080 by a preparation operation such as pre-inputting a list of the survey target products. In this case, the second determination unit 150 can acquire information for identifying the survey target product from the memory 1060 or the storage device 1080. Alternatively, the second determination unit 150 may be configured to display the processing target image on a display of a mobile terminal of a retail store staff member or a manufacturer staff member, and have the retail store staff member or the manufacturer staff member select the survey target product. In this case, the second determination section 150 can acquire the product information of the product selected on the image as "information for identifying the survey target product."

[0050] [Hardware configuration] The information processing device 10 of this embodiment has the same hardware configuration (e.g., FIG. 2) as the first and second embodiments. The storage device 1080 of this embodiment further stores a program module that realizes the function of the above-mentioned second determination unit 150. The processor 1040 executes the program module, thereby realizing the function of the above-mentioned second determination unit 150.

[0051] [Example of operation] The information processing device 10 according to this embodiment performs at least one of the processes shown in the first embodiment and the second embodiment described above, and also performs the process shown in Fig. 7. Fig. 7 is a flowchart showing an example of the operation of the information processing device 10 according to the fourth embodiment.

[0052] In the process shown in FIG. 7, first, the second determination unit 150 acquires information for identifying the survey target product (for example, at least one of product information and image feature amount) (S302). For example, the second determination unit 150 can read out product information of the survey target product from a storage area such as the memory 1060 or the storage device 1080. Then, the second determination unit 150 uses the information for identifying the survey target product to determine whether or not the survey target product exists among the products recognized from the processing target image (S304). Specifically, when the second determination unit 150 acquires the product information of the survey target product acquired in S302, the second determination unit 150 compares the product information of the survey target product with the product information acquired by the product information acquisition unit 110. In other words, the second determination unit 150 determines whether or not the product information acquired by the product information acquisition unit 110 includes product information that matches the product information of the survey target product. If product information that matches the product information of the survey target product exists in the product information acquired by the product information acquisition unit 110 (S304: YES), the second determination unit 150 determines that the survey target product exists in the processing target image. In this case, the second determination unit 150 acquires price information written on the shelf tag associated with the product determined to be the survey target product (S306). The price information acquired here is acquired, for example, from the shelf tag product information shown in FIG. 3 or FIG. 5.

[0053] Then, the output unit 140 outputs the price information acquired in the process of S306 (S308). For example, the output unit 140 can generate drawing data for superimposing the price information acquired in the process of S306 on the product determined to be the "survey target product" based on the processing target image, and output the drawing data on the display of the target terminal. In the above description, the second determination unit 150 has been described as acquiring information for identifying the survey target product, but is not limited thereto. For example, in the following description, the output unit 140 outputs the recognition result (recognized product) recognized by the product information acquisition unit 110 on the display. In response to receiving an input operation (i.e., an operation to select the survey target product) by the user on the display, the second determination unit 150 may use the information on the survey target product to determine whether the survey target product is present among the products recognized from the processing target image.

[0054] Furthermore, the information processing device 10 according to the present embodiment may perform the process shown in Fig. 8 instead of or in addition to the process shown in Fig. 8. Fig. 8 is a flowchart showing another example of the operation of the information processing device 10 according to the fourth embodiment.

[0055] In the process shown in FIG. 8, first, the second determination unit 150 acquires information for identifying the survey target product (for example, at least one of product information and image feature amount) (S402). For example, the second determination unit 150 can read out the product information of the survey target product from a storage area such as the memory 1060 or the storage device 1080. Then, the second determination unit 150 uses the information for identifying the survey target product to determine whether or not the survey target product exists among the products recognized from the processing target image (S404). Specifically, when the second determination unit 150 acquires the product information of the survey target product acquired in S302, the second determination unit 150 compares the product information of the survey target product with the product information acquired by the product information acquisition unit 110. When product information matching the product information of the survey target product exists in the product information acquired by the product information acquisition unit 110 (S404: YES), the second determination unit 150 determines that the survey target product exists in the processing target image. In this case, the second determination unit 150 identifies other products located around the survey target product and acquires the price information written on the shelf label associated with the product (S406). The price information acquired here is acquired, for example, from the shelf label product information shown in FIG. 3 or FIG. 5, similar to the process shown in FIG. 7.

[0056] Then, the output unit 140 outputs the price information acquired in the process of S406 (S408). For example, the output unit 140 can generate drawing data for superimposing the price information acquired in S406 on each of the products identified as "other products around the survey target product" based on the processing target image, and output the generated drawing data on the display of the target terminal.

[0057] This embodiment also makes it possible to accurately detect errors in the correspondence between products and shelf labels. In addition, for example, if the holder of a terminal that receives information from the output unit 140 designates a product to be surveyed in advance, he or she can easily recognize the price information of the product to be surveyed and the price information of products around the product.

[0058] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various configurations other than those described above can also be adopted.

[0059] In addition, in the flowcharts used in the above description, multiple steps (processes) are described in order, but the order of execution of the steps in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the figures can be changed to the extent that the content is not affected. Furthermore, the above-mentioned embodiments can be combined to the extent that the content is not contradictory.

[0060] A part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes. 1. a product information acquiring means for acquiring a position of the product by analyzing an image of the product and the shelf label; a shelf label information acquisition means for acquiring the position of the shelf label by analyzing the image; a first determination means for determining consistency of a correspondence relationship between the product and the shelf label based on the acquired relationship between the position of the product and the position of the shelf label; an output means for outputting a result of the determination of the consistency of the correspondence relationship; An information processing device comprising: 2. The product information acquisition means further acquires product information associated with the product by analyzing the image; the shelf label information acquisition means further acquires shelf label product information written on the shelf label by analyzing the image, the first determination means further determines consistency of a correspondence relationship between the product and the shelf label based on a relationship between the acquired product information and the shelf label product information. 1. An information processing device according to the present invention. 3. the output means outputs at least one of a positional deviation of the shelf label, a missing shelf label, an excess shelf label, and an error in the shelf label product information as a result of the determination of the consistency of the correspondence relationship. 2. An information processing device according to the present invention. 4. The product information acquisition means acquires a standard price associated with the product, The shelf label information acquisition means acquires the product price written on the shelf label, The first determination means determines whether or not the product price indicated on the shelf label is within a standard price range associated with the product. 1. An information processing device according to any one of 1 to 3. 5. A second determination means for determining whether or not a survey target product is present among the products in the image, the output means further outputs, when the survey target product is present, a product price indicated on a shelf tag corresponding to the survey target product. 5. An information processing device according to any one of 1 to 4. 6. A third determination means for determining whether or not a survey target product is present among the products in the image, the output means further outputs, when the survey target product is present, product prices indicated on shelf tags corresponding to other products present around the survey target product. 6. An information processing device according to any one of 1 to 5. 7. The first determination means determining whether or not an arrangement order of the products based on the product information matches an arrangement order of the products based on the shelf label product information; 7. An information processing device according to any one of 2. to 6.

[0061] 8. The computer Acquire the position of the product by analyzing the captured image of the product and the shelf label; acquiring a position of the shelf label by analyzing the image; determining the consistency of the correspondence between the product and the shelf tag based on the acquired relationship between the product position and the shelf tag; outputting a result of the determination of the consistency of the correspondence relationship; An information processing method comprising the steps of: 9 The computer, Analyzing the image to further obtain product information associated with the product; Analyzing the image to further obtain shelf label product information described on the shelf label; and determining whether the correspondence between the product and the shelf label is consistent based on the relationship between the acquired product information and the shelf label product information. The information processing method according to claim 8, further comprising: 10. The computer, outputting at least one of a positional deviation of the shelf label, a missing shelf label, an excess shelf label, and an error in the shelf label product information as a result of determining the consistency of the correspondence relationship; 9. The information processing method according to claim 9, 11. The computer, Obtaining a list price associated with the product; Obtain the product price listed on the shelf label; determining whether the product price indicated on the shelf label is within a standard price range associated with the product; 11. The information processing method according to any one of 8. to 10., 12. The computer, determining whether a survey target product is present among the products in the image; If the survey target product is present, further outputting the product price indicated on the shelf label corresponding to the survey target product. 12. The information processing method according to any one of 8. to 11., 13. The computer determining whether a survey target product is present among the products in the image; If the survey target product is present, further outputting the product prices indicated on the shelf tags corresponding to other products present around the survey target product. 13. The information processing method according to any one of 8. to 12., 14. The computer, determining whether or not an arrangement order of the products based on the product information matches an arrangement order of the products based on the shelf label product information; 14. The information processing method according to any one of 9. to 13.,

[0062] 15. A program for causing a computer to execute the information processing method according to any one of 8. to 14.

[0063] This application claims priority based on Japanese Patent Application No. 2017-191791, filed September 29, 2017, the disclosure of which is incorporated herein in its entirety.

Claims

1. a product information acquiring means for acquiring the arrangement order of the products by analyzing an image of the products and the shelf tags; a shelf label information acquisition means for acquiring the arrangement order of the shelf labels by analyzing the image; a first determination means for determining whether or not the arrangement order of the products in the image matches the arrangement order of the shelf tags based on the acquired arrangement order of the products and the arrangement order of the shelf tags; an output means for outputting information indicating a result of the determination; An information processing device comprising:

2. The output means outputs information notifying a state in which the arrangement orders do not match as information indicating a result of the determination. The information processing device according to claim 1 .

3. The product information acquisition means further acquires product information associated with the product by analyzing the image; the shelf label information acquisition means further acquires shelf label product information written on the shelf label by analyzing the image, the first determination means further determines whether the arrangement orders match by using a result of comparing the acquired product information regarding the products and shelf tags associated based on the arrangement orders with the shelf tag product information.

3. The information processing device according to claim 1 or 2.

4. The product information acquisition means acquires a standard price associated with the product, The shelf label information acquisition means acquires the product price written on the shelf label, The first determination means determines whether or not the product price indicated on the shelf label is within a standard price range associated with the product. The information processing device according to claim 1 .

5. A second determination means for determining whether or not a survey target product is present among the products in the image, The output means further outputs information related to the survey target product when the survey target product exists. The information processing device according to claim 1 .

6. the output means further outputs, when the survey target product is present, a product price indicated on a shelf label corresponding to the survey target product as information related to the survey target product. The information processing device according to claim 5 .

7. the output means further outputs, when the survey target product is present, product prices indicated on shelf tags corresponding to other products present around the survey target product as information related to the survey target product.

7. The information processing device according to claim 5 or 6.

8. the first determination means determines an association between the product in the image and the shelf label based on whether a distance calculated using the acquired position of the product and the position of the shelf label is within a predetermined distance. The information processing device according to claim 1 .

9. The computer Acquire the arrangement order of the products by analyzing the captured image of the products and the shelf tags; The image is analyzed to obtain the arrangement order of the shelf labels; determining whether or not the arrangement order of the products in the image matches the arrangement order of the shelf tags based on the acquired arrangement order of the products and the arrangement order of the shelf tags; outputting information indicating the result of the determination; An information processing method comprising the steps of:

10. Computer, a product information acquisition means for acquiring the arrangement order of the products by analyzing an image of the products and the shelf tags; a shelf label information acquisition means for acquiring the arrangement order of the shelf labels by analyzing the image; a first determination means for determining whether or not the arrangement order of the products in the image matches the arrangement order of the shelf tags based on the acquired arrangement order of the products and the arrangement order of the shelf tags; an output means for outputting information indicating the result of the determination; A program to function as a

Images