Information processing program

Abstract

To provide an information processing program that can easily perform appropriate management of inventory information of commodities in a store.SOLUTION: An information processing program is for managing inventory information of commodities in a store, and causes a computer to execute an image acquisition step and an inventory information update step. The image acquisition step is processing of acquiring image data obtained by photographing at least part of the inside of the store. The inventory information update step is processing of updating the inventory information of the store on the basis of commodity information included in the image data. The inventory information update step updates the inventory information by collating a text indicating the names of commodities obtained from the image data with master data including the names of commodities sold in the store.SELECTED DRAWING: Figure 17

Description

[Technical field]

[0001] The present invention relates to an information processing program. [Background technology]

[0002] Inventory management has been practiced for some time in retail stores such as supermarkets and convenience stores. Displaying appropriate quantities of products in appropriate locations leads to increased sales and reduced excess inventory, and appropriate inventory management is an important management issue. For this reason, various inventory management systems have been proposed (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2000-048262 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the system of Patent Document 1, the weight sensor can detect the movement of products, but it is difficult to identify the moved products. For this reason, it is necessary to input position information indicating the position of each product in the sales floor in advance. Furthermore, in order to manage the inventory of all products, weight sensors must be placed throughout the sales floor of the store, which requires a large cost. For this reason, there has been a demand for a technology for inventory management that can be installed and used at low cost without imposing a large effort on the user.

[0005] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide an information processing program that can easily and appropriately manage inventory information of products in a store. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, one information processing program of the present invention is an information processing program for managing inventory information of products in a store, which causes a computer to execute an image acquisition step of acquiring image data obtained by photographing at least a portion of the store, and an inventory information update step of updating the inventory information of the store based on product information contained in the image data, and the inventory information update step updates the inventory information by comparing text indicating the name of the product obtained from the image data with master data containing the names of each product handled in the store.

[0007] In order to achieve the above-mentioned object, another information processing program of the present invention is an information processing program for managing inventory information of products in a store, which causes a computer to execute an image acquisition step of acquiring first image data obtained by photographing at least a portion of the store and second image data obtained by photographing at least a portion of the store after the first image data is photographed, and an inventory information update step of updating the inventory information of the store based on product information included in the second image data, wherein the inventory information update step updates inventory information for the first product if the first image data includes a first range and a second range in the store and the second image data includes the first range but not the second range in the store, and if a first product included in the first range of the first image data is not included in the second image data, and does not update inventory information for the second product if a second product included in the second range of the first image data is not included in the second image data.

[0008] In one aspect of the information processing program of the present invention, in the image acquisition step, first image data and second image data obtained by photographing after the first image data are acquired as the image data, and in the inventory update step, if the first image data includes a first range and a second range in the store and the second image data includes the first range but not the second range in the store, and if a first product included in the first range of the first image data is not included in the second image data, the inventory information for the first product is updated, and if a second product included in the second range of the first image data is not included in the second image data, the inventory information for the second product is not updated, and the text compared with the master data is text obtained from product information included in the second image data.

[0009] An information processing program according to one embodiment of the present invention further causes a computer to execute a range identification step of comparing the first image data with the second image data and identifying a range of the second image data within the range of the first image data, wherein the first range is the range of the second image data and the second range is another range of the second range within the range of the first image data.

[0010] In one embodiment of the information processing program of the present invention, in the range identification step, the range of the second image data is identified within the range of the first image data by a matching process between product information contained in the first image and product information contained in the second image.

[0011] In an information processing program according to one embodiment of the present invention, the first image data includes a plurality of image data obtained by photographing different sections within the store, and the range identification step performs a matching process for each piece of product information included in the second image data with product information included in the plurality of image data held by the first image data, and identifies the image data of the first image data that matches most in the matching process as image data that includes the range of the second image data.

[0012] In another aspect of the information processing program of the present invention, the product information is text.

[0013] An information processing program according to one embodiment of the present invention further causes a computer to execute a product name identification step of identifying the name of a product included in the second image data, the product name identification step including a category identification step of identifying a product category for each piece of product information based on the product information included in the second image data, and a product name matching step of performing a matching process for the product information included in the second image data with the names of products in the store that belong to each of the product categories identified by the category identification step, thereby identifying the name of the product included in the second image data. Effect of the Invention

[0014] According to the information processing program of the present invention, it is possible to easily and appropriately manage inventory information of products in a store. [Brief description of the drawings]

[0015] [Figure 1] 1 is a diagram illustrating a schematic configuration of an information processing system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a store terminal in the information processing system. [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a DB server in the information processing system. [Figure 4]11 is a diagram illustrating an example of a data configuration of a store product information table stored in a database of a DB server. FIG. [Diagram 5] 13 is a diagram illustrating an example of a data configuration of a stock information table stored in a database of a DB server. [Figure 6] 13 is a diagram illustrating an example of a data configuration of a store image table stored in a database of a DB server. FIG. [Figure 7] 13 is a diagram showing an example of a data configuration of an internal image information table stored in a database of a DB server. FIG. [Figure 8] FIG. 2 is a diagram illustrating an example of a hardware configuration of an update server in the information processing system. [Figure 9] FIG. 2 is a functional block diagram illustrating an example of a functional configuration of a store terminal. [Figure 10] FIG. 2 is a functional block diagram illustrating an example of a functional configuration of an update server. [Figure 11] 11 is a diagram for explaining functions of a product recognition unit and a character extraction unit of the update server. FIG. [Figure 12] 13 is a diagram illustrating example character information extracted by a character extraction unit of the update server and a category identified for each piece of character information by an extracted character classification unit of the update server. FIG. [Figure 13] 11 is a flowchart showing an example of the flow of an image information registration process performed by the information processing system. [Figure 14] 13 is a flowchart showing an example of the flow of a product specification process executed by an update server. [Figure 15] FIG. 2 is a diagram showing an example of target image data. [Figure 16] FIG. 4 is a diagram showing an example of reference image data. [Figure 17] 11 is a flowchart showing an example of the flow of an inventory management process carried out by the information processing system. [Figure 18] 13 is a flowchart showing an example of the flow of an inventory information update process executed by an inventory information update unit of the update server. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0017] Fig. 1 is a diagram showing a schematic configuration of an information processing system 1 according to an embodiment of the present invention. As shown in Fig. 1, the information processing system 1 according to the present invention can be applied to a plurality of stores. The information processing system 1 according to the present invention may also be applied to one store. In the following explanation, the configuration and functions of the information processing system 1 in one store among the plurality of stores will be explained, but the information processing system 1 has a similar configuration and functions in the other stores as well.

[0018] An information processing system 1 shown in FIG. 1 is an example of a system that realizes processing that an information processing program according to an embodiment of the present invention causes a computer to execute. The information processing program according to the embodiment of the present invention is an information processing program for managing inventory information of products in a store, and causes a computer to execute an image acquisition step and an inventory information update step. The image acquisition step is, for example, a process of acquiring image data obtained by photographing at least a part of the inside of the store. The inventory information update step is, for example, a process of updating inventory information of the store based on product information contained in the image data. The inventory information update step updates inventory information by, for example, comparing text indicating the name of the product obtained from the image data with master data including the names of each product handled in the store.

[0019] In another embodiment, the image acquisition step of the information processing program is, for example, a process of acquiring first image data obtained by photographing at least a part of the inside of the store and second image data obtained by photographing at least a part of the inside of the store after the photographing of the first image data. Also, the inventory information update step of the information processing program is, for example, a process of updating inventory information of the store based on product information included in the second image data. For example, when the first image data includes a first range and a second range in the store, and the second image data includes the first range but not the second range in the store, the inventory information update step updates inventory information about the first product when a first product included in the first range of the first image data is not included in the second image data, and does not update inventory information about the second product when a second product included in the second range of the first image data is not included in the second image data.

[0020] The configuration of the information processing system 1 and the processing of the information processing program are specifically described below. The program is a data processing method described in any language or description method, and may be in the form of source code, binary code, or the like. The program is not necessarily limited to a single configuration, but may be a distributed configuration as multiple modules or libraries, or may work with a separate program, such as an OS, to achieve its function. The inventory information is information indicating whether or not each product is in stock. In the present embodiment, the inventory refers to store inventory, where products are on the sales floor of a store, but is not necessarily limited to this. The inventory may also refer to products in a store, including products stored in a warehouse or the like that are not on the sales floor, as well as products on the sales floor of a store.

[0021] The information processing system 1 includes a store terminal 10, a DB server 20, and an update server 30, as shown in FIG. 1, for example. The store terminal 10 is a terminal capable of transmitting image data, and may be a mobile terminal, a fixed terminal, or the like. The DB server 20 stores information about products in the store, including inventory information, and image data obtained in the image acquisition step. The update server 30 updates the inventory information stored in the DB server 20 based on the product information included in the image data acquired in the image acquisition step. The product information included in the image data is information about each product that is present (shown) in an image (hereinafter referred to as a "display image") displayed based on the image data. The product information included in the image data is specifically information that identifies each product, such as information about the name of each product. The product information includes, for example, text. The product information is not limited to text, and may be graphic information such as a logo or mark, or may be a combination of text and graphic information.

[0022] As shown in FIG. 1, the store terminal 10 is provided in each store and is used by users such as inventory managers in each store. One or more store terminals 10 may be provided in each store. The DB server 20 stores information about products in each store, image data, information based on image data, and the like. The products in each store are, for example, all products in each store, and include not only products displayed in each store, but also out-of-stock products and products stored in the warehouses of each store. The store terminal 10 is a tool that supports inventory management for users. The store terminal 10 transmits image data used for inventory information management processing in the information processing system 1. The DB server 20 is a database that stores data required for inventory information management processing of the store to which the information processing system 1 is applied. The update server 30 is a server that executes an extraction process to extract information required for inventory information management processing and also executes inventory information update processing. As shown in FIG. 1, the DB server 20 and the update server 30 are placed in, for example, a management department. The management unit is a location such as a facility that has a function of managing the information processing system 1. As shown in Fig. 1, the store terminal 10, the DB server 20, and the update server 30 are computer systems connected via a network 2 such as the Internet.

[0023] The network 2 is not limited to the Internet, but may be, for example, a dedicated line, a public line (telephone line, mobile communication line, etc.), a wired LAN (Local Area Network), a wireless LAN, etc., as long as it can connect the store terminal 10, the DB server 20, and the update server 30 to each other so that they can communicate with each other, or it may be a combination of these with the Internet.

[0024] The store terminal 10 is, for example, a mobile terminal. The mobile terminal may be any of various information processing devices that a user can carry around. For example, a smartphone, a tablet computer, a personal computer, etc. can be applied to the mobile terminal. In this embodiment, the store terminal 10 is a mobile terminal. As shown in FIG. 1, the store terminal 10 has a camera 11 and a display unit 12. The display unit 12 is also a user interface. It is to be noted that the store terminal 10 does not need to have a camera. In this case, the store terminal 10 receives and stores image data. For example, a user creates image data, and the store terminal 10 receives and stores the image data by downloading or the like. The image data is created, for example, by a user taking a picture with a camera.

[0025] FIG. 2 is a diagram showing an example of the hardware configuration of the store terminal 10. As described above, the store terminal 10 includes a camera 11 and a display unit 12. As shown in FIG. 2, the store terminal 10 includes a CPU (Central Processing Unit) 13, a ROM (Read Only Memory) 14 and a RAM (Random Access Memory) 15 as main storage devices, an input unit 16, and a communication unit 17. The components 11 to 17 are connected to each other via a bus line 18. Note that an interface circuit is interposed between the bus line 18 and each component as necessary, but the interface circuit is not shown in FIG. 2. Note that the store terminal 10 may include a GPU (Graphics Processing Unit) in addition to or instead of the CPU 13.

[0026] The camera 11 is a known camera used in mobile terminals and the like. The display unit 12 is an interface that displays various information, and is, for example, an image display device such as a liquid crystal display. The input unit 16 is an interface that allows a user to input information to the store terminal 10, and includes, for example, a touch panel, a touch pad, a keyboard, a mouse, and a microphone provided on the display unit 12. The communication unit 17 is a communication interface that connects the store terminal 10 to the network 2. The communication unit 17 of the store terminal 10 transmits and receives information via the network 2 to and from communication devices connected to the network 2, such as the DB server 20 and the update server 30.

[0027] The CPU 13 interprets and executes commands from various programs, and controls the entire store terminal 10. The ROM 14 stores programs and the like necessary for basic operational control of the store terminal 10. The RAM 15 stores various programs and data loaded from the ROM 14 and the like, and ensures a working area for the CPU 13. Various programs, such as programs related to the processing of the store terminal 10 of the information processing system 1, are stored in the ROM 14, and are loaded into the RAM 15 and executed by the CPU 13.

[0028] FIG. 3 is a diagram showing an example of a hardware configuration of the DB server 20. The DB server 20 is, for example, configured by a server device that is a computer. As shown in FIG. 3, the DB server 20 includes a storage device 21, a CPU 22, a ROM 23 and a RAM 24 as main storage devices, and a communication unit 25. The components 21 to 25 are connected to each other via a bus line 26. Note that, although an interface circuit is interposed between the bus line 26 and each component as necessary, the interface circuit is omitted from FIG. 3. Note that the DB server 20 may include a GPU (Graphics Processing Unit) together with the CPU 22 or instead of the CPU 22.

[0029] The CPU 22 interprets and executes commands of various programs, and controls the entire DB server 20. The ROM 23 stores programs and the like necessary for basic operational control of the DB server 20. The RAM 24 stores various programs and data loaded from the ROM 23 and the like, and secures a working area for the CPU 22. Various programs, such as programs related to the processing of the DB server 20 of the information processing system 1, are stored in the ROM 23, loaded into the RAM 24, and executed by the CPU 22. The communication unit 25 is a communication interface that connects the DB server 20 to the network 2, and the DB server 20 transmits and receives information to and from communication devices connected to the network 2, such as the store terminal 10 and the extraction server 30, via the network 2, in the communication unit 25.

[0030] 3, the storage device 21 of the DB server 20 stores a database 40. The database 40 stores information about products in each store. For example, the database 40 stores information about products in each store by store. The programs stored in the ROM 23 include programs related to processes for accessing data stored in the database 40, updating data stored in the database 40, and the like.

[0031] The database 40 includes a store product information table 41 and an inventory information table 42, as shown in Fig. 3, for example. The store product information table 41 is an information table that stores information about the product names and types (categories) of all products handled at each store. The database 40 has a store product information table 41 for each store, for example. The inventory information table 42 is an information table that stores information about the inventory of all products handled at each store, for example. The database 40 has an inventory information table 42 for each store, for example.

[0032] The database 40 also includes a store image table 43 and an in-image information table 44, as shown in Fig. 3, for example. The store image table 43 is an information table in which image data received from the store terminal 10 and information identifying each image data are stored. The in-image information table 44 is provided for each image data stored in the store image table 43, for example, and is an information table in which product information, which is information about products included in each image data, i.e., about each product in the display image of each image data, and information about the position of each product included in each image data in the display image, and the like are stored. The database 40 has the store image table 43 and the in-image information table 44 for each store, for example.

[0033] FIG. 4 is a diagram showing an example of the data configuration of the store product information table 41. As shown in FIG. 4, the store product information table 41 has a product name 41a in which product name information indicating the product names of all products handled in each store is stored. As shown in FIG. 4, the product name 41a in the store product information table 41 may store information indicating the product name of a product not currently handled in the store. Products not handled in the store include, for example, products that have been discontinued. In this case, as shown in FIG. 4, the store product information table 41 has a handling 41b in which information indicating whether the product included in the product name 41a is handled in the store is stored. In addition, the store product information table 41 has a product category 41c in which a plurality of category information indicating categories into which all products handled in each store are classified is stored. The category information is, for example, information indicating the type and form of each product. The product name information stored in the product name 41a is linked to the corresponding category information stored in the product category 41c. In other words, the category to which each product belongs can be identified based on the store product information table 41. The categories are, for example, the type and form of a product, such as fried food, beverages, side dishes, snacks, rice balls, lunch boxes, snacks, salads, food ingredients, seasonings, chilled noodles, frozen foods, retort foods, etc. The store product information table 41 may also be provided with a product location 41d that stores location information indicating where each product is located in the store.

[0034] FIG. 5 is a diagram showing an example of the data configuration of the inventory information table 42. As shown in FIG. 5, the inventory information table 42 has a product name 42a in which information indicating the product names of all products handled in each store is stored. Also, as shown in FIG. 5, the inventory information table 42 has inventory information 42b in which inventory information of all products handled in the store is stored. The inventory information 42b stores, for example, information indicating whether each product is in the store. Note that the information indicating each product name in the product name 42a of the inventory information table 42 is, for example, a value linked to information indicating the same product name in the product name 41a of the store product information table 41 shown in FIG. 4.

[0035] Fig. 6 is a diagram showing an example of the data configuration of the store image table 43. As shown in Fig. 6, the store image table 43 has a store image 43a in which image data obtained by photographing a store is stored, an image ID 44b in which ID information for identifying each image data stored in the store image 43a is stored, and a representative category 44c in which representative category information that is information indicating a category that roughly groups together products in each display image stored in the store image 43a is stored.

[0036] For example, a plurality of image data are stored in the store image 43. For example, each of the plurality of image data is image data obtained by photographing different places in the store, and is image data received from the store terminal 10. Also, the plurality of image data is, for example, image data obtained by photographing all of the sales floor of the store. The sales floor of the store is a place where products are placed, such as product shelves and product cases. Note that the plurality of image data stored in the store image 43a does not have to be image data obtained by photographing all of the sales floor of the store. In other words, the plurality of image data stored in the store image 43 may be image data obtained by photographing a specific range of the sales floor of the store. Note that in this case, the plurality of image data is, for example, image data obtained by photographing different parts of the specific range of the sales floor of the store. The ID information is, for example, an ID arbitrarily set for each image data. For example, the ID information stored in the image ID 43b is linked to the corresponding image data of the store image 43a.

[0037] The representative category information is, for example, information that indicates the type or form of a product, and is information that collectively refers to all or some of the products in the display image of one piece of image data. The representative category information is selected from the category information of the product category 41c in the store product information table 41. A plurality of pieces of representative category information may be set (stored) for one piece of image data, or one piece of representative category information may be set for one piece of image data. All of the products in the display image of one piece of image data belong to one or more pieces of representative category information for the one piece of image data. In other words, one piece of product information in the display image of one piece of image data is included in a category corresponding to at least one piece of representative category information. Therefore, when multiple representative categories are set for one piece of image data, each piece of product information obtained from this piece of image data may belong to all of the multiple representative categories.

[0038] Fig. 7 is a diagram showing an example of the data configuration of the in-image information table 44. As shown in Fig. 7, the in-image information table 44 is provided for each image data stored in the store image 43a of the store image table 43. That is, as shown in Fig. 7, the database 40 has in-image information tables in the number corresponding to the number of image data stored in the store image 43a, for example, a plurality of in-image information tables 44A, 44B, 44C, 44D, etc. Each in-image information table 44 stores, for example, ID information of the corresponding image data, and also has product name 44a, coordinate information 44b, product identification information 44c, and category information 44d.

[0039] The product name 44a stores product name information, which is information indicating the product name of each product included in the corresponding image data. Note that the product name information stored in the product name 44a is a value linked to information indicating the same product name in the product name 41a in the store product information table 41 and the product name 42a in the inventory information table 42 shown in Figs. 4 and 5.

[0040] The coordinate information 44b stores coordinate information indicating the position of each product in the display image of the corresponding image data. The coordinate information is, for example, information specifying the range and position in the display image of the area occupied by each product in the display image (see FIG. 11 described later). Note that the coordinate information stored in the coordinate information 44b is linked to the corresponding product name information stored in the product name 44a, for example.

[0041] The product identification information 44c stores product identification information that can be recognized from the image data of each product included in the corresponding image data. The product identification information is, for example, character information (text) related to the product that is located in an area that each product specified by the coordinate information occupies in the display image. The product identification information 44c may store one product identification information for each product, or may store multiple product identification information. In this example, as shown in FIG. 7, one piece of character information is stored for each product. The character information is, for example, a price tag, a sticker, or characters written on the product package on which the product name of each product is written. The stored character information is, for example, a single meaningful character or a set of meaningful characters. The character information stored in the product identification information 44c is, for example, linked to the corresponding product name information stored in the product name 44a.

[0042] The category information 44d stores category information indicating the type of each item identification information. As with the representative category information described above, the category information is selected from the item categories 41c in the store item information table 41. For example, the category information 44d stores one item of category information for one item identification information. Note that the category information stored in the category information 44d is linked to the corresponding item identification information (text information) stored in the item identification information 44c, for example.

[0043] The database 40 may store the information of the store product information table 41, the inventory information table 42, the store image table 43, and the in-image information table 44 in a single table, or may store the information in multiple tables by combining any of the information.

[0044] FIG. 8 is a diagram showing an example of a hardware configuration of the update server 30. The update server 30 is configured by a server device that is a computer. As shown in FIG. 8, the update server 30 includes a CPU 31, a ROM 32 and a RAM 33 as main storage devices, and a communication unit 34. The components 31 to 34 are connected to each other via a bus line 35. Note that an interface circuit is interposed between the bus line 35 and each component as necessary, but the interface circuit is not shown in FIG. 8. Note that the extraction server 30 may include a GPU (Graphics Processing Unit) in addition to or instead of the CPU 31.

[0045] The CPU 31 interprets and executes commands of various programs, and controls the entire update server 30. The ROM 32 stores programs and the like required for basic operational control of the update server 30. The RAM 33 stores various programs and data loaded from the ROM 32 and the like, and secures a working area for the CPU 31. Various programs, such as programs related to the processing of the update server 30 of the information processing system 1, are stored in the ROM 32 and loaded into the RAM 33 and executed by the CPU 31. The communication unit 34 is a communication interface that connects the update server 30 to the network 2, and the update server 30 transmits and receives information through the communication unit 34 to and from communication devices connected to the network 2, such as the store terminal 10 and the DB server 20, via the network 2.

[0046] Next, the functional configuration of the store terminal 10 will be described. Fig. 9 is a functional block diagram showing an example of the functional configuration of the store terminal 10. As shown in Fig. 9, the store terminal 10 has, as its functional configuration, for example, an image forming unit 101, an image transmitting unit 102, and an operation recognizing unit 103. Each of these functional configurations is realized by the CPU 13 of the store terminal 10 executing a program according to this embodiment.

[0047] When a user operates input unit 16 to command camera 11 to take a picture, image forming unit 101 causes camera 11 to take a picture of a subject and generates image data. Image forming unit 101 attaches information for identifying the image data to the generated image data. This information for identification is, for example, the above-mentioned ID information. In addition, image forming unit 101 stores the generated image data in ROM 14, for example. Operation recognition unit 103 recognizes a user's operation of input unit 16 on display unit 12.

[0048] The image transmission unit 102 transmits image data designated from the image data stored in the ROM 14 to the DB server 20 from the communication unit 17 by the user operating the input unit 16. The operation recognition unit 103 recognizes the user's operation of the input unit 16 in the display unit 12. When the image forming unit 101 generates image data, the image transmission unit 102 may transmit the generated image to the DB server 20 without the need for the user to operate the input unit 16. The image transmission unit 102 may transmit the image data to the update server 30 instead of the DB server 20, or may transmit the image data to both the DB server 20 and the update server 30.

[0049] Next, the functional configuration of the update server 30 will be described. Fig. 10 is a functional block diagram showing an example of the functional configuration of the update server 30. As shown in Fig. 10, the update server 30 has, as its functional configuration, for example, a product recognition unit 201, a character extraction unit 202, an extracted character classification unit 203, an image data classification unit 204, a product identification unit 205, an image range identification unit 206, and an inventory information update unit 207. Each of these functional configurations is realized by the CPU 31 of the update server 30 executing a program according to this embodiment.

[0050] The commodity recognition unit 201 is a function that generates, from image data, coordinate information to be stored in the coordinate information 44b of the intra-image information table 44 of the database 40 of the DB server 20. That is, the commodity recognition unit 201 recognizes each commodity in the display image from the image data, specifies the area that each commodity occupies in the display image (hereinafter also referred to as the commodity area), and generates coordinate information that specifies the range and position of the specified commodity area in the display image. FIG. 11 is a diagram for explaining the functions of the commodity recognition unit 201 and the character extraction unit 202. FIG. 11 shows an example of image data obtained by photographing a part of a store. Specifically, FIG. 11 shows a display image of the image data obtained by photographing a part of a store. For example, as shown in FIG. 11, the commodity recognition unit 201 recognizes a commodity in the display image, and specifies the commodity areas A1, A2, A3, etc. of the recognized commodities M1, M2, M3, etc. The product areas A1, A2, A3, etc. are box areas surrounding the recognized products M1, M2, M3, etc., as shown in FIG. 11, for example. The product recognition unit 201 calculates the range and position of each of the identified product areas A1, A2, A3, etc. in the display image, and generates coordinate information. The product recognition unit 201 stores the generated coordinate information of each product in, for example, the ROM 32. The function of the product recognition unit 201 may be performed using, for example, AI. The product recognition unit 201 may realize its function by implemented hardware, or may realize its function in cooperation with an external service on the network 2 via, for example, an API (Application Programming Interface).

[0051] The character extraction unit 202 is a function for extracting character information (text information) from image data. That is, the character extraction unit 202 extracts character information representing characters in a display image from the image data. The characters in an image are, for example, characters written on a price tag, sticker, or product package on which the product name of each product is written. The character extraction unit 202 extracts unit characters, for example, from image data stored in the store image table 43 of the DB server 20 or from image data transmitted from the store terminal 10. A unit character is assumed to be a single character that makes sense or a set of characters that make sense. For example, the character extraction unit 202 extracts characters in a display image in units of meaningful characters. In addition, the character extraction unit 202 extracts character information in the product area of ​​each product identified by the product recognition unit 201. For example, the character extraction unit 202 extracts multiple characters in each product area and combines these multiple characters to form a unit character that makes sense. Furthermore, the character extraction unit 201 generates unit characters from the extracted characters using, for example, an edit distance. The function of the character extraction unit 202 is performed using, for example, optical character recognition (OCR). The function of the character extraction unit 202 may be performed using optical character recognition using AI. The character extraction unit 202 may implement the function, or may realize the function in cooperation with an external service on the network 2 via, for example, an API (Application Programming Interface). The character information (unit characters) extracted by the character extraction unit 202 is linked to the corresponding product area and stored in the ROM 32.

[0052] Extracted character classification unit 203 is a function of identifying a category of each piece of character information extracted by character extraction unit 202. Specifically, extracted character classification unit 203 identifies a category to which each piece of character information extracted by character extraction unit 202 belongs from category information stored in product category 41c of store product information table 41 of database 40. Extracted character classification unit 203, for example, deciphers the meaning or content of each piece of character information, or compares each piece of character information with predetermined reference information, selects category information of a category that matches the deciphering result or comparison result from category information stored in product category 41c, and identifies the selected category information as category information of character information to be processed by extracted character classification unit 203. The function of extracted character classification unit 203 may be performed using AI. Extracted character classification unit 203 may be implemented with the function, or may realize the function by cooperating with an external service on network 2 via, for example, an API (Application Programming Interface). The category information identified by the extracted character classifying unit 203 is associated with the corresponding character information and stored in the ROM 32.

[0053] The image data classification unit 204 is a function for identifying the category of the image data based on the category information identified by the extracted character classification unit 203 for each of all character information included in the image data. That is, the image data classification unit 204 identifies the category of the display image based on the type of product in the display image. The category identified for the image data is a representative category that identifies the type, form, etc. of the product in the display image. Specifically, the image data classification unit 204 extracts a predetermined number of category information items in descending order of number from among the multiple category information items identified for each of all character information items included in the image data, and sets the extracted category information items as the category information (representative category information) of the image data. Specifically, for example, the image data classification unit 204 selects the two most numerous category information items from among the multiple category information items identified for each of all character information items included in the image data, and identifies the two selected category information items as the representative category information representing the category to which the image data belongs. Note that the number of category information items identified by the image data classification unit 204 for the image data is not limited to two, and may be one or three or more. The function of the image data classification unit 204 may be performed using AI. The image data classification unit 204 may implement the function, or may realize the function in cooperation with an external service on the network 2 via, for example, an API (Application Programming Interface). The category information (representative category information) identified by the image data classification unit 204 is associated with the corresponding image data and stored in the ROM 32.

[0054] The product identification unit 205 is a function for identifying the product name of each product included in the image data. For example, the product identification unit 205 identifies the product name of each product based on the category information identified by the extracted character classification unit 203 for each of all character information included in the image data. Specifically, for example, the product identification unit 205 extracts the product names of products included in the category indicated by the representative category information identified for the image data by the image data classification unit 204 from the store product information table 41, and creates a product candidate list in which the extracted product name information is compiled. Then, the product identification unit 205 performs a matching process between the character information extracted by the character extraction unit 202 for each product included in the image data and the created product candidate list, selects the product name information of the product that best matches the character information of each product, and identifies this product name information as information representing the product name of the product. This product name matching process, for example, determines whether a character string represented by the character information is included in the product name, and whether the thing or meaning represented by the character information matches or is similar to the product name. Note that the identification of the product name of each product included in the image data by the product identification unit 205 is not limited to using the product candidate list based on the representative category information described above. For example, the product identification unit 205 may perform a matching process between the character information extracted by the character extraction unit 202 for each product included in the image data and the product name information stored in the store product information table 41, select the product name information of the product that best matches the character information of each product, and identify this product name information as information representing the product name of the product.

[0055] The function of the product identification unit 205 may be performed using AI. The product identification unit 205 may implement the function, or may realize the function in cooperation with an external service on the network 2 via, for example, an API (Application Programming Interface). The category information identified by the product identification unit 205 is associated with the corresponding coordinate information and stored in the ROM 32.

[0056] The image range specifying unit 206 is a function of specifying image data (hereinafter also referred to as reference image data) that displays the same store location as the store location displayed by the image data to be processed (hereinafter also referred to as target image data) from among a plurality of image data stored in the store image 43a of the store image table 43 of the database 40 of the DB server 20, and specifying the range displayed by the target image data in the display image of the specified reference image data. For example, the image range specifying unit 206 compares a plurality of image data (hereinafter also referred to as master image data) obtained by photographing different locations of the whole or part of the sales floor of the store stored in the store image 43a of the store image table 43 with the target image data, and specifies image data (reference image data) that displays the same store location as the store location displayed by the target image data from among the master image data. In addition, the image range specifying unit 206 specifies the range of the display image of the target image data in the display image of the specified reference image data. The reference image data specified by the image range specifying unit 206 is linked to the target image data and stored in the ROM 32. For example, the ID information of the reference image data specified by the image area specifying unit 206 is associated with the ID information of the target image data and stored in the ROM 32.

[0057] The image area specification unit 206 specifies the reference image data by, for example, comparing character information extracted by optical character recognition (OCR). Specifically, for example, it performs a matching process between character information obtained by OCR processing on the target image data and character information obtained by OCR processing on each of the multiple image data stored in the store image 44. Then, image data including character information having the highest matching rate with the character information of the target image data is specified as reference image data that displays the same store location as the store location displayed in the target image data. The image data including character information having the highest matching rate is specified, for example, as follows. That is, for each of all character information extracted from the target image data by OCR processing, it performs a matching process with all character information extracted by OCR processing from the multiple image data stored in the store image 44. Then, ID information of the image data of the store image 43a including character information determined to match each character information extracted from the target image data is linked to each character information extracted from the target image data and stored in the ROM 32 or the RAM 33. Then, it specifies the most frequent ID information from the stored ID information, and sets the image data indicated by this most frequent ID information as the reference image data. The ID information of the reference image data is linked to, for example, the target image data and stored in the ROM 32. Specifically, for example, the ID information of the reference image data is linked to the ID information of the target image data and stored in the ROM 32. Note that the identification of the reference image data by the image range identification unit 206 is not limited to the matching process of the character information obtained by the above-mentioned OCR process. For example, the reference image data may be identified by a comparison of images in a display image or a matching process of representative category information. The comparison of images in a display image is performed by a process such as feature point matching.

[0058] Furthermore, the image range specifying unit 206 specifies the range of the display image of the target image data in the display image of the reference image data (range specifying step), for example, by comparing the target image data with the reference image data by comparing the image in the display image with a process such as feature point matching. Note that the above-mentioned image range specifying unit 206 does not have to specify the image range by comparing the image in the display image with a process such as feature point matching. The above-mentioned image range specifying unit 206 may specify the image range by, for example, a matching process of character information obtained by OCR processing, or may specify the image range by a matching process of category information of each product included in the image data. Furthermore, the image range specifying unit 206 calculates specific range information, which is information representing the range in the display image of the reference image data specified as the range of the image displayed by the target image data. The specific range information calculated by the image range specifying unit 206 is linked to the image data to be processed and stored in the ROM 32. For example, the specific range information calculated by the image range specifying unit 206 is linked to ID information of the image data to be processed and stored in the ROM 32.

[0059] The image range specification unit 206 also creates a list of product name information of products in the range of the display image of the target image data within the display image of the reference image data specified as described above (hereinafter referred to as a specific range product list). The specific range product name list is created by referring to the database 40 of the DB server 20 based on the ID information and specific range information of the reference image data. Specifically, coordinate information included in the range represented by the specific range information is selected from the coordinate information 44b of the intra-image information table 44 of the image data linked to the ID information of the reference image data stored in the image ID 43b of the store image table 43, and product name information linked to the selected coordinate information is extracted from the product name information stored in the product name information 44a to create the specific range product list. The created specific range product list is linked to the ID information of the target image data, for example, and stored in the ROM 32.

[0060] The above-mentioned function of the image area identification unit 206 may be performed using AI. The image area identification unit 206 may implement the function, or may realize the function in cooperation with an external service on the network 2 via an API (Application Programming Interface), for example.

[0061] The inventory information update unit 207 regards the product included in the target image data as being in stock, and updates the inventory information stored in the inventory information 42b of the inventory information table 42 of the database 40 of the DB server 20. The product included in the target image data is identified by the product identification unit 205. This inventory information is updated by changing the inventory information stored in the inventory information 42b, which is linked to the product name information indicating the product name of the product included in the target image data stored in the product name 42a in the inventory information table 42 of the database 40, to information indicating the availability of stock, if the inventory information stored in the inventory information 42b is information indicating the availability of stock, if the inventory information stored in the inventory information 42b is information indicating the availability of stock, if the inventory information stored in the inventory information 42b is information indicating the availability of stock, if the inventory information stored in the inventory information table 42 of the database 40 is information indicating the availability of stock, the inventory information stored in the inventory information 42b is not changed.

[0062] In addition, the inventory information update unit 207 regards a product that is within the range of the reference image data that displays the same location as the store location displayed by the target image data but is not included in the target image data as out of stock, and updates the inventory information stored in the inventory information 42b of the inventory information table 42 of the database 40 of the DB server 20. This inventory information update is performed, for example, as follows. That is, the inventory information update unit 207 compares the product name information of the product included in the target image data with the product name information in the specific range product list created by the above-mentioned image range specification unit 206, and specifies the product name information that is not included in the target image data but is in the specific range product list. The inventory information update unit 207 regards the product indicated by this specified product name information as out of stock. Then, when the inventory information stored in the inventory information 42b linked to the product name information of the product regarded as out of stock stored in the product name 42a in the inventory information table 42 of the database 40 indicates that the product is in stock, the inventory information stored in the inventory information 42b is changed to information indicating that the product is out of stock. On the other hand, in the inventory information table 42 of the database 40, if the inventory information stored in the inventory information 42b linked to the product name information of the product considered to be out of stock, stored in the product name 42a, indicates that the product is out of stock, then the inventory information stored in the inventory information 42b will not be changed.

[0063] The above-mentioned function of the inventory information update unit 207 may be performed using AI. The inventory information update unit 207 may implement the function, or may realize the function by cooperating with an external service on the network 2 via an API (Application Programming Interface), for example.

[0064] Next, the inventory management process that the information processing program according to the embodiment of the present invention causes the information processing system 1 having the above-mentioned configuration to execute will be described. In the following description, the information processing system 1 performs inventory management processing of products displayed and sold on all product shelves installed in a convenience store, as an example. Note that the inventory management processing executed by the information processing system 1 is not limited to managing inventory information of products on the product shelves of a convenience store, but may also manage inventory information of products in all sales areas such as product shelves and product cases of a convenience store, or products in stores other than convenience stores. As an example, the inventory management processing of the information processing system 1 performs inventory management of each product after a period of time has passed for a state (hereinafter referred to as a reference state) in which a predetermined number of predetermined products are placed in a predetermined position on each product shelf. Note that the reference state is a state in which the largest number of products are displayed on the product shelves, and the predetermined number of each product in the reference state is the maximum number of each product placed on the product shelves. The reference state may be, for example, a state immediately after the products in the store are received, in which each product is lined up in a predetermined position.

[0065] Therefore, in order to execute inventory management processing by the information processing system 1, the user photographs all product shelves in a reference state and creates multiple image data that display all product shelves in the reference state. Note that one image data is image data that displays a part of all product shelves, and each of the multiple image data displays a different part of all product shelves, and the multiple image data collectively display all product shelves. Hereinafter, image data obtained by photographing product shelves in a reference state is referred to as reference image data (first image data). For example, one reference image data is image data obtained by photographing one product shelf. Note that one reference image data may be image data obtained by photographing multiple product shelves, or may be image data obtained by photographing a part of one product shelf. Also, the user may create only one reference image data.

[0066] The reference image data is created by a user photographing a product shelf in a reference state with the camera 11 of the store terminal 10. Furthermore, the store terminal 10 creates ID information that enables each created reference image data to be identified, and assigns it to the reference image data. The reference image data with the ID information assigned thereto is stored in, for example, the ROM 14.

[0067] In order to execute the inventory management process, first, an image information registration process is performed for the reference image data, in which various information of the image data is registered in the database 40 of the DB server 20. The image information registration process is executed, for example, by a user starting an image information registration application of the store terminal 10. In the image information registration process for the reference image data, first, the reference image data stored in the ROM 14 of the store terminal 10 is transmitted from the communication unit 17 of the store terminal 10 to the communication unit 25 of the DB server 20. The reference image data received by the DB server 20 is stored in the store image 43a of the store image table 43 of the database 40, and the ID information is stored in the image ID 43b. In the store image table 43, the reference image data is linked to the ID information. The reference image data may also be transmitted to the communication unit 34 of the update server 30. The reference image data may not be transmitted to the DB server 20, but may be transmitted only to the update server 30. In this case, the DB server 20 receives the reference image data via the update server 30.

[0068] The update server 30 executes a product identification process for identifying products included in each piece of reference image data transmitted from the store terminal 10. In the product identification process, the product recognition unit 201 first generates coordinate information of products included in each piece of reference image data. Specifically, the product recognition unit 201 recognizes products in the display image from the reference image data, and identifies product areas that are areas that each recognized product occupies in the display image (see FIG. 11). Next, the product recognition unit 201 generates coordinate information that identifies the range and position of each identified product area in the display image.

[0069] Next, the character extraction unit 202 extracts character information (text information) from each reference image data. Specifically, the character extraction unit 202 extracts character information representing characters included in the reference image data for each product recognized by the product recognition unit 201. That is, the character extraction unit 202 extracts character information within each product area identified by the product recognition unit 201 (see FIG. 11). As described above, the character information is information representing unit characters written on, for example, a price tag, a sticker, or a product package on which the product name of each product is written. As described above, a unit character is a single character that has meaning, or a set of characters that have meaning. As shown in FIG. 11, one piece of character information is extracted for each product.

[0070] Next, the extracted character classifying unit 203 specifies a category for each piece of character information extracted by the character extracting unit 202. Specifically, the extracted character classifying unit 203 specifies category information representing a category to which each piece of character information extracted by the character extracting unit 202 belongs from category information stored in the product category 41c of the store product information table 41 of the database 40. The category is, for example, a type or form of a product such as fried food, beverage, side dish, snack, rice ball, lunch box, light meal, salad, ingredient food, seasoning, chilled noodles, frozen food, retort food, etc. The extracted character classifying unit 20 may determine that there is no category to which the character information belongs, and in this case, the category information is not specified for the character information. For example, as shown in FIG. 12, a category is specified for the character information extracted from one reference image data.

[0071] Next, the image data classification unit 204 identifies the representative category of each reference image data. In this example, it is assumed that two representative categories are identified for each reference image data. Specifically, the image data classification unit 204 selects the two most numerous categories from among all the categories of character information included in each reference image data identified by the extracted character classification unit 203, and identifies the selected two categories as the representative category of the reference image data. In other words, the image data classification unit 204 sets the two most numerous categories from among all the categories of character information included in one reference image data as the representative category of the reference image data. In the case of the reference image data shown in FIG. 12, the categories of bento and rice ball are the two most numerous categories, and the category information representing these two categories is set as the representative category information. Note that the update server 30 does not need to identify the category of each character information, and does not need to identify the representative category of the reference image data.

[0072] Next, the product identification unit 205 identifies the product name of each product included in each reference image data. Specifically, the product identification unit 205 extracts product name information of products included in the category indicated by the representative category information identified for each reference image data by the image data classification unit 204 from the store product information table 41, and creates a product candidate list in which the extracted product name information is compiled. Then, the product identification unit 205 performs a matching process between the character information extracted by the character extraction unit 202 for each product included in each reference image data and the created product candidate list, selects a product name that best matches the character information of each product, and identifies this product name as the product name of the product. Note that the product identification unit 205 may perform a matching process between the character information extracted by the character extraction unit 202 for each product included in each reference image data and the product name information stored in the store product information table 41, selects a product name that best matches the character information of each product, and identifies this product name as the product name of the product, without creating a product candidate list.

[0073] As described above, the product recognition unit 201, character extraction unit 202, extracted character separation unit 203, image data classification unit 204, and product identification unit 205 of the update server 30 each process each reference image data and information extracted or created therefrom, thereby identifying the product name of each product included in each reference image data. The coordinate information generated by the product recognition unit 201, the character information extracted by the character extraction unit 202, the category information identified for each character information by the extracted character separation unit 203, the representative category of each reference image data identified by the image data classification unit 204, and the product name information identified by the product identification unit 205 are transmitted from the communication unit 34 of the update server 30 to the communication unit 25 of the DB server 20.

[0074] The DB server 20 stores the received coordinate information, character information, category information, representative category information, and product name information in the database 40, linking them to each product name information. Specifically, the product name information, coordinate information, character information, and category information are stored in the product name 44a, coordinate information 44b, product identification information 44c, and category information 44d of the intra-image information table 44, respectively. The representative category information is stored in the representative category 43c of the store image table 43. The intra-image information table 44 has ID information of the corresponding reference image data, and information on each reference image data is linked by the ID information of the reference image data. The DB server 20 also updates the inventory information of the inventory information table 42 of the product corresponding to the product name information stored in the product name 44a of the intra-image information table 44 (44A, 44B, 44c, . . .) of all the reference image data. That is, the inventory information 42b of the product included in all the reference image data is set to in stock, and the inventory information 42b of the product not included in all the reference image data is set to out of stock. In this way, the reference information (master data) related to each reference image data is stored in the database 40 of the DB server 20, and the image information registration process for the reference image data is completed. When the DB server 20 receives the reference image data from the update server 30, the DB server 20 receives the reference image data together with the information received from the update server 30 in the image information registration process for the reference image data described above, and the reference image data and its ID information are stored in the store image 43a and image ID 43b of the store image table 43 in the database 40, respectively.

[0075] Next, the flow of the image information registration process performed by the information processing system 1 will be described. FIG. 13 is a flowchart showing an example of the flow of the image information registration process performed by the information processing system 1. As shown in FIG. 13, in the image information registration process, first, the store terminal 10 transmits image data (step S1). In the image information registration process for the reference image data described above, the store terminal 10 transmits the reference image data. In step S1, for example, the image data is transmitted to the DB server 20. Next, the DB server 20 stores the received image data in the store image 43a of the store image table 43 of the database 40, and also stores the ID information in the image ID 43b (step S2). Note that in the store image table 43, the image data and the ID information are linked. Next, the update server 30 accesses the DB server 20, accesses the store image table 43 of the database 40, receives the image data to be processed, and executes the product specification process on the image data (step S3). In the product identification process, the update server 30 then creates coordinate information, character information, category information, and product name information for each product included in the image data to be processed, and also creates representative category information for the image data to be processed. The update server 30 then transmits these pieces of information to the D server 20. The DB server 20 stores the received coordinate information, character information, category information, representative category information, and product name information in the database 40 in association with each piece of product name information (step S4), and the image registration process ends. In step S4, specifically, the product name information, coordinate information, character information, and category information are stored in the product name 44a, coordinate information 44b, product identification information 44c, and category information 44d of the intra-image information table 44, respectively. Furthermore, the representative category information is stored in the representative category 43c of the store image table 43. In the intra-image information table 44, information related to the image data to be processed is linked by the ID information of this image data. In step S4, the inventory information in the inventory information table 42 is updated based on the product name information stored in the product name 44a in the in-image information table 44.

[0076] In step S1, the image data may be transmitted from the store terminal 10 to the update server 30. In this case, the process of step S2 described above is not performed, and in step S3, the update server 30 executes product identification processing on the received image. Then, in step S4, the DB server 20 receives the image data to be processed from the update server 30 together with information generated in the product identification processing, and stores this information in the database 40. In this case, the DB server 20 stores the image data and its ID information in the store image 43a and image ID 43b of the store image table 43 in the database 40, respectively.

[0077] Next, the flow of the above-mentioned product identification process executed by the update server 30 in the image information registration process will be described. FIG. 14 is a flowchart showing an example of the flow of the above-mentioned product identification process executed by the update server 30. In the above-mentioned image information registration process for the reference image data, the product identification process is executed for the reference image data. In the product identification process, first, the product recognition unit 201 generates coordinate information of the product included in the image data (step S11). Next, the character extraction unit 202 extracts character information (text information) from the image data (step S12). In step S12, character information is extracted within each product area specified by each piece of coordinate information generated in step S11. Next, the extracted character classification unit 203 specifies category information for each piece of character information extracted in step S12 (step S13). Next, the image data classification unit 204 specifies representative category information of the image data to be processed based on the category information specified in step S13 (step S14). Next, the product identification unit 205 identifies product name information of each product included in the image data to be processed (step S15), as described above, and the product identification process ends.

[0078] Next, the inventory management process will be specifically described. In the inventory management process, for example, inventory management of products is performed based on information on the reference image data registered by the image information registration process for the reference image data described above. That is, in the inventory management process, the user photographs a part of the product shelf displayed by all the reference image data with the camera 11 of the store terminal 10 at a time after the time when the reference image data (first image data) was created, creates image data (second image data) (hereinafter referred to as target image data), and performs inventory management of products at the location displayed by this target image data. As an example, the target image data is assumed to be obtained by photographing a part of one product shelf. Note that the target image data is not limited to image data obtained by photographing a part of one product shelf. For example, the target image data may be image data obtained by photographing the entire product shelf, or image data obtained by photographing a part or all of a plurality of product shelves. FIG. 15 is a diagram showing an example of the target image data. Furthermore, in the store terminal 10, ID information that enables the created target image data to be identified is created and attached to the target image data. The target image data to which the ID information is attached is stored, for example, in the ROM 14.

[0079] The inventory management process is executed, for example, by a user starting an inventory management application in the store terminal 10. In the inventory management process, first, the target image data stored in the ROM 14 of the store terminal 10 is transmitted from the communication unit 17 of the store terminal 10 to the communication unit 25 of the DB server 20. The target image data received by the DB server 20 is stored in the store image 43a of the store image table 43 of the database 40, and the ID information is stored in the image ID 43b. In the store image table 43, the target image data is linked to the ID information. The target image data may also be transmitted to the communication unit 34 of the update server 30. The target image data may be transmitted only to the update server 30 without being transmitted to the DB server 20. In this case, the DB server 20 receives the reference image data via the update server 30.

[0080] In the inventory management process, the information processing system 1 executes the image information registration process for the target image data in the same manner as for the above-mentioned reference image data (see Figs. 13 and 14), and information including the product name information of each product included in the target image data is specified. Similarly, the product name information, coordinate information, character information, and category information of each product included in the target image data, as well as the representative category information of the target image data, are stored in the database 40 of the DB server 20. That is, the product name information, coordinate information, character information, and category information are stored in the product name 44a, coordinate information 44b, product identification information 44c, and category information 44d of the intra-image information table 44, respectively. Furthermore, the representative category information is stored in the representative category 43c of the store image table 43. The intra-image information table 44 has ID information of the corresponding reference image data, and information related to the target image data is linked by the ID information of the target image data.

[0081] Next, the image range specifying unit 206 of the update server 30 executes a target range specifying process. The target range specifying process is a process of specifying reference image data, which is image data that displays the same store location as the store location displayed by the target image data, from among a plurality of basic image data stored in the store image 43a in the store image table 43 in the database 40 of the DB server 20, and specifying the range of the display image displayed by the target image data within the display image of this specified reference image data.

[0082] In the target range specification process, the image range specification unit 206 first refers to the product identification information 44c in the store image table 44 of the target image data, and also refers to the product identification information 44c in the store image table 44 of all the reference image data, and performs a matching process for character information on all the products included in the target image data with character information on all the products included in all the reference image data. Then, the ID information of the reference image data including character information determined to match each character information on each product included in the target image data is stored in the ROM 32 or the RAM 33, for example, in association with each product or character information on each product in the target image data. Then, the most frequent ID information is specified from the stored ID information, and the reference image data corresponding to this most frequent ID information is set as the reference image data. The ID information of the reference image data is stored in the ROM 32, for example. FIG. 16 is a diagram showing an example of reference image data.

[0083] Next, the image area specifying unit 206 specifies the area displayed by the target image data within the display image of the reference image data. The image area specifying unit 206 also calculates specific area information, which is information representing the area within the display image of the reference image data specified as the area displayed by the target image data. The specific area information calculated by the image area specifying unit 206 is stored in the ROM 32. In this way, by the target area specifying process, a area (target reference area TA) that displays the same location as the location displayed by the target image data is specified in the reference image data, as shown in Fig. 16.

[0084] Next, the image range specification unit 206 creates a list (specific range product list) of product name information of products in the target reference range TA of the reference image data specified by the target range specification process. The specific range product name list is created by referring to the database 40 of the DB server 20 based on the ID information and specific range information of the reference image data. Specifically, coordinate information included in the range represented by the specific range information is selected from the coordinate information 44b of the intra-image information table 44 of the reference image data linked to the ID information of the reference image data stored in the image ID 43b of the store image table 43, and product name information linked to the selected coordinate information is extracted from the product name information stored in the product name 44a to create the specific range product list. The created specific range product list is linked to the ID information of the target image data, for example, and stored in the ROM 32.

[0085] Next, the inventory information update unit 207 refers to the product name 44c in the store image table 44 of the target image data and the specific range product name list created by the image range specification unit 206, and compares the product name information of the product displayed in the target image data with the product name information of the product displayed in the target reference range TA of the reference image data. Specifically, it compares the product name information stored in the product name 44c in the store image table 44 of the target image data with the product information name in the specific range list. Then, the inventory information update unit 207 regards a product represented by product name information that is in the specific range list but not stored in the product name 44c in the store image table 44 of the target image data as a stock-out product. In addition, the inventory information update unit 207 regards a product represented by product name information that is not in the specific range list and is stored in the product name 44c in the store image table 44 of the target image data as in stock.

[0086] Next, the inventory information update unit 207 accesses the database 40 of the DB server 20 and updates the inventory information stored in the inventory information 42b of the inventory information table 42. Specifically, when the inventory information stored in the inventory information 42b of the inventory information table 42, which corresponds to the product name information of the product considered to be out of stock, indicates that the product is in stock, the inventory information update unit 207 changes the inventory information stored in the inventory information 42b to information indicating that the product is out of stock. On the other hand, when the inventory information stored in the inventory information 42b of the inventory information table 42, which corresponds to the product name information of the product considered to be out of stock, indicates that the product is out of stock, the inventory information update unit 207 does not change the inventory information stored in the inventory information 42b.

[0087] Furthermore, when the inventory information stored in inventory information 42b of inventory information table 42, which corresponds to the product name information of a product considered to be in stock, indicates that the product is out of stock, inventory information update unit 207 changes the inventory information stored in inventory information 42b to information indicating that the product is in stock. On the other hand, when the inventory information stored in inventory information 42b of inventory information table 42, which corresponds to the product name information of a product considered to be in stock, indicates that the product is in stock, inventory information update unit 207 does not change the inventory information stored in inventory information 42b.

[0088] Furthermore, the inventory information update unit 207 regards the product represented by the product name information in the specific range list and stored in the product name field 44c of the store image table 44 of the target image data as a replenishment confirmation product, and checks whether the product has been replenished. That is, the inventory information update unit 207 accesses the database 40 of the DB server 20, and if the inventory information stored in the inventory information 42b of the inventory information table 42 corresponding to the product name information of the product regarded as a replenishment confirmation product indicates out of stock, the inventory information update unit 207 changes the inventory information stored in the inventory information 42b to information indicating in stock. On the other hand, if the inventory information stored in the inventory information 42b indicates in stock, the inventory information stored in the inventory information 42b is not changed.

[0089] The inventory information update unit 207 performs the above-mentioned comparison process between the product name information of the target image data and the product information name in the specific range list for all product name information stored in the product name field 44c of the store image table 44 of the target image data. Note that the inventory information update unit 207 does not perform update processing for products displayed outside the target reference range TA of the reference image data. In this way, when the inventory information update unit 207 ends, the inventory management process ends.

[0090] Next, the flow of the above-mentioned inventory management process performed by the information processing system 1 will be described. FIG. 17 is a flowchart showing an example of the flow of the above-mentioned inventory management process performed by the information processing system 1. As shown in FIG. 17, in the inventory management process, first, the image information registration process is performed on the target image data in the same manner as the reference image data (see FIGS. 13 and 14) (step S21). By the image information registration process of step S21, coordinate information, character information, category information, and product name information of each product included in the target image data are created, and representative category information of the target image data is also created. Then, the product name information, coordinate information, character information, and category information are stored in the product name 44a, coordinate information 44b, product identification information 44c, and category information 44d of the intra-image information table 44 of the database 40 of the DB server 20, respectively. Furthermore, the representative category information is stored in the representative category 43c of the store image table 43 of the database 40. In the intra-image information table 44, information on the image data to be processed is linked by the ID information of this image data.

[0091] Next, the image range specifying unit 206 of the update server 30 executes a target range specifying process (step S22). The target range specifying process of step S22 specifies reference image data, which is image data that displays the same store location as the store location displayed by the target image data, from among a plurality of basic image data stored in the store image 43a of the store image table 43 of the database 40 of the DB server 20. Also, specific range information, which is information that represents a target reference range TA, which is a range displayed by the target image data in the display image of the reference image data, is calculated. In other words, by comparing the first image data and the second image data as a reference, it is possible to calculate (specify) the overlapping range. Next, the image range specifying unit 206 creates a specific range product list, which is a list of product name information of products included in the target reference range TA of the reference image data specified by the target range specifying process (step S23).

[0092] Next, the inventory information update unit 207 executes an inventory information update process (step S24). The inventory information update process refers to the product name 44c in the store image table 44 of the target image data and the specific range product name list created by the image range specification unit 206, and updates the inventory information stored in the inventory information 42b in the inventory information table 42. FIG. 18 is a flowchart showing an example of the flow of the inventory information update process executed by the inventory information update unit 207.

[0093] As shown in Fig. 18, in the inventory information update process, the inventory information update unit 207 first accesses the data table 40 of the DB server 20, refers to the product name 44c in the store image table 44 of the target image data, and also refers to the specific range product list created in step S23, and compares the product name information of each product included in the target image data with the product information name in the specific range list. In this comparison process, the inventory information update unit 207 first determines whether the product name information in the specific range product list is stored in the product name 44c in the store image table 44 of the target image data (step S31). In step S31, if it is determined that the product name information in the specific range product list is not stored in the product name 44c in the store image table 44 of the target image data (No in step S31), the product represented by this product name information is considered to be out of stock (step S32). On the other hand, if it is determined in step S31 that the product name information in the specific range product list is stored in the product name 44c of the store image table 44 of the target image data (Yes in step S31), the process proceeds to step S36 described later.

[0094] Next, the inventory information update unit 207 accesses the database 40 of the DB server 20 and updates the inventory information stored in the inventory information 42b of the inventory information table 42. Specifically, if the inventory information stored in the inventory information 42b of the inventory information table 42, which corresponds to the product name information of the product deemed to be out of stock in steps S31 and S32, is information indicating availability (Yes in step S33), the inventory information update unit 207 changes the inventory information stored in the inventory information 42b to information indicating out of stock (step S34). On the other hand, if the inventory information stored in the inventory information 42b of the inventory information table 42, which corresponds to the product name information of the product deemed to be out of stock, is information indicating out of stock (No in step S33), the inventory information update unit 207 does not change the inventory information stored in the inventory information 42b (step S35). After steps S34 and S35, the inventory information update process proceeds to step S39, which will be described later.

[0095] On the other hand, if the answer is Yes in step S31, the inventory information update unit 207 then checks whether the product has been replenished. That is, if the inventory information stored in the inventory information 42b of the inventory information table 42 corresponding to the product name information stored in the product name 44c of the store image table 44 of the target image data is information indicating out of stock (No in step S36), the inventory information update unit 207 changes the inventory information stored in the inventory information 42b to information indicating in stock (step S37). This appropriately updates the inventory information of the product that was once out of stock and then replenished. On the other hand, if the inventory information stored in the inventory information 42b of the inventory information table 42 corresponding to the product name information stored in the product name 44c of the store image table 44 of the target image data is information indicating in stock (Yes in step S36), the inventory information update unit 207 does not change the inventory information stored in the inventory information 42b (step S38). Next, the inventory information update process proceeds to step S39.

[0096] In step S39, the inventory information update unit 207 executes the above-mentioned comparison and update process for all product name information stored in the product name 44c of the store image table 44 of the target image data (step S39), and the inventory information update process ends.

[0097] As shown in FIG. 17, in the inventory management process, when the inventory information update process in step S24 ends, the inventory management process ends.

[0098] As described above, according to the information processing system 1 that executes the information processing program of the embodiment of the present invention, reference image data (first image data) that displays a store in a reference state is created, and target image data (second image data) that displays a sales floor of products for which inventory management is desired after the reference state is created, thereby making it possible to perform inventory management of products in the sales floor (first range) displayed by the target image data. Furthermore, inventory management of products in sales floors (second range) not displayed by the target image data is not performed. In this way, according to the information processing system 1 of the embodiment of the present invention, appropriate management of inventory information of products in a store can be easily performed.

[0099] Furthermore, by having the information processing system 1 execute the above-mentioned inventory management process at predetermined time intervals, inventory management of products in the sales area displayed by the target image data can be easily performed on a regular basis. Furthermore, by creating multiple sets of target image data, inventory management of multiple store locations can be easily performed.

[0100] Although the present invention has been described above through the above embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications or improvements can be made to the above embodiments. It is clear from the claims that such modifications or improvements can also be included in the technical scope of the present invention.

[0101] The above-described embodiments are intended to facilitate understanding of the present invention, and are not intended to limit the present invention. Moreover, the above-described embodiments do not limit the use of the present invention, and the present invention may include anything as its use. The components of the above-described embodiments, as well as their arrangement, materials, conditions, shapes, sizes, etc., are not limited to those exemplified, and can be changed as appropriate. For example, the present invention includes differences that occur in the implementation of manufacturing tolerances, etc. Furthermore, the components shown in different embodiments can be partially replaced or combined with each other within a range that is not technically contradictory. Furthermore, each configuration can be appropriately and selectively combined so as to achieve at least a part of the above-described problems and effects.

[0102] The program for executing the processing of the information processing system 1 described in this embodiment may be stored in any appropriate computer-readable recording medium. The computer-readable recording medium is any appropriate computer-readable recording medium for a program to a processor, and includes a non-transitory computer-readable storage medium. [Explanation of symbols]

[0103] 1 Information processing system, 2 Network, 10 Store terminal, 11 Camera, 12 Display unit, 13 CPU, 14 ROM, 15 RAM, 16 Input unit, 17 Communication unit, 18 Bus line, 20 DB server, 21 Storage device, 22 CPU, 23 ROM, 24 RAM, 25 Communication unit, 26 Bus line, 30 Update server, 31 CPU, 32 ROM, 33 RAM, 34 Communication unit, 35 Bus line, 40 Database, 41 Store product information table, 41a Product name, 41b Handling, 41c Product category, 41d Product position, 42 Stock information table, 42a Product name, 42b Stock information, 43 Store image table, 43a Store image, 43b Image ID, 43c Representative category, 44 In-image information table, 44a Product name, 44b Coordinate information, 44c Product identification information, 44d Category information, 101 Image forming unit, 102 image transmission unit, 103 operation recognition unit, 201 product recognition unit, 202 character extraction unit, 203 extracted character classification unit, 204 image data classification unit, 205 product identification unit, 206 image range identification unit, 207 inventory information update unit, A (A1, A2, A3...) product area, M (M1, M2, M3...) product, TA target reference range

Claims

1. An information processing program for managing inventory information of products in a store, On the computer, An image acquisition step of acquiring image data obtained by photographing at least a part of the inside of the store; an inventory information updating step of updating the inventory information of the store based on the product information included in the image data; the inventory information update step updates the inventory information by comparing a text indicating a name of the product obtained from the image data with master data including the names of each of the products handled in the store; Information processing program.

2. An information processing program for managing inventory information of products in a store, On the computer, An image acquisition step of acquiring first image data obtained by photographing at least a portion of the interior of the store, and second image data obtained by photographing at least a portion of the interior of the store after the photographing of the first image data; an inventory information updating step of updating the inventory information of the store based on the product information included in the second image data; The inventory information update step includes: When the first image data includes a first range and a second range in the store, and the second image data includes the first range in the store but does not include the second range, updating inventory information for a first product included in the first range of the first image data if the first product is not included in the second image data; when a second product included in the second range of the first image data is not included in the second image data, not updating inventory information about the second product; Information processing program.

3. In the image acquisition step, first image data and second image data obtained by capturing an image after the first image data are captured are acquired as the image data. The inventory update step includes: When the first image data includes a first range and a second range in the store, and the second image data includes the first range in the store but does not include the second range, updating inventory information for a first product included in the first range of the first image data if the first product is not included in the second image data; if a second product included in the second range of the first image data is not included in the second image data, do not update inventory information about the second product; The text to be compared with the master data is text obtained from product information included in the second image data. The information processing program according to claim 1 .

4. The method further includes a step of: comparing the first image data with the second image data to specify a range of the second image data within the range of the first image data; the first range is a range of the second image data, The second range is another range of the second range in the range of the first image data.

4. The information processing program according to claim 2 or 3.

5. In the range specifying step, a range of the second image data is specified within a range of the first image data by a matching process between product information included in the first image and product information included in the second image. The information processing program according to claim 4.

6. The first image data includes a plurality of image data obtained by photographing different sections in the store, The range specifying step performs a matching process for each piece of product information included in the second image data with product information included in the plurality of image data held by the first image data, and specifies image data of the first image data that matches most in the matching process as image data that includes the range of the second image data.

6. An information processing program according to claim 5.

7. The product information includes text. The information processing program according to claim 2.

8. A product name specification step of specifying a product name included in the second image data is further performed by the computer. The product name identification step includes: a category identification step of identifying a product category of each of the product information based on the product information included in the second image data; a product name matching step of performing a matching process for matching the product information included in the second image data with the names of products in the store that belong to each of the product categories identified by the category identification step, and identifying the names of the products included in the second image data.

4. The information processing program according to claim 2 or 3.

Images