Information processing device, information processing system, store operations support method and program

Abstract

This information processing device comprises: an image acquisition unit that acquires a first image of fixtures imaged by a portable communication device and a second image of fixtures imaged by a fixed-point camera; a planogram information generation unit that processes the first image to generate, for each product present in the first image, planogram information associating product area information indicating the position of an area where said product is displayed with product information pertaining to said product; a correspondence relationship information generation unit that generates correspondence relationship information indicating a correspondence relationship between a position in the first image and a position in the second image on the basis of feature information pertaining to a common indicator object present in each of the first image and the second image; and an information storage unit that uses the correspondence relationship information to convert the product area information for each product included in the planogram information to a position in the second image, and stores, in a predetermined storage area, planogram information in which the product area information for each product has been converted.

Description

Information processing device, information processing system, store operation support method, and non-transitory computer-readable recording medium

[0001] The present invention relates to an information processing device, an information processing system, a store operation support method, and a non-transitory computer-readable recording medium.

[0002] Patent Document 1 listed below discloses a system that detects products on display shelves arranged in a store and supports operations when a shortage is detected. The system described in Patent Document 1 performs the following processes, for example: (1) An appearance image of an item included in a first shelf image taken of the display shelf of the items is associated with the placement position of the item on the display shelf indicated by the appearance image and stored. (2) An appearance image stored in a storage unit is detected from a second shelf image taken of the display shelf after the shelf image, and notification regarding the placement of the item indicated by the appearance image is controlled based on the detection result.

[0003] Japanese Patent Application Laid-Open No. 2021-081911

[0004] In order to photograph fixtures on which products are displayed with a fixed camera and provide services based on the images, it is necessary to prepare in advance information indicating where each product is located in the image captured by the fixed camera. This preparation work can be performed, for example, by a store clerk manually entering the information, but this requires a significant workload. Furthermore, product layouts in stores can be changed on a daily basis. Because the preparation work must be repeated every time the product layout is changed, the preparation work described above can place a significant burden on the store clerk.

[0005] The information processing device of the present disclosure comprises: an image acquisition means for acquiring a first image of a fixture taken by a portable communication device and a second image of the fixture taken by a fixed camera; a shelf allocation information generation means for processing the first image to generate, for each product present in the first image, shelf allocation information that associates product area information indicating the position of the area where the product is displayed with product information related to the product; a correspondence information generation means for generating correspondence information that indicates the correspondence between positions on the first image and positions on the second image based on feature information related to a common index object that is present in the first image and the second image; and an information storage means for converting the product area information of each product included in the shelf allocation information into a position on the second image using the correspondence information, and storing the shelf allocation information after the product area information of each product has been converted in a predetermined storage area.

[0006] The information processing system of the present disclosure comprises: a fixed camera installed in a store; a portable communication device used in the store; an image acquisition means for acquiring a first image of fixtures captured by the portable communication device and a second image of the fixtures captured by the fixed camera; a shelf allocation information generation means for processing the first image to generate, for each product present in the first image, shelf allocation information that associates product area information indicating the position of an area in which the product is displayed with product information related to the product; a correspondence information generation means for generating correspondence information that indicates the correspondence between positions on the first image and positions on the second image based on feature information related to a common index object that is present in the first image and the second image; and an information storage means for converting the product area information of each product included in the shelf allocation information into a position on the second image using the correspondence information, and storing the shelf allocation information after the product area information of each product has been converted in a predetermined storage area.

[0007] The store operation support method disclosed herein includes: at least one computer acquiring a first image of fixtures taken by a portable communication device and a second image of the fixtures taken by a fixed camera; processing the first image to generate, for each product present in the first image, shelf allocation information that associates product area information indicating the position of the area in which the product is displayed with product information related to the product; generating correspondence information that indicates the correspondence between positions on the first image and positions on the second image based on feature information related to a common index object that is present in the first image and the second image; converting the product area information of each product included in the shelf allocation information into a position on the second image using the correspondence information; and storing the shelf allocation information after the product area information of each product has been converted in a predetermined memory area.

[0008] A first non-transitory computer-readable recording medium in the present disclosure records a program for causing at least one computer to execute operations including: acquiring a first image of a fixture taken by a portable communication device and a second image of the fixture taken by a fixed camera; processing the first image to generate, for each product present in the first image, shelf allocation information that associates product area information indicating the position of the area in which the product is displayed with product information related to the product; generating correspondence information that indicates the correspondence between positions on the first image and positions on the second image based on feature information related to a common index object that is present in the first image and the second image; converting the product area information of each product included in the shelf allocation information into a position on the second image using the correspondence information; and storing the shelf allocation information after the product area information of each product has been converted in a predetermined storage area.

[0009] A second non-transitory computer-readable recording medium in the present disclosure records a program for causing a portable communication device to perform operations including acquiring guide information based on an image of the fixture to be photographed generated by a fixed camera, and outputting the guide information on a photographing screen displayed on a display of the communication device when photographing.

[0010] According to the present disclosure, the workload of store staff who manage product observation services using fixed cameras can be reduced.

[0011] 1 is a diagram illustrating a first example of a functional configuration of an information processing device according to the present disclosure. FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer that realizes the information processing device according to the present disclosure. FIG. 3 is a flowchart illustrating a first example of operation of an information processing device according to the present disclosure. FIG. 4 is a diagram illustrating an example of information stored by an information storage unit. FIG. 5 is a diagram illustrating an example of a configuration of an information processing system including an information processing device according to the present disclosure. FIG. 6 is a diagram illustrating a first example of a functional configuration of a portable communication device according to the present disclosure. FIG. 7 is a diagram illustrating an example of a hardware configuration of a computer that realizes the portable communication device according to the present disclosure. FIG. 8 is a sequence diagram illustrating a first example of operation of an information processing system according to the present disclosure. FIG. 9 is a flowchart illustrating a first example of operation of a communication device according to the present disclosure. FIG. 10 is a diagram illustrating an example of guide information generated and output by a guide information output unit. FIG. 11 is a diagram illustrating an example of guide information generated and output by a guide information output unit. FIG. 12 is a diagram illustrating a second example of a functional configuration of an information processing device according to the present disclosure. FIG. 13 is a flowchart illustrating a second example of operation of an information processing device according to the present disclosure. FIG. 10 is a diagram showing another example of a screen output by the display output unit to the terminal for a store clerk.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In this disclosure, the drawings relate to one or more embodiments. In all drawings, similar components are denoted by similar reference numerals, and descriptions thereof will be omitted where appropriate. Unless otherwise specified, each block in each block diagram represents a functional configuration, not a hardware configuration. When elements such as blocks in a diagram are linked by arrows, the direction of the arrows is merely intended to make the flow of information easier to understand. In this case, the direction of the arrows does not limit the direction of communication (one-way communication / two-way communication) unless otherwise specified.

[0013] In the following description, "acquisition" includes at least one of the following: a device going to retrieve data or information stored in another device or storage medium (active acquisition), and a device inputting data or information output from another device (passive acquisition). Examples of active acquisition include making a request or inquiry to another device and receiving a reply, and accessing and reading information from another device or storage medium. Examples of passive acquisition include receiving information that is distributed (or transmitted, push notification, etc.). Furthermore, "acquisition" may also mean selecting and acquiring data or information from received data or information, or selecting and receiving distributed data or information.

[0014] First Embodiment <Functional Configuration> Fig. 1 is a diagram showing a first example of the functional configuration of an information processing device according to the present disclosure. The information processing device 10 shown in Fig. 1 includes an image acquisition unit 110, a shelf allocation information generation unit 120, a correspondence information generation unit 130, and an information storage unit 140.

[0015] The image acquisition unit 110 acquires images of fixtures installed in a store. Specifically, the image acquisition unit 110 acquires images of the fixtures taken by a portable communication device (not shown) used by a store clerk and images of the fixtures taken by a fixed camera (not shown) installed in the store. In the following description, the former image will be referred to as a "first image" and the latter image will be referred to as a "second image."

[0016] The shelf allocation information generation unit 120 processes the first image to generate information regarding the shelf allocation of the target fixture (hereinafter also referred to as "shelf allocation information"). For each product present in the first image, the shelf allocation information generation unit 120 identifies the position of the area in which the product is displayed and generates information indicating the position of that area (hereinafter also referred to as "product area information"). In addition, the shelf allocation information generation unit 120 generates information regarding each product present in the first image (hereinafter also referred to as "product information"). The shelf allocation information generation unit 120 can generate shelf allocation information for the target fixture by associating the product area information generated for each product present in the first image with the product information.

[0017] The correspondence information generating unit 130 generates information indicating the correspondence between positions on a first image generated by a portable communication device and positions on a second image generated by a fixed camera (hereinafter, "correspondence information"). The correspondence information generating unit 130 can generate the correspondence information based on feature information about a common index object that exists in each of the first image and the second image. The correspondence information generating unit 130 can obtain the correspondence information based on the feature information about the index object in each image, for example, by using a known image processing algorithm that specifies the positional correspondence between multiple images.

[0018] The information storage unit 140 processes the shelf allocation information generated by the shelf allocation information generation unit 120 based on the correspondence information generated by the correspondence information generation unit 130. The information storage unit 140 converts the product area information of each product included in the shelf allocation information into information indicating its position on the second image using the correspondence information. The information storage unit 140 then stores the shelf allocation information after converting the product area information of each product in a predetermined storage area.

[0019] <Example of Hardware Configuration of Information Processing Device> FIG. 2 is a diagram illustrating an example of the hardware configuration of a computer that realizes the information processing device according to the present disclosure.

[0020] The computer 1000 illustrated in this figure includes a bus 1010 , a processor 1020 , a memory 1030 , a storage device 1040 , an input / output interface 1050 , and a network interface 1060 .

[0021] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0022] The processor 1020 is a central processing unit (CPU) or a graphics processing unit (GPU).

[0023] The memory 1030 is a main storage device realized by a random access memory (RAM) or the like.

[0024] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), etc. The storage device 1040 stores at least program modules that realize the functions of the information processing device 10 described above (the image acquisition unit 110, the shelf allocation information generation unit 120, the correspondence information generation unit 130, and the information storage unit 140).

[0025] The processor 1020 loads a program module read from the storage device 1040 onto the memory 1030 and executes the program module, thereby realizing the function corresponding to the program module. For example, the processor 1020 reads a program module corresponding to the image acquisition unit 110 onto the memory 1030 and executes the program module, thereby realizing the function of the image acquisition unit 110 described in this disclosure. Similarly, the processor 1020 reads a program module corresponding to the shelf allocation information generation unit 120 onto the memory 1030 and executes the program module, thereby realizing the function of the shelf allocation information generation unit 120 described in this disclosure. Similarly, the processor 1020 reads a program module corresponding to the correspondence information generation unit 130 onto the memory 1030 and executes the program module, thereby realizing the function of the correspondence information generation unit 130 described in this disclosure. Similarly, the processor 1020 reads a program module corresponding to the information storage unit 140 onto the memory 1030 and executes the program module, thereby realizing the function of the information storage unit 140 described in this disclosure. This operation of the processor 1020 is common to each embodiment included in this disclosure.

[0026] The above-described program modules may be recorded on a recording medium other than the storage device 1040. The recording medium on which the program modules are recorded may include any non-transitory, tangible medium usable by the computer 1000. The recording medium on which the program modules are recorded may have embedded therein program code that is readable by the computer 1000 (processor 1020).

[0027] The input / output interface 1050 is an interface for connecting the computer 1000 (information processing device 10) to various input / output devices.

[0028] The network interface 1060 is an interface for connecting the computer 1000 (information processing device 10) to a network. This network includes, for example, a local area network (LAN) and a wide area network (WAN). The network interface 1060 connects the computer 1000 (information processing device 10) to various networks via a wireless or wired connection. For example, the computer 1000 (information processing device 10) is connected via the network interface 1060 to other devices such as a portable communication device used by a store clerk or a fixed camera installed in a store.

[0029] The computer 1000 (information processing device 10) can also be connected to various devices (not shown) via the input / output interface 1050 or the network interface 1060. The "various devices" referred to here are not particularly limited, but include, for example, input devices such as operation buttons, a keyboard, a touch panel, a mouse, and a microphone, and output devices such as a display, a speaker, and a printer.

[0030] <Example of Operation of Information Processing Apparatus> The operation of the information processing apparatus according to the present disclosure will now be described with reference to Figure 3. Figure 3 is a flowchart showing a first example of the operation of the information processing apparatus according to the present disclosure.

[0031] First, the image acquisition unit 110 acquires an image (first image) of the target fixture from a portable communication device used by a store clerk (step S102).The image acquisition unit 110 also acquires an image (second image) of the target fixture from a fixed camera placed in a position where the target fixture can be photographed (step S104).

[0032] The shelf allocation information generation unit 120 generates shelf allocation information based on the first image acquired from the portable communication device used by the store clerk (step S106). The shelf allocation information generation unit 120 can identify the area in the first image where each product is displayed using known image recognition technology. The shelf allocation information generation unit 120 can also acquire information about the products displayed in each identified area (e.g., information identifying the product, the product name, the product selling price, etc.). The shelf allocation information generation unit 120 generates shelf allocation information by associating product area information with product information for each product present in the first image.

[0033] The correspondence information generating unit 130 generates correspondence information indicating the correspondence between positions on the first image and positions on the second image (step S108). As an example, the correspondence information generating unit 130 detects a common index object in each of the first image and the second image. Then, based on the positional relationship of the index objects detected in each image, the correspondence information generating unit 130 generates, as correspondence information, transformation parameters for transforming a position on one image into a position on the other image.

[0034] The information storage unit 140 uses the correspondence information generated in step S108 to convert the product area information of each product included in the shelf planogram information generated in step S106 into information indicating a position on the second image (step S110). The information storage unit 140 then stores the shelf planogram information after converting the product area information of each product in a predetermined storage area, such as the memory 1030 or the storage device 1040 (step S112). The information storage unit 140 stores, for example, information such as that shown in FIG. 4 in a predetermined storage area. FIG. 4 is a diagram showing an example of information stored by the information storage unit 140. In the example of FIG. 4, the information storage unit 140 stores the shelf planogram information after the conversion process of step S110 in association with the identification information (camera ID) of the corresponding fixed camera.

[0035] In this manner, shelf allocation information for a fixture is generated based on an image obtained by photographing the target fixture using a portable communication device. This eliminates the need for a store clerk to manually input information about the shelf allocation for the fixture. Here, the shelf allocation information includes product area information indicating the area where each product is displayed, but the product area information is based on the coordinate system of the first image (a coordinate system based on the communication device). According to the information processing device 10 of the present disclosure, the product area information for each product included in the shelf allocation information is converted into information indicating its position in the coordinate system of the second image (a coordinate system based on the fixed camera) using correspondence information indicating the correspondence between the positions of the first image and the second image. In other words, the shelf allocation information based on the coordinate system of the first image is converted into shelf allocation information based on the coordinate system of the second image. The converted shelf allocation information is then stored in a predetermined memory area. This makes it possible to utilize shelf allocation information automatically generated from an image (first image) generated by the communication device in processing using an image (second image) generated by a fixed camera. In other words, according to the present disclosure, an information processing device, a store operation support method, and a non-transitory computer-readable recording medium that stores a program that realizes the functions of the information processing device are provided, which solve the problem of reducing the workload of store staff who manage product observation services using fixed cameras.

[0036] Furthermore, as will be described in detail later, an information processing system having the same effect by including the information processing device is also provided. Also, as will be described in detail later, a non-transitory computer-readable recording medium is provided that stores a program for operating a portable communication device used in the information processing system so as to achieve the above-described effect.

[0037] Below, a detailed example of an information processing system including an information processing device according to the present disclosure and a portable communication device used in the information processing system will be described.

[0038] <System Configuration Example> Fig. 5 is a diagram illustrating the configuration of an information processing system including an information processing device according to the present disclosure. The information processing system 1 shown in the example of this diagram is configured to include an information processing device 10, a communication device 20, and a fixed camera 30. In the example of this diagram, the information processing device 10, the communication device 20, and the fixed camera 30 are each connected via an arbitrary network 40. Although not clearly illustrated, multiple information processing devices 10, multiple communication devices 20, and / or multiple fixed cameras 30 may be connected via the network 40.

[0039] The information processing device 10 has, for example, the functions described with reference to Fig. 1. The information processing device 10 may also have other functions as will be described in other embodiments below.

[0040] The communication device 20 is a portable device such as a smartphone or tablet terminal used by a store clerk in the store. The communication device 20 has a camera function. The store clerk takes a picture of the target fixture 50 using the camera function of the communication device 20. As a result, the communication device 20 generates an image of the target fixture 50, i.e., the above-mentioned first image. The communication device 20 transmits the first image to the information processing device 10 via the network 40.

[0041] The fixed camera 30 is positioned by adjusting its position and attitude so that the target fixture 50 is included in the imaging range. The fixed camera 30 then begins imaging the target fixture 50. As a result, the fixed camera 30 generates an image of the target fixture 50, i.e., the second image described above. The fixed camera 30 transmits the second image to the information processing device 10 via the network 40.

[0042] <Functional Configuration of Communication Device> FIG. 6 is a diagram illustrating a first example of the functional configuration of a portable communication device according to the present disclosure. The communication device 20 illustrated in FIG. 6 includes an image generation unit 210 and a guide information output unit 220. When photographing using the communication device 20, the image generation unit 210 displays a screen (hereinafter also referred to as a "photography screen") on the display of the communication device 20, displaying a preview of an image generated by the camera function of the communication device 20. The guide information output unit 220 acquires guide information based on an image of the fixtures to be photographed generated by the fixed camera 30. The guide information output unit 220 then outputs the guide information based on the image of the fixtures generated by the fixed camera 30 on the photographing screen displayed by the image generation unit 210. Here, the "guide information" is information that guides the user of the communication device 20 (controls the photographing operation of the user of the communication device 20) to facilitate association between the first image and the second image.

[0043] <Example of Hardware Configuration of Communication Device> FIG. 7 is a diagram illustrating an example of the hardware configuration of a computer that realizes a portable communication device in the present disclosure.

[0044] The computer 2000 illustrated in this figure has a bus 2010 , a processor 2020 , a memory 2030 , a storage device 2040 , an input / output interface 2050 , and a network interface 2060 .

[0045] The bus 2010 is a data transmission path for transmitting and receiving data among the processor 2020, memory 2030, storage device 2040, input / output interface 2050, and network interface 2060. However, the method of connecting the processor 2020 and the like to each other is not limited to bus connection.

[0046] The processor 2020 is a central processing unit (CPU) or a graphics processing unit (GPU).

[0047] The memory 2030 is a main storage device realized by a random access memory (RAM) or the like.

[0048] The storage device 2040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), etc. The storage device 2040 stores at least a program module that realizes the functions of the communication device 20 described above (the image generation unit 210 and the guide information output unit 220).

[0049] The processor 2020 loads a program module read from the storage device 2040 onto the memory 2030 and executes the program module, thereby realizing a function corresponding to the program module. For example, the processor 2020 reads a program module corresponding to the image generation unit 210 onto the memory 2030 and executes the program module, thereby realizing the function of the image generation unit 210 described in the present disclosure. Similarly, the processor 2020 reads a program module corresponding to the guide information output unit 220 onto the memory 2030 and executes the program module, thereby realizing the function of the guide information output unit 220 described in the present disclosure. The operation of the processor 2020 is common to each embodiment included in the present disclosure.

[0050] The above-described program modules may be recorded on a recording medium other than the storage device 2040. The recording medium on which the program modules are recorded includes any non-transitory, tangible medium usable by the computer 2000. The recording medium on which the program modules are recorded may have embedded therein program code that is readable by the computer 2000 (processor 2020).

[0051] The input / output interface 2050 is an interface for connecting the computer 2000 (communication device 20) with the display 22 and other input / output devices. Here, the "other input / output devices" are not particularly limited, but include, for example, input devices such as operation buttons, keyboards, touch panels, mice, and microphones, and output devices such as speakers.

[0052] The network interface 2060 is an interface for connecting the computer 2000 (communication device 20) to a network. This network includes, for example, a local area network (LAN) and a wide area network (WAN). The network interface 2060 connects the computer 2000 (communication device 20) to various networks via a wireless or wired connection. For example, the computer 2000 (communication device 20) is connected to other devices such as one or more information processing devices 10 and one or more fixed cameras 30 via the network interface 2060.

[0053] <Example of Operation of Information Processing System> FIG. 8 is a sequence diagram showing a first example of operation of the information processing system according to the present disclosure.

[0054] First, the store clerk starts the application installed on the communication device 20 and takes a picture of the target fixture 50 using the camera function of the communication device 20. As a result, the communication device 20 generates an image (first image) of the target fixture 50 (step S202).

[0055] The store is also provided with one or more fixed cameras 30. Each of the one or more fixed cameras 30 installed in the store generates an image of fixtures present within the imaging range of the respective fixed cameras 30 (step S204).

[0056] When the communication device 20 generates an image (first image) of the fixture 50 in the processing of step S202, the communication device 20 transmits the first image to the information processing device 10 via the network 40 (step S206). As a result, the image acquisition unit 110 of the information processing device 10 acquires the first image.

[0057] Furthermore, the image acquisition unit 110 of the information processing device 10 acquires an image (second image) of the fixture 50 generated by the fixed camera 30 (step S208).

[0058] Here, when a store has multiple fixtures, at least one fixed camera 30 may be installed for each fixture. In this case, the image acquisition unit 110 of the information processing device 10 needs to identify, from among the multiple fixed cameras 30, the fixed camera 30 that is capturing the same fixture 50 captured by the communication device 20. In such a case, as an example, the image acquisition unit 110 of the information processing device 10 further acquires information identifying the fixture 50 to be captured (hereinafter also referred to as "fixture identification information"). Then, the image acquisition unit 110 of the information processing device 10 identifies the target fixed camera 30 from among the multiple fixed cameras 30 installed in the store based on the acquired fixture identification information. For example, the image acquisition unit 110 of the information processing device 10 can identify the fixed camera 30 capturing the fixture 50 identified by the fixture identification information by referring to information indicating the correspondence between the multiple fixtures installed in the store and the multiple fixed cameras. In this case, information indicating the correspondence between the plurality of fixtures installed in the store and the plurality of fixed cameras is stored in advance in a storage area accessible by the image acquisition unit 110 of the information processing device 10, such as the memory 1030 or the storage device 1040. Below, some specific examples of the process of acquiring fixture identification information are given.

[0059] As a first example, communication device 20 may receive input of fixture identification information indicating fixture 50 to be photographed from the user of communication device 20. For example, image generation unit 210 of communication device 20 may display a store map on the display before displaying the photographing screen, and receive input specifying location information of fixture 50 to be photographed from the user of communication device 20. Here, the "input specifying location information of fixture 50 to be photographed" is not particularly limited, but may be, for example, an input of tapping the position of fixture 50 on the map, an input of specifying identification information assigned to fixture 50 using various input devices, etc.

[0060] As a second example, the communication device 20 may generate fixture identification information indicating the fixture 50 to be photographed using the position and orientation of the communication device 20. For example, the communication device 20 estimates the position of the communication device 20 in the store using a signal from a beacon or the like. The communication device 20 also estimates the direction in which the communication device 20 is facing using a sensor device (e.g., an Inertial Measurement Unit (IMU)) built into the communication device 20. The communication device 20 then identifies the fixture 50 located in front of the communication device 20 based on the estimated position and orientation of the communication device 20, and acquires fixture identification information indicating the fixture 50. The communication device 20 then transmits the acquired fixture identification information to the information processing device 10. Note that the information processing device 10 may perform a process of identifying the target fixture 50 using the position and orientation of the communication device 20. In this case, the communication device 20 transmits the estimation results of the position of the communication device 20 and the direction in which the communication device 20 is facing to the information processing device 10 .

[0061] As a third example, each piece of fixture installed in a store may be assigned information for identifying the fixture. For example, string information for identifying the fixture or a code symbol (one-dimensional code or two-dimensional code) encoding the string information may be assigned at a predetermined position on each piece of fixture. In this case, the communication device 20 can generate a first image from which fixture identification information for the fixture 50 being photographed can be acquired by photographing the fixture 50 so that the image includes the string information itself or a code symbol encoding the string information. The image acquisition unit 110 of the information processing device 10 can then analyze the string information or code symbol included in the first image acquired from the communication device 20 to acquire fixture identification information for identifying the fixture 50 depicted in the first image. Note that the communication device 20 may analyze the string information or code symbol in the first image to acquire the fixture identification information. In this case, the communication device 20 associates the first image with the fixture identification information and transmits the image to the information processing device 10.

[0062] Then, the shelf allocation information generating unit 120 of the information processing device 10 uses the first image acquired in the process of step S206 to generate shelf allocation information for the target fixture 50 (step S210). This process is similar to the process executed in step S106 of the flowchart in FIG. 3.

[0063] If shelf labels corresponding to each product are provided on the fixture 50, in the processing of step S210, the shelf allocation information generation unit 120 may acquire product information to be included in the shelf allocation information as follows. First, the shelf allocation information generation unit 120 detects shelf labels corresponding to each product displayed on the fixture 50 from the first image, for example, using a known object recognition algorithm. The shelf allocation information generation unit 120 can acquire product information such as product identification information, product name, and product price indicated on the shelf label by analyzing the image area of ​​the shelf label corresponding to each product detected from the first image. Here, each shelf label may be assigned a one-dimensional or two-dimensional code symbol that encodes information about the product. In this case, the shelf allocation information generation unit 120 can acquire product information about the product corresponding to each shelf label by decoding the code symbol. Note that the shelf allocation information generation unit 120 may perform any correction processing on the image area corresponding to each shelf label detected from the first image, and then execute processing to decode the code symbol assigned to each shelf label. Here, "any correction process" is not particularly limited, but includes a correction process that cancels out distortion in an image using parameters corresponding to the lens characteristics of the camera module installed in the communication device 20, a camera shake correction process, and the like. By performing such correction processes, the shelf allocation information generation unit 120 can accurately acquire product information from the first image. Furthermore, by utilizing information (e.g., code symbols) indicated on the shelf tags of each product, there is no need to create a product image database for identifying products. For example, when a new product arrives, there is no need for a store clerk to take an image of the new product and register it in the database. This is expected to reduce the maintenance and management costs of the entire system.

[0064] Here, the shelf allocation information generation unit 120 may be configured to output predetermined notification information to the portable communication device 20 when product information cannot be acquired from the first image. For example, when a code symbol attached to a shelf label cannot be decoded, the shelf allocation information generation unit 120 outputs the predetermined notification information. Here, the "predetermined notification information" is not particularly limited, but includes information notifying at least one of the following: that the first image needs to be retaken and that product information for which the code symbol could not be decoded needs to be manually entered. Note that the predetermined notification information may be visual information displayed on the display 22 of the communication device 20 or audio information output from the speaker of the communication device 20. By outputting such notification information, the user of the communication device 20 can notice that there is a defect in the captured first image. The user of the communication device 20 can then take action to generate a more appropriate first image.

[0065] Furthermore, the correspondence information generating unit 130 of the information processing device 10 generates correspondence information based on the common index object present in the first image acquired in the process of step S206 and the second image acquired in the process of step S208 (step S212). This process is similar to the process executed in step S108 of the flowchart in FIG. 3.

[0066] In the processing of step S212, the correspondence information generation unit 130 can use, for example, products displayed on the fixture 50, specific parts of the fixture 50, or shelves provided on the fixture 50 as "index objects." As a first example, the correspondence information generation unit 130 first recognizes objects (products) displayed on the fixture 50 from each image by analyzing the first image and the second image. Then, the correspondence information generation unit 130 can generate correspondence information indicating the correspondence between positions in the first image and positions in the second image based on commonalities in the arrangement of the recognized objects (products). As a second example, the correspondence information generation unit 130 first recognizes specific parts of the fixture 50 (e.g., corners or pillars of the fixture 50) from each image by analyzing the first image and the second image. The correspondence information generating unit 130 can generate correspondence information indicating the correspondence between positions on the first image and positions on the second image by comparing and matching the arrangements of specific parts of the fixture 50 recognized in each image between the two images. As a third example, the correspondence information generating unit 130 first recognizes shelves provided on the fixture 50 from each image by analyzing the first image and the second image, respectively. The correspondence information generating unit 130 can then generate correspondence information indicating the correspondence between positions on the first image and positions on the second image by comparing and matching the arrangements of the shelves of the fixture 50 recognized in each image between the two images.

[0067] Then, the information storage unit 140 of the information processing device 10 uses the correspondence information generated in the process of step S212 to convert the product area information of each product included in the shelf allocation information generated in the process of step S210 into information indicating a position on the second image (step S214). Then, the information storage unit 140 of the information processing device 10 stores the shelf allocation information after converting the product area information of each product in a predetermined storage area, such as the memory 1030 or the storage device 1040 (step S216). These processes are similar to the processes executed in steps S110 and S112 of the flowchart in FIG. 3.

[0068] In this manner, shelf allocation information for a fixture is generated based on an image obtained by photographing the target fixture using a portable communication device. This eliminates the need for a store clerk to manually input information about the shelf allocation for the fixture. Here, the shelf allocation information includes product area information indicating the area in which each product is displayed, but the product area information is information in the coordinate system of the first image (a coordinate system based on the communication device 20). According to the information processing device 10 of the present disclosure, the product area information for each product included in the shelf allocation information is converted into information indicating its position in the coordinate system of the second image (a coordinate system based on the fixed camera 30) using correspondence information indicating the correspondence between the positions of the first image and the second image. The converted shelf allocation information is then stored in a predetermined memory area. In other words, the shelf allocation information based on the coordinate system of the first image is converted into shelf allocation information based on the coordinate system of the second image. This makes it possible to utilize shelf allocation information automatically generated from images for image processing using a fixed camera. That is, an information processing system including the information processing device 10 according to the present disclosure can solve the problem of reducing the workload of store staff who manage a product observation service using fixed cameras.

[0069] 9 is a flowchart illustrating a first example of the operation of the communication device 20 according to the present disclosure. The process described in this flowchart is executed, for example, at the timing of the process of step S202 in the sequence diagram of FIG.

[0070] First, when an application for the information processing system 1 is started in the communication device 20, the image generating unit 210 displays a photographing screen on the display 22 (step S302).

[0071] Furthermore, the guide information output unit 220 outputs guide information based on the image of the fixture 50 being photographed, which is generated by the fixed camera 30, onto the photographing screen (step S304).

[0072] Here, when a plurality of fixtures are installed in a store, at least one fixed camera 30 may be provided for each fixture. In this case, the guide information output unit 220 needs to identify, from among the plurality of fixed cameras 30, the fixed camera 30 that is capturing the same fixture 50 as the fixture 50 that the communication device 20 is capturing. In such a case, the guide information output unit 220 can identify the fixed camera 30 capturing the fixture 50 by using fixture identification information that indicates the fixture 50 that is the capture target, as shown in some specific examples in the description of the processing of step S208 in the sequence diagram of FIG. 8 . For example, the guide information output unit 220 may receive in advance input of information that indicates the fixture 50 that is the capture target (information equivalent to the fixture identification information) from an application running on the communication device 20. For example, the guide information output unit 220 may estimate the fixture 50 to be photographed based on the estimated position and orientation of the communication device 20, and acquire information indicating the estimated fixture 50 as fixture identification information. For example, if each fixture is assigned string information identifying it or a code symbol encoding the string information, the guide information output unit 220 may acquire the fixture identification information of the fixture 50 to be photographed by analyzing the string information or code symbol detected from the preview image displayed on the photographing screen. The guide information output unit 220 can then identify the fixed camera 30 corresponding to the fixture 50 to be photographed by referring to information indicating the correspondence between the multiple fixtures installed in the store and the multiple fixed cameras. In this case, the information indicating the correspondence between the multiple fixtures installed in the store and the multiple fixed cameras is stored in advance in a storage area accessible to the guide information output unit 220, such as the memory 2030 or the storage device 2040.

[0073] The guide information output unit 220 then outputs guide information based on the image of the fixture 50 captured by the identified fixed camera 30 onto the captured screen. As an example, the guide information output unit 220 detects the shape of the fixture 50 from the image of the fixture 50 captured by the identified fixed camera 30. The guide information output unit 220 then superimposes guide information (e.g., a predetermined mark indicating the shape of the fixture 50 in the second image) based on the shape of the fixture 50 detected from the image of the fixed camera 30 onto the captured screen. FIGS. 10 and 11 are diagrams showing an example of guide information generated and output by the guide information output unit 220. In FIGS. 10 and 11, the guide information is indicated by dotted lines. In the example shown in FIG. 10, the guide information output unit 220 outputs guide information G1 onto the captured screen, which indicates the positions of the four corners of the fixture 50 captured by the identified fixed camera 30 using predetermined marks. 11 , the guide information output unit 220 outputs, as guide information G2, a rectangular frame indicating the outline of the fixture 50 photographed by the identified fixed camera 30 on the photographing screen. As another example, the guide information output unit 220 may perform a transparency process on the image of the fixture 50 to be photographed, generated by the identified fixed camera 30, and output the transparent image on the photographing screen as guide information.

[0074] As described above, when photographing a target fixture 50 using the communication device 20, guide information based on the image of the fixture 50 generated by the fixed camera 30 is output on the photographing screen. By photographing in accordance with the guide information, the position of the fixture 50 to be photographed can be brought closer between the first image and the second image. As a result, it becomes possible to more accurately calculate the correspondence between the position on the first image and the position on the second image. Furthermore, the communication device 20, together with the above-described information processing device 10, constitutes an information processing system 1. In other words, the communication device 20 (a program that causes the communication device 20 to execute the above-described functions) according to the present disclosure contributes to solving the problem of reducing the workload of store clerks who manage product observation services using fixed cameras.

[0075] <Modification> The guide information output unit 220 may be provided in the information processing device 10. In this case, the information processing device 10 can acquire fixture identification information indicating the fixture 50 to be photographed, as shown in several specific examples in the description of the processing of step S208 in the sequence diagram of FIG. 8 . The information processing device 10 then identifies the fixed camera 30 photographing the fixture 50 identified by the fixture identification information, and generates the guide information described above based on the image of the fixture 50 generated by the fixed camera 30. The information processing device 10 then transmits the generated guide information to the communication device 20. As a result, when the communication device 20 photographs an image of the target fixture 50, guide information based on the image of the fixture 50 generated by the fixed camera 30 is output on the photographing screen displayed on the display of the communication device 20. This configuration also provides the same effects as those described above.

[0076] Second Embodiment The information processing device according to the present disclosure may further include a function related to a fixed-point observation service of products displayed in a store, as shown in Fig. 12, for example. For example, the information processing device 10 may further include the following functions (1) and (2): (1) a function of periodically acquiring images (second images) of the fixtures 50 from the fixed camera 30; and (2) a function of outputting various information related to each product displayed on the fixtures 50 to a terminal used by a store clerk (for example, a communication device 20 used by each store clerk or a management terminal installed in the back room).

[0077] <Functional Configuration Example> Fig. 12 is a diagram showing a second example of the functional configuration of an information processing device according to the present disclosure. The information processing device 10 illustrated in this figure further includes a product area identification unit 112 and a display output unit 150 in addition to the functional configuration illustrated in Fig. 1.

[0078] The product area identification unit 112 periodically acquires images (second images) of the fixtures 50 taken by the fixed cameras 30 installed in the store during store business hours. Then, the product area identification unit 112 identifies the image area of ​​each product in the image acquired from the fixed cameras 30 (second images) based on the product area information of each product included in the shelf planogram information stored in the information storage unit 140. The product area identification unit 112 can identify the image area in which each product is displayed on the second image by referencing the product area information of each product included in the shelf planogram information. Furthermore, the display output unit 150 outputs product information associated with the product area information of each product included in the shelf planogram information, in association with the image area of ​​each product identified by the product area identification unit 112.

[0079] <Example of operation of information processing device> Fig. 13 is a flowchart showing a second example of operation of the information processing device according to the present disclosure. The processing illustrated in the flowchart in this figure is executed after shelf allocation information is generated by the processing shown in the flowchart in Fig. 3, for example.

[0080] First, the image acquisition unit 110 acquires an image (second image) of the fixture 50 from each fixed camera 30 (step S402).

[0081] Next, the product area identification unit 112 retrieves the shelf allocation information corresponding to the fixture 50 shown in the acquired image from the information stored in a predetermined storage area (e.g., FIG. 4 ) (step S404). For example, if the image acquired in step S402 was generated by a fixed camera 30 with camera ID "#0001," the product area identification unit 112 can retrieve the shelf allocation information associated with that camera ID. The product area identification unit 112 then identifies the image area corresponding to each product in the image acquired in step S402 based on the product area information for each product included in the retrieved shelf allocation information (step S406). FIG. 14 is a diagram outlining the process by which the product area identification unit 112 identifies the image area corresponding to each product. As illustrated in this diagram, the product area identification unit 112 can identify the area R corresponding to each product in the second image based on the second image acquired from the fixed camera 30 and the product area information for each product included in the shelf allocation information for the fixture 50 corresponding to the fixed camera.

[0082] The display output unit 150 combines the product information for each product included in the shelf allocation information with the image area identification result for each product by the product area identification unit 112, and outputs the combined information on the display of the store clerk terminal (step S408). The store clerk terminal may be, for example, a portable communication device 20 carried by each store clerk or a management terminal installed in the backroom. FIG. 15 is a diagram outlining the process flow of the display output unit 150 outputting information to the store clerk terminal. As illustrated in this diagram, the display output unit 150 displays on the store clerk terminal a screen including the area in which each product is displayed and information about the products displayed in each area in the second image acquired from the fixed camera 30.

[0083] According to the above example, the shelf allocation information generated by the information storage unit 140 is used to add information about the areas of various products displayed on the fixtures 50 and the corresponding products to the images of the fixtures 50 captured by each fixed camera 30. Then, a screen combining the image of the fixtures 50 and the areas of various products displayed on the fixtures 50 and the corresponding product information is output to the display of the terminal for the store clerk. By checking the screen on the display of the terminal, the store clerk can easily grasp the status of each product displayed on the fixtures 50.

[0084] <Variation 1> As a variation, the information processing device 10 may have a function for monitoring the in-store inventory status of products displayed on the fixtures 50. For example, the display output unit 150 processes the image area of ​​each product to generate information indicating the in-store inventory status for each product (hereinafter, "in-store inventory information"). In this case, the display output unit 150 may output a screen including information regarding the in-store inventory of each product, as illustrated in FIG. 16 . FIG. 16 is a diagram illustrating an example of a screen output by the display output unit 150 to a terminal for a store clerk. In the example of FIG. 16 , the display output unit 150 generates and outputs information I indicating the fill rate of each product as the in-store inventory information for each product. This information allows the store clerk to more easily and accurately grasp the status of each product displayed on the fixtures 50.

[0085] The display output unit 150 may also be configured to output an alert based on the in-store inventory information for each product. For example, the display output unit 150 determines whether the in-store inventory information generated for each product satisfies a predetermined alert criterion. If there is a product whose in-store inventory information satisfies the predetermined alert criterion, the display output unit 150 outputs an alert to a store clerk. The display output unit 150 outputs an alert (e.g., a highlighted display as shown by hatching in FIG. 16 ) to, for example, a portable communication device 20 carried by the store clerk or a management terminal installed in the backroom. As an example, the display output unit 150 performs the following operation. First, the display output unit 150 selects an image area of ​​one product as a processing target. Then, the display output unit 150 further identifies free space (areas where no product is present) in the image area of ​​the selected product. Then, the display output unit 150 calculates the ratio of free space in the image area of ​​the selected product as the fill rate of the product. The display output unit 150 repeatedly performs this process for each product area. As a result, the fill rate of each product is calculated. The display output unit 150 then compares the calculated fill rate for each product with a preset threshold. This threshold is pre-stored in a storage area accessible to the display output unit 150, such as the memory 1030 or the storage device 1040. The display output unit 150 outputs an alert if a product has a calculated fill rate that reaches a predetermined threshold. For example, assume that the alert criterion is "a product's fill rate is 50% or less." In this case, as illustrated in FIG. 16 , an alert is output for the product "Drink β," whose fill rate is below 50%. Furthermore, as illustrated in FIG. 17 , a predetermined message may be output to the communication device 20 carried by the store clerk using video and / or audio. FIG. 17 is a diagram illustrating an example of an alert output from the display output unit 150 to the communication device 20 carried by the store clerk. As illustrated in FIG. 17 , the display output unit 150 may output information about the product that has reached the alert criterion, as well as information indicating the fixture 50 on which the product is displayed.

[0086] According to the configuration of this modified example, an alert is automatically output to the store staff regarding products that need to be replenished, etc. This makes it possible to maintain the appropriate inventory status of each product in the store.

[0087] <Variation 2> The shelf allocation information generation unit 120 may also extract images of each product detected from the first image generated by the portable communication device 20 and include the extracted product in the shelf allocation information. In this case, the display output unit 150 may further output images of each product extracted from the first image, as shown in FIG. 18 , for example. FIG. 18 is a diagram illustrating another example of a screen output by the display output unit 150 to a terminal for a store clerk. For example, when identifying an image area of ​​each product in the second image using product area information included in the shelf allocation information, the display output unit 150 further reads out product images corresponding to each product area information. Then, based on the association, the display output unit 150 associates the image area of ​​each product identified in the second image with the product image IMG, as shown in FIG. 18 , for example. Here, the resolution of the image generated by the communication device 20 carried by the store clerk is generally higher than the resolution of the image generated by the fixed camera 30. That is, the screen as shown in FIG. 18 allows the store clerk to more easily and accurately grasp the products displayed on the fixtures 50.

[0088] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0089] In addition, although the flowcharts used in the above description show a sequence of steps (processes), the order of steps executed in each embodiment is not limited to the sequence shown in the flowcharts. In each embodiment, the order of steps shown in the diagrams can be changed as long as it does not cause any problems in terms of the content.

[0090] Some or all of the above embodiments can be described as in the following supplementary notes, but are not limited to the following: 1. An information processing device comprising: image acquisition means for acquiring a first image of a fixture taken by a portable communication device and a second image of the fixture taken by a fixed camera; shelf allocation information generation means for processing the first image to generate, for each product present in the first image, shelf allocation information that associates product area information indicating the position of the area where the product is displayed with product information related to the product; correspondence information generation means for generating correspondence information that indicates the correspondence between positions on the first image and positions on the second image based on feature information related to a common index object that exists in the first image and the second image, respectively; and information storage means for converting the product area information of each product included in the shelf allocation information into a position on the second image using the correspondence information, and storing the shelf allocation information after the product area information of each product has been converted in a predetermined storage area. The information processing device according to 1., further comprising: a product area specifying means for specifying an image area of ​​each product in the second image based on the product area information of each product included in the shelf plan information; and a display output means for outputting product information associated with the product area information of each product included in the shelf plan information, in association with the specified image area of ​​each product. 3. The information processing device according to 1. or 2., further comprising: a guide information output means for outputting guide information generated based on the second image on a capture screen displayed on a display of the portable communication device when the portable communication device captures the first image. 4. The information processing device according to any one of 1. to 3., wherein the index object includes at least one of products displayed on fixtures, specific parts of the fixtures, and shelves provided on the fixtures. 5. The shelf plan information generating means detects shelf tags corresponding to each of the products displayed on the fixtures from the first image, and obtains the product information corresponding to each product by decoding code symbols attached to the shelf tags detected for each product. 4. The information processing device according to any one of the above.6. The information processing device described in any one of 2. or 3. to 5. that quotes 2., wherein the display output means processes the image area of ​​each product to generate in-store inventory information indicating the in-store inventory status for each product, and outputs an alert to a store clerk when the in-store inventory information indicates that there is a product that meets a predetermined alert standard. 7. The shelf allocation information generation means generates the shelf allocation information by further associating images of each product detected from the first image, and the display output means outputs product images associated with the product area information of each product included in the shelf allocation information, by associating them with the identified image area of ​​each product. 8. The information processing device described in any one of 5. to 7., wherein the shelf allocation information generation means corrects the image area corresponding to the shelf tag detected from the first image, and then executes processing to decode the code symbol. 9. The information processing device according to 6. or 7. or 8., which quotes 6., wherein the display output means calculates a fill rate for each product using the ratio between the image area of ​​each product and the free space in the image area of ​​each product, and outputs the alert if there is a product whose fill rate has reached a predetermined threshold. 10. The information processing device according to 9., which quotes 8. or 8., wherein the shelf allocation information generation means outputs predetermined notification information to the portable communication device if the code symbol cannot be decoded. 11. The information processing device according to 10., wherein the shelf allocation information generation means outputs information as the notification information notifying at least one of the fact that the first image needs to be re-taken and that information on products whose code symbols could not be decoded needs to be manually entered.12. An information processing system comprising: a fixed camera installed in a store; a portable communication device used in the store; an image acquisition means for acquiring a first image of a fixture taken by the portable communication device and a second image of the fixture taken by the fixed camera; a shelf allocation information generation means for processing the first image to generate, for each product present in the first image, shelf allocation information that associates product area information indicating the position of the area where the product is displayed with product information related to the product; a correspondence information generation means for generating correspondence information that indicates the correspondence between positions on the first image and positions on the second image based on feature information related to a common index object that exists in the first image and the second image; and an information storage means for converting the product area information of each product included in the shelf allocation information into a position on the second image using the correspondence information, and storing the shelf allocation information after the product area information of each product has been converted in a predetermined storage area. a first image of a fixture taken by a portable communication device and a second image of the fixture taken by a fixed camera; processing the first image to generate, for each product present in the first image, shelf allocation information that associates product area information indicating the position of the area in which the product is displayed with product information related to the product; generating correspondence information that indicates the correspondence between positions on the first image and positions on the second image based on feature information related to a common index object that is present in the first image and the second image; converting the product area information of each product included in the shelf allocation information to a position on the second image using the correspondence information; and storing the shelf allocation information after the product area information of each product has been converted in a specified memory area.14. A non-transitory computer-readable recording medium having recorded thereon a program for causing at least one computer to perform operations including: acquiring a first image of a fixture taken by a portable communication device and a second image of the fixture taken by a fixed camera; processing the first image to generate, for each product present in the first image, shelf allocation information that associates product area information indicating the position of the area in which the product is displayed with product information related to the product; generating correspondence information that indicates the correspondence between positions on the first image and positions on the second image based on feature information related to a common index object that exists in the first image and the second image; converting the product area information of each product included in the shelf allocation information into a position on the second image using the correspondence information; and storing the shelf allocation information after the product area information of each product has been converted in a predetermined storage area. A non-transitory computer-readable recording medium that records a program for causing a portable communication device to perform operations including: acquiring guide information based on an image of a fixture to be photographed, generated by a fixed camera; and outputting the guide information on a photographing screen that is displayed on a display of the communication device when photographing.

[0091] Furthermore, some or all of the configurations described in Supplementary Notes 2 to 11, which are subordinate to Supplementary Note 1 (information processing device), may also be subordinate to Supplementary Note 12 (information processing system), Supplementary Note 13 (store business support method), and Supplementary Note 14 (non-transitory computer-readable recording medium for recording a program related to the functions of an information processing device) in the same subordinate relationship as Supplementary Note 2 to 11. Furthermore, some or all of the configurations described in Supplementary Notes 2 to 11 may also be subordinate to Supplementary Note 15 (non-transitory computer-readable recording medium for recording a program related to the functions of a communication device), to the extent that they do not deviate from the respective embodiments described above. Furthermore, not limited to Supplementary Note 1, Supplementary Note 12, Supplementary Note 13, Supplementary Note 14, and Supplementary Note 15, to the extent that they do not deviate from the respective embodiments described above, some or all of the configurations described as Supplements may also be subordinate to various hardware, software, various recording means for recording software, or systems.

[0092] 1 Information processing system 10 Information processing device 110 Image acquisition unit 112 Product area identification unit 120 Shelf allocation information generation unit 130 Correspondence information generation unit 140 Information storage unit 150 Display output unit 1000 Computer 1010 Bus 1020 Processor 1030 Memory 1040 Storage device 1050 Input / output interface 1060 Network interface 20 Communication device 22 Display 210 Image generation unit 220 Guide information output unit 2000 Computer 2010 Bus 2020 Processor 2030 Memory 2040 Storage device 2050 Input / output interface 2060 Network interface 30 Fixed camera 40 Network 50 Fixtures

Claims

1. Image acquisition means for acquiring a first image of the fixture taken by a portable communication device and a second image of the fixture taken by a fixed-point camera, A shelf layout information generation means that processes the first image and generates shelf layout information for each product present in the first image, which associates product area information indicating the location of the area where the product is displayed with product information related to the product. Correspondence relationship information generation means generates correspondence relationship information indicating the correspondence between a position in the first image and a position in the second image, based on feature information relating to a common indicator object present in the first image and the second image, respectively. Information storage means that converts the product area information of each product included in the shelf layout information into a position on the second image using the correspondence relationship information, and stores the shelf layout information after the product area information of each product has been converted in a predetermined storage area, An information processing device equipped with the following features.

2. A product area identification means for identifying the image area of ​​each product in the second image based on the product area information of each product included in the shelf arrangement information, The system further includes a display output means that outputs product information associated with the product area information of each product included in the shelf layout information, associated with the image area of ​​each identified product. The information processing apparatus according to claim 1.

3. The portable communication device further includes a guide information output means that causes the portable communication device to output guide information generated based on the second image on the shooting screen displayed on the display of the portable communication device when the portable communication device takes a first image. The information processing apparatus according to claim 1.

4. The aforementioned indicator object includes at least one of the following: goods displayed in the fixture, a specific part of the fixture, and a shelf provided in the fixture. The information processing apparatus according to claim 1.

5. The shelf layout information generation means is The shelf labels corresponding to each product displayed on the fixture are detected from the first image. By decrypting the code symbol attached to the shelf label detected for each product, the product information corresponding to each product is obtained. The information processing apparatus according to claim 1.

6. The aforementioned display output means is By processing the image area of ​​each of the aforementioned products, store inventory information indicating the store inventory status for each product is generated. When the aforementioned in-store inventory information contains products that meet predetermined alert criteria, an alert is issued to store employees. The information processing apparatus according to claim 2.

7. The shelf allocation information generation means generates the shelf allocation information by further associating the images of each product detected from the first image, The display output means outputs the image of each product associated with the product area information of each product included in the shelf layout information, associating it with the image area of ​​each identified product. The information processing apparatus according to claim 2.

8. The shelf layout information generation means is After correcting the image region corresponding to the shelf label detected from the first image, the process of decoding the code symbol is performed. The information processing apparatus according to claim 5.

9. The aforementioned display output means is The filling rate for each product is calculated using the ratio of the image area of ​​each product to the empty area within the image area of ​​each product. If there are products whose filling rate has reached a predetermined threshold, the alert will be output. The information processing apparatus according to claim 6.

10. The shelf layout information generation means is If the aforementioned code symbol cannot be decoded, predetermined notification information is output to the portable communication device. The information processing apparatus according to claim 8.

11. The shelf layout information generation means is The system outputs information as notification information indicating that at least one of the following is necessary: ​​that the first image needs to be retaken, and that information about the product whose code symbol could not be decoded needs to be manually entered. The information processing apparatus according to claim 10.

12. Fixed cameras installed inside the store, A portable communication device used within the aforementioned store, Image acquisition means for acquiring a first image of the fixture taken by the portable communication device and a second image of the fixture taken by the fixed-point camera, A shelf layout information generation means that processes the first image and generates shelf layout information for each product present in the first image, which associates product area information indicating the location of the area where the product is displayed with product information related to the product. Correspondence relationship information generation means generates correspondence relationship information indicating the correspondence between a position in the first image and a position in the second image, based on feature information relating to a common indicator object present in the first image and the second image, respectively. Information storage means that converts the product area information of each product included in the shelf layout information into a position on the second image using the correspondence relationship information, and stores the shelf layout information after the product area information of each product has been converted in a predetermined storage area, An information processing system equipped with the following features.

13. At least one computer, A first image of the fixture taken by a portable communication device and a second image of the fixture taken by a fixed-point camera are acquired. By processing the first image, shelf allocation information is generated for each product present in the first image, which associates product area information indicating the location of the area where the product is displayed with product information related to that product. Based on feature information relating to a common indicator object present in the first image and the second image, correspondence information is generated that shows the correspondence between the position in the first image and the position in the second image. The product area information for each product included in the shelf layout information is converted to a position on the second image using the correspondence relationship information, and the shelf layout information after the conversion of the product area information for each product is stored in a predetermined storage area. A method of supporting store operations, including the following.

14. On at least one computer, A first image of the fixture taken by a portable communication device and a second image of the fixture taken by a fixed-point camera are acquired. By processing the first image, shelf allocation information is generated for each product present in the first image, which associates product area information indicating the location of the area where the product is displayed with product information related to that product. Based on feature information relating to a common indicator object present in the first image and the second image, correspondence information is generated that shows the correspondence between the position in the first image and the position in the second image. The product area information for each product included in the shelf layout information is converted to a position on the second image using the correspondence relationship information, and the shelf layout information after the conversion of the product area information for each product is stored in a predetermined storage area. A program that performs an action that includes the following.

15. Portable communication devices, Guide information is obtained based on images of the target fixtures generated by a fixed-point camera. When taking a picture, the guide information is output on the shooting screen displayed on the communication device's display. A program that performs an action that includes the following.