Display control device, control method, and program

Abstract

This display control device (2000) acquires a target image (10). The target image (10) includes a display device (30) disposed at a display site (40) and merchandise (20) on display at the display site (40). The display control device (2000) recognizes the merchandise (20) included in the target image (10), and acquires related information (54) about that merchandise (20). On the basis of the relative positional relationship between the display device (30) and the merchandise (20) within the target image (10), the display control device (2000) determines a display position at which a display (32) showing the related information (54) of the merchandise (20) is to be indicated on the display device (30). The display control device (2000) causes the display (32) of the merchandise (20) to be displayed at the display position thus determined on the display device (30).

Description

[Technical Field]

[0001] The present invention relates to control of a display device that displays information about a product. [Background technology]

[0002] Shelf labels indicating product prices and other information are provided on product display locations such as display shelves. Technologies are being developed to use electronic media (such as electronic shelf labels) instead of paper media for such shelf labels.

[0003] Patent Document 1 discloses an electronic shelf label system that displays information about products displayed on a display shelf on an electronic shelf label with the longitudinal direction of the display screen aligned with the horizontal direction of the display shelf. In Patent Document 1, a store clerk or other person manually inputs the product layout on the display shelf, and a display simulation is performed based on the input. Then, based on the results of the simulation, the layout of information display is determined for each of multiple types of products displayed on the display shelf. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-193218 [Patent Document 2] International Publication No. 2016 / 052382 Summary of the Invention [Problem to be solved by the invention]

[0005] The electronic shelf label system of Patent Document 1 requires store clerks and the like to manually input the layout of products, which requires a lot of work from store clerks and the like.

[0006] The present invention has been made in view of the above-mentioned problems, and one of its objects is to provide a technique that reduces the effort required to display information about a product on a display device. [Means for solving the problem]

[0007] The display control device of the present invention has 1) a first acquisition unit that acquires a target image. The target image includes a display device provided in a product display area and the product displayed in the display area. The display control device further has 2) a second acquisition unit that recognizes the product included in the target image and acquires related information related to the product, 3) a display position determination unit that determines a display position on the display device to display a related display showing related information about the product based on the relative positional relationship between the product and the display device in the target image, and 4) a display control unit that displays the related display of the product at a display position on the display device.

[0008] A control method of the present invention is executed by a computer. The control method includes a first acquisition step of acquiring a target image. The target image includes a display device provided in a product display area and the product displayed in the display area. The control method further includes 2) a second acquisition step of recognizing the product included in the target image and acquiring related information related to the product, 3) a display position determination step of determining a display position on the display device for displaying a related display showing related information about the product based on the relative positional relationship between the product and the display device in the target image, and 4) a display control step of displaying the related display of the product at a display position on the display device.

[0009] The program of the present invention causes a computer to execute the control method of the present invention. [Effects of the Invention]

[0010] According to the present invention, a technique is provided that reduces the effort required to display information about a product on a display device. [Brief explanation of the drawings]

[0011] [Figure 1]2 is a diagram conceptually illustrating an example of the operation of the display control device according to the first embodiment. FIG. [Figure 2] FIG. 2 is a block diagram illustrating a functional configuration of a display control device. [Figure 3] FIG. 1 is a diagram illustrating a computer for realizing a display control device. [Figure 4] 4 is a flowchart illustrating the flow of processing executed by the display control device of the first embodiment. [Figure 5] FIG. 1 is a diagram illustrating an example of a usage environment of a display control device. [Figure 6] FIG. 1 is a diagram illustrating an example of how a bar-type display is used. [Figure 7] FIG. 10 is a first diagram illustrating an example of the relative positional relationship between a display and a product on a target image. [Figure 8] FIG. 2 is a second diagram illustrating an example of the relative positional relationship between the display and the product on the target image. [Figure 9] FIG. 10 is a diagram illustrating a case where the number of faces of a product is two or more. [Figure 10] FIG. 10 is a diagram illustrating a case where a product display is displayed on a display device provided on a shelf on which the product is displayed. [Figure 11] FIG. 10 is a diagram illustrating a case in which a plurality of display devices are provided on one shelf board. [Figure 12] 10 is a diagram illustrating a case in which a plurality of display devices each long in the vertical direction are provided at different positions in the horizontal direction. [Figure 13] FIG. 10 is a diagram illustrating a case where multiple display devices are provided on one side wall. [Figure 14] 10A and 10B are diagrams illustrating a case in which the size of a display device is determined depending on the display state of products. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all drawings, like components are denoted by like reference numerals, and their description will be omitted where appropriate. Furthermore, unless otherwise specified, each block in each block diagram represents a functional configuration, not a hardware configuration. In the following description, unless otherwise specified, various predetermined values ​​(such as threshold values) are stored in advance in a storage device accessible from the functional configuration unit that uses the values.

[0013] [Embodiment 1] <Summary> Fig. 1 is a diagram conceptually illustrating an example of the operation of a display control device 2000 according to the first embodiment. Here, the operation of the display control device 2000 described using Fig. 1 is an example for facilitating understanding of the display control device 2000, and does not limit the operation of the display control device 2000. Details and variations of the operation of the display control device 2000 will be described later.

[0014] The display control device 2000 displays information about the products 20 displayed in the display area 40 on the display device 30 provided in the display area 40. The information about the products 20 is, for example, the name and price of the products 20, or an advertisement for the products 20. The operation of the display control device 2000 will be described in more detail below.

[0015] The display control device 2000 acquires a target image 10. The target image 10 is generated by a camera capturing an image of a display location 40. The target image 10 includes the products 20 displayed in the display location 40 and the display device 30 provided in the display location 40.

[0016] The display control device 2000 recognizes each product 20 included in the target image 10 by performing image analysis on the target image 10. Furthermore, the display control device 2000 acquires related information 54 for each product 20 from the related information storage device 50. The related information storage device 50 stores identification information 52 for the product 20 and the related information 54 in association with each other. The identification information 52 is any information that can identify the product 20. For example, the identification information 52 can be a Japan Article Number (JAN) code. The related information 54 indicates information about the product 20 that should be displayed on the display device 30.

[0017] The display control device 2000 performs image analysis on the target image 10 to determine the position on the display device 30 where the related information 54 of the product 20 should be displayed. The display position where the related information 54 of the product 20 should be displayed is determined based on the relative positional relationship between the product 20 and the display device 30 in the target image 10.

[0018] The display control device 2000 uses the related information 54 of the product 20 to generate a display 32 that represents the content of the product 20. For example, the display 32 is an image that represents the content of the related information 54 of the product 20. Then, the display control device 2000 displays the display 32 of the product 20 at a display position on the display device 30 that has been determined for that product 20.

[0019] 1, for example, a product 20-1 and a product 20-2 are included in the target image 10. The display control device 2000 recognizes the product 20-1 and the product 20-2 by performing image analysis on the target image 10. The display control device 2000 acquires related information 54 for each of the product 20-1 and the product 20-2.

[0020] The display control device 2000 determines the display position at which to display the related information 54 for the product 20-1, based on the positional relationship between the product 20-1 and the display device 30 in the target image 10. Similarly, the display control device 2000 determines the display position at which to display the related information 54 for the product 20-2, based on the positional relationship between the product 20-2 and the display device 30 in the target image 10.

[0021] The display control device 2000 generates a display 32-1 that shows the related information 54 of the product 20-1 and displays the display 32-1 at the display position determined for the product 20-1. Similarly, the display control device 2000 generates a display 32-2 that shows the related information 54 of the product 20-2 and displays the display 32-2 at the display position determined for the product 20-2.

[0022] <Typical effects> According to the display control device 2000 of this embodiment, image analysis is performed on the target image 10 including the product 20 and the display device 30, and as a result, information about the product 20 (related information 54) is acquired, the position on the display device 30 where the display 32 representing the related information 54 should be displayed is determined, and the display 32 is displayed at that position. Therefore, the user of the display control device 2000 (such as a store clerk) does not need to manually input the layout of the product 20 in the display location 40. Therefore, compared to the invention of Patent Document 1, there is an advantage in that less effort is required to display information about the product on the display.

[0023] The display control device 2000 will be described in more detail below.

[0024] <Example of functional configuration> 2 is a block diagram illustrating an example of the functional configuration of the display control device 2000. The display control device 2000 includes a first acquisition unit 2020, a second acquisition unit 2040, a display position determination unit 2060, and a display control unit 2080. The first acquisition unit 2020 acquires a target image 10. The second acquisition unit 2040 performs image processing on the target image 10 to recognize a product 20 included in the target image 10. The second acquisition unit 2040 then acquires related information 54 for the recognized product 20. The display position determination unit 2060 determines a display position on the display device 30 where the related information 54 for the product 20 should be displayed, based on the relative positional relationship between the product 20 and the display device 30 in the target image 10. The display control unit 2080 displays a display 32 representing the related information 54 for the product 20 at the determined display position on the display device 30.

[0025] <Example of Hardware Configuration of Display Control Device 2000> Each functional component of the display control device 2000 may be realized by hardware that realizes the functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a case where each functional component of the display control device 2000 is realized by a combination of hardware and software will be further described.

[0026] 3 is a diagram illustrating an example of a computer 1000 for realizing the display control device 2000. The computer 1000 is any computer. For example, the computer 1000 is a stationary computer such as a PC (Personal Computer) or a server machine. Alternatively, the computer 1000 may be a portable computer such as a smartphone or a tablet terminal.

[0027] The computer 1000 may be a dedicated computer designed to implement the display control device 2000, or may be a general-purpose computer. In the latter case, for example, by installing a predetermined application on the computer 1000, the computer 1000 implements each function of the display control device 2000. The application is configured by a program for implementing the functional components of the display control device 2000.

[0028] The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. However, the method for connecting the processor 1040 and the like to each other is not limited to bus connection.

[0029] The processor 1040 is one of various processors such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 1060 is a main storage device realized using a random access memory (RAM) or the like. The storage device 1080 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, or a read only memory (ROM) or the like.

[0030] The input / output interface 1100 is an interface for connecting the computer 1000 to an input / output device. For example, the input / output interface 1100 is connected to an input device such as a keyboard and an output device such as a display device.

[0031] The network interface 1120 is an interface for connecting the computer 1000 to a communication network. This communication network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network).

[0032] The storage device 1080 stores a program (a program that realizes the above-mentioned application) that realizes each functional component of the display control device 2000. The processor 1040 reads this program into the memory 1060 and executes it, thereby realizing each functional component of the display control device 2000.

[0033] <Processing flow> FIG. 4 is a flowchart illustrating the flow of processing executed by the display control device 2000 of the first embodiment. The first acquisition unit 2020 acquires the target image 10 (S102). The second acquisition unit 2040 recognizes each product 20 included in the target image 10 (S104). Steps S106 to S118 represent loop processing A, which is executed for each recognized product 20. In step S106, the second acquisition unit 2040 determines whether loop processing A has already been executed for all recognized products 20. If loop processing A has been executed for all products 20, the processing of FIG. 4 ends. If there are products 20 that have not yet been subjected to loop processing A, the display control device 2000 selects one of them. The selected product 20 is referred to as product i. The processing of FIG. 4 then proceeds to step S108.

[0034] The display position determination unit 2060 determines a display position on the display device 30 at which to display the display 32 of the product i, based on the relative positional relationship between the product i and the display device 30 in the target image 10 (S108). The second acquisition unit 2040 acquires the related information 54 of the product i (S110). The display position determination unit 2060 generates a display 32 that shows the content of the related information 54 of the product i (S112). The display position determination unit 2060 displays the display 32 of the product i at the determined display position (S114). Since S116 is the end of loop processing A, the processing in FIG. 4 proceeds to S106.

[0035] The flow of the process executed by the display control device 2000 is not limited to that shown in Fig. 4. For example, the process (S116) of displaying the related information 54 of each product 20 on the display device 30 may be performed all at once after the display positions of all the products 20 have been determined and the displays 32 have been generated.

[0036] <Example of usage environment> To make it easier to understand the display control device 2000, a more specific example will be given of the usage environment of the display control device 2000. The usage environment of the display control device 2000 described here is merely an example, and the usage environment of the display control device 2000 is not limited to the example shown here.

[0037] 5 is a diagram illustrating an example of a usage environment of the display control device 2000. In FIG. 5, the display control device 2000 is realized by a server machine 70. Furthermore, the related information storage device 50 indicates image features 56 in addition to identification information 52 and related information 54. The image features 56 are data representing the feature amounts on the image of the product 20, and are used to recognize the product 20 included in the target image 10. However, the image features 56 may be stored in a storage device separate from the related information storage device 50.

[0038] The camera 80 is a camera installed in a store and captures images of the display area 40. For example, the camera 80 periodically captures images of the display area 40 and transmits the generated target image 10 to the server machine 70. The camera 80 may be a still camera that generates still images, or a video camera that generates moving images. In the latter case, the target image 10 is one of the video frames that make up the moving image.

[0039] The server machine 70 calculates image features for each product 20 included in the received target image 10 and recognizes each product 20 by matching the image features with image features 56 stored in the related information storage device 50. Furthermore, the server machine 70 obtains related information 54 for each product 20 from the related information storage device 50.

[0040] The server machine 70 determines, for each product 20, the display position on the display device 30 where the related information 54 for that product 20 should be displayed. Then, the server machine 70 displays the display 32 generated for each product 20 at each of the determined display positions.

[0041] In this way, by receiving target image 10 from camera 80 installed in the store and setting the display of display device 30 according to the content of target image 10, the display of display device 30 is changed appropriately in response to a store clerk displaying products 20 in display area 40 or a store clerk or customer moving displayed products 20. Therefore, it is possible to easily maintain a state in which appropriate information is displayed in an appropriate position on display device 30.

[0042] As mentioned above, the usage environment described with reference to Fig. 5 is merely an example, and the display control device 2000 can be used in various other environments. For example, the camera that generates the target image 10 does not need to be fixedly installed in the store, but may be movable, such as a camera installed in a mobile terminal. In this case, for example, a store clerk captures an image of the display location 40 with a camera installed in the mobile terminal, and the generated target image 10 is transmitted from the mobile terminal to the server machine 70.

[0043] Furthermore, the display control device 2000 does not necessarily have to be realized by a server machine, but may be realized, for example, as a mobile terminal equipped with the above-mentioned camera. In this case, the display of the display device 30 is controlled by the mobile terminal that generates the target image 10.

[0044] <Regarding the display device 30> The display device 30 is any display device that can be installed in the display location 40. For example, the display device 30 has a shape (e.g., a rectangular shape) in which the length in the vertical direction or the width direction is longer than the other. A display device having such a shape is also called a bar-type display.

[0045] 6A and 6B are diagrams illustrating examples of how a bar-type display is used. In Fig. 6A, a display device 30 that is long in the horizontal direction is used. The display device 30 displays a display 32 for each of a plurality of products 20 that are arranged horizontally in a display area 40.

[0046] 6(b), a display device 30 that is long in the vertical direction is used. The display device 30 displays a display 32 for each of a plurality of products 20 that are arranged vertically in a display area 40.

[0047] <Acquisition of target image 10: S102> The first acquisition unit 2020 acquires the target image 10 (S102). The first acquisition unit 2020 may acquire the target image 10 by any method. For example, the first acquisition unit 2020 acquires the target image 10 by receiving the target image 10 transmitted by the camera 80 that generated the target image 10. Alternatively, for example, the first acquisition unit 2020 may acquire the target image 10 by accessing a storage device in which the target image 10 is stored. This storage device may be provided either inside or outside the camera 80.

[0048] When accessing the storage device to acquire the target image 10, the first acquisition unit 2020 determines, by any method, that a new target image 10 that has not yet been acquired has been stored in the storage device, and acquires the target image 10. For example, the first acquisition unit 2020 periodically accesses the storage device to determine the existence of the new target image 10, and acquires the target image 10. Alternatively, for example, the first acquisition unit 2020 may determine the existence of the new target image 10 by receiving a notification indicating that the new target image 10 has been stored in the storage device. For example, this notification is transmitted from the camera 80 that has stored the target image 10 in the storage device.

[0049] <Product 20 Recognition: S104> The second acquisition unit 2040 recognizes each product 20 included in the target image 10 (S104). Recognizing the product 20 here means detecting an image area representing the product from within the target image 10 and identifying the product's identification information (such as a JAN code). Note that existing technology can be used to recognize the products included in the image.

[0050] <Determining the display position of display 32: S108> The display position determination unit 2060 determines the display position at which the display 32 of each product 20 is displayed on the display device 30, based on the relative positional relationship between each product 20 in the target image 10 and the display device 30 (S108). Here, the display position of the display 32 on the display device 30 is expressed by the coordinates of a reference position (e.g., a center position) of the display 32 in the coordinate system of the display device 30 (hereinafter referred to as the display coordinate system). Therefore, the display position determination unit 2060 determines the coordinates of the reference position of the display 32 in the display coordinate system. For example, the display coordinate system is a coordinate system in which the upper left corner of the display device 30 in a planar view is the origin, the rightward direction is the x direction, and the downward direction is the y direction. Furthermore, the lengths of one pixel in the x direction and the y direction are the unit lengths in the x direction and the y direction, respectively.

[0051] As described above, for example, a display device 30 having a shape in which the horizontal length is longer than the vertical length (hereinafter referred to as a horizontally long display device 30) or a display device 30 having a shape in which the vertical length is longer than the horizontal length (hereinafter referred to as a vertically long display device 30) can be used as the display device 30. Below, a method for determining the display position of the display 32 for each of these two cases will be described.

[0052] <> The display position determination unit 2060 determines the x coordinate of the reference position of the display 32 in the display coordinate system based on the relative positional relationship between the product 20 in the target image 10 and the display device 30. In this case, the y coordinate of the reference position of the display 32 can be determined arbitrarily (for example, in the vertical direction, the display 32 is displayed at the center of the display device 30).

[0053] For example, in the real world, the reference position of the display 32 is made to coincide in the horizontal direction with the reference position of the product 20. To achieve this, the display position determination unit 2060 calculates the x coordinate in the display coordinate system for the position on the display device 30 that coincides in the horizontal direction with the reference position of the product 20, based on the positional relationship between the product 20 on the target image 10 and the display device 30. The calculated x coordinate is then set as the x coordinate of the reference position of the display 32.

[0054] 7 is a first diagram illustrating the relative positional relationship between the display 32 and the product 20 on the target image 10. Position P is the reference position of the display 32, and position Q is the reference position of the product 20. The reference position P of the display 32 coincides with the reference position Q of the product 20 in the horizontal direction. In the target image 10, the horizontal distance from the left edge of the display device 30 to the reference position Q of the product 20 is d1. Furthermore, in the target image 10, the width of the display device 30 is W.

[0055] Here, in the display coordinate system, the width of the display device 30 is assumed to be A (i.e., the number of pixels in the horizontal direction is A). In this case, the position on the display device 30 that coincides in the horizontal direction with the reference position Q of the product 20 has an x ​​coordinate in the display coordinate system of (d1 / W)*A. Therefore, the display position determination unit 2060 sets the x coordinate of the reference position P of the display 32 to (d1 / W)*A. By doing so, the horizontal direction of the display 32 in the real world is Reference position P and horizontally for item 20 Reference position Q will match.

[0056] Note that the reference position of the display 32 and the reference position of the product 20 need not necessarily coincide in the horizontal direction as long as the positional relationship between them is determined in advance. For example, a horizontal offset between the reference position of the display 32 and the reference position of the product 20 is determined in advance. The display position determination unit 2060 determines the x coordinate of the reference position of the display 32 so that the reference position of the display 32 is shifted from the reference position of the product 20 by that offset.

[0057] <> The display position determination unit 2060 determines the y coordinate of the reference position of the display 32 in the display coordinate system based on the relative positional relationship between the product 20 and the display device 30. In this case, the x coordinate of the reference position of the display 32 can be determined arbitrarily (for example, in the horizontal direction, the display 32 can be displayed at the center of the display device 30).

[0058] For example, in the real world, the vertical reference position of the display 32 is made to coincide with the vertical reference position of the product 20. To achieve this, the display position determination unit 2060 calculates the y coordinate in the display coordinate system for the position on the display device 30 that coincides with the vertical reference position of the product 20, based on the positional relationship between the product 20 on the target image 10 and the display device 30. The calculated y coordinate is then set as the y coordinate of the reference position of the display 32.

[0059] FIG. 8 is a second diagram illustrating the relative positional relationship between the display 32 and the product 20 on the target image 10. Position P is the reference position of the display 32, and position Q is the reference position of the product 20. The reference position P of the display 32 coincides with the reference position Q of the product 20 in the vertical direction. In the target image 10, Reference position Q The vertical distance to is d2. Furthermore, in the target image 10, the height of the display device 30 is H.

[0060] Here, in the display coordinate system, the height of the display device 30 is B (i.e., the number of pixels in the vertical direction is B). In this case, the vertical height of the product 20 is Reference position Q The position on the display device 30 that matches this has a y coordinate of (d2 / H)*B in the display coordinate system. Therefore, the display position determination unit 2060 sets the y coordinate of the display position P of the display 32 to (d / W)*B. By doing so, the vertical direction of the display 32 in the real world Reference position P and vertical direction of product 20 Reference position Q will match.

[0061] Note that the reference position of the display 32 and the reference position of the product 20 need not necessarily coincide in the vertical direction as long as the positional relationship between them is determined in advance. For example, a vertical offset is determined in advance between the reference position of the display 32 and the reference position of the product 20. The display position determination unit 2060 determines the y coordinate of the reference position of the display 32 so that the reference position of the display 32 is shifted from the reference position of the product 20 by that offset.

[0062] <When multiple items of the same type are displayed> The same type of product 20 may be displayed so that multiple products are visible from the front. This can also be expressed as "a case where the number of faces of the product 20 is two or more." Figure 9 is a diagram illustrating a case where the number of faces of the product 20 is two or more. In Figure 9, the number of faces of product A and product B is three, respectively.

[0063] In such a case, for example, the display position determination unit 2060 displays one display 32 for each of the same type of product 20. For example, in Fig. 9(a), a display 32 is displayed below each of the three products A that are visible when viewed from the front. The same applies to product B.

[0064] Alternatively, for example, the display position determination unit 2060 may display only one display 32 for the same type of products 20. For example, in Fig. 9(b), one display 32 is displayed in the center of three products A that are visible when viewed from the front. The same applies to product B.

[0065] Here, when one display 32 is displayed for multiple products 20 of the same type in this way, for example, the display position determination unit 2060 treats these multiple products 20 collectively as a single product 20. In this case, for example, the center position, left edge position, or right edge position of the area occupied by the multiple products 20 of the same type is used as the reference position of the product 20. Alternatively, for example, the center position of any one of the multiple products 20 of the same type may be treated as the reference position of the product 20.

[0066] <Case where multiple display devices 30 exist> There are cases where the target image 10 includes multiple display devices 30. For example, there is a case where the display location 40 is a display shelf with multiple shelves, and a display device 30 is installed on each shelf.

[0067] In this way, when there are multiple display devices 30 in the target image 10, the display position determination unit 2060 determines, for each product 20, the display device 30 on which to display the display 32 of that product 20. In other words, the display position determination unit 2060 associates the product 20 with the display device 30. Then, for each product 20, the display position determination unit 2060 determines the display position of the display 32 of that product 20 based on the relative positional relationship between that product 20 and the display device 30 associated with that product 20. The display position of the display 32 of the product 20 is the position on the display device 30 associated with that product 20.

[0068] A method for associating the product 20 with the display device 30 (a method for determining the display device 30 that will display the display 32 of the product 20) will be specifically described below.

[0069] <> <<<Vertical position is different>>> This is the case where the target image 10 includes multiple display devices 30 that are long in the horizontal direction, but whose vertical positions are different. Specifically, this is the case where the display location 40 is a display shelf with multiple shelves, and display devices 30 are provided on two or more shelves. In this case, the display position determination unit 2060 associates the product 20 with the display device 30 based on the relative positional relationship between the product 20 and the display device 30 in the vertical direction and a predetermined rule. The rule for associating the product 20 with the display device 30 is stored in advance in a storage device accessible from the display position determination unit 2060.

[0070] For example, a rule may be established that "a display device 30 corresponds to each product 20 displayed on the shelf on which the display device 30 is installed." According to this rule, the display device 30 displays a display 32 of each product 20 located immediately above it.

[0071] 10 is a diagram illustrating a case where a display 32 of a product 20 is displayed on a display device 30 provided on a shelf 42 on which the product 20 is displayed. In this example, a display device 30-1 and a display device 30-2 are provided on two shelves 42-1 and 42-2, respectively. The product 20 is also displayed on the shelf 42-2.

[0072] According to the above-mentioned rules, the product 20 is associated with the display device 30-2 installed on the shelf board 42-2 on which the product 20 is displayed. Therefore, the display 32 is displayed on the display device 30-2.

[0073] Any specific method can be used to associate products 20 with display devices 30 in accordance with this rule. For example, the display position determination unit 2060 identifies on which shelf each display device 30 included in the target image 10 is installed (for example, which shelf is located from the top in the target image 10). Similarly, the display position determination unit 2060 identifies on which shelf each product 20 included in the target image 10 is displayed (for example, which shelf is located from the top in the target image 10). Then, the display position determination unit 2060 associates products 20 and display devices 30 located on the same shelf.

[0074] Another rule that may be defined is, for example, that "a display device 30 is associated with each product 20 displayed on the shelf immediately below the shelf on which the display device 30 is installed." According to this rule, the display device 30 displays a display 32 of each product 20 located immediately below it.

[0075] The specific method for associating products 20 with display devices 30 in accordance with this rule is arbitrary. For example, the display position determination unit 2060 uses the method described above to identify on which shelf each display device 30 included in the target image 10 is located. Similarly, the display position determination unit 2060 identifies on which shelf each product 20 is located. Then, the display position determination unit 2060 associates a display device 30 with each product 20 displayed on the shelf immediately below the shelf on which that display device 30 is located.

[0076] <<<Case where the horizontal position is different>>> This is a case where the target image 10 includes multiple display devices 30 that are long in the horizontal direction, but the horizontal positions of these display devices 30 are different. For example, this is a case where multiple display devices 30 are installed side by side in the horizontal direction on a single shelf 42. In this case, the display position determination unit 2060 associates the product 20 with the display device 30 based on the positional relationship between the product 20 and the display device 30 in the horizontal direction and a predetermined rule. For example, a predetermined rule may be defined such that "among the multiple display devices 30, a product 20 located between the left end and the right end of the display device 30 is associated with the product 20."

[0077] 11 is a diagram illustrating a case where multiple display devices 30 are provided on one shelf 42. In this example, the shelf 42 on which the products 20 are displayed is provided with a display device 30-1 and a display device 30-2.

[0078] Here, the product 20 is located between the left and right ends of the display device 30-1. On the other hand, the product 20 is not located between the left and right ends of the display device 30-2. Therefore, the display position determination unit 2060 associates the product 20 with the display device 30-1. Therefore, the display 32 is displayed on the display device 30-1.

[0079] <> <<<Case where the horizontal position is different>>> This is the case where the target image 10 includes a plurality of display devices 30 that are long in the vertical direction and are positioned differently in the horizontal direction. Specifically, this is the case where the display area 40 has a plurality of side walls (partitions or pillars), and a display device 30 is installed on each of them.

[0080] In this case, the display position determination unit 2060 associates the product 20 with the display device 30 based on the positional relationship between the product 20 and the display device 30 in the horizontal direction and a predetermined rule. For example, a rule may be defined that "the product 20 is associated with the display device 30 that is located to its left and is closest to the product 20 in the horizontal direction." In this case, for example, the display position determination unit 2060 identifies the horizontal positions (e.g., center positions) of each display device 30 and each product 20 included in the target image 10. Then, the display position determination unit 2060 associates the product 20 with the display device 30 that is located to the left of the product 20 and that is closest to the product 20 in the horizontal direction. Note that in the above rule, "left" may be changed to "right."

[0081] Fig. 12 is a diagram illustrating a case in which a plurality of display devices 30 that are long in the vertical direction are provided at different positions in the horizontal direction. Fig. 12 includes shelf board 42-1, shelf board 42-2, and side walls 44-1 to 44-3. Display devices 30-1 to 30-3 are provided on side walls 44-1 to 44-3, respectively. Products 20 are displayed on shelf board 42-2.

[0082] Here, the product 20 and the display device 30 are associated with each other according to the rule that "the product 20 is associated with the display device 30 that is located to the left of the product 20 and is closest to the product 20 in the horizontal direction." death Furthermore, the display device 30 that is closest to the product 20 in the horizontal direction is the display device 30-2. Therefore, the display position determination unit 2060 associates the product 20 with the display device 30-2. Therefore, the display 32 is displayed on the display device 30-2.

[0083] <<<About the case where multiple display devices 30 are present in the vertical direction>>> This is a case where the target image 10 includes a plurality of display devices 30 that are long in the vertical direction, but are positioned at different vertical positions. Specifically, this is a case where a plurality of display devices 30 are installed side by side in the vertical direction on one side wall 44 of a display shelf.

[0084] In this case, the display position determination unit 2060 associates the product 20 with the display device 30 based on the positional relationship between the product 20 and the display device 30 in the vertical direction and a predetermined rule. For example, a predetermined rule may be defined such that "among multiple display devices 30, a display device 30 in which the product 20 is located between the top and bottom ends is associated with the product 20."

[0085] 13 is a diagram illustrating a case where multiple display devices 30 are provided on one side wall 44. Two display devices 30, display device 30-1 and display device 30-2, are provided on the side wall 44. In addition, shelf boards 42-1 and 42-2 are provided, and products 20 are displayed on shelf board 42-2.

[0086] Here, the product 20 is not located between the top and bottom of the display device 30-1, but between the top and bottom of the display device 30-2. Therefore, the display position determination unit 2060 associates the product 20 with the display device 30-2. Therefore, the display 32 is displayed on the display device 30-2.

[0087] <Retrieving related information 54: S110> The second acquisition unit 2040 acquires the related information 54 of the product 20 ( S110 Specifically, the second obtaining unit 2040 obtains the related information 54 associated with the identification information 52 of the product 20 from the related information storage device 50.

[0088] The related information 54 is information related to the product 20 and is displayed on the display device 30. For example, the related information 54 includes the name and price of the product 20. For example, in FIG. 1, the product names and prices of the products 20-1 and 20-2 are displayed on the display device 30.

[0089] The data format of the related information 54 indicating the name and price of the product 20 is arbitrary. For example, the related information 54 indicates the name and price of the product 20 as text data. Alternatively, for example, the related information 54 may indicate the name and price of the product 20 as image data.

[0090] The related information 54 may be information representing an advertisement related to the product 20. In this case, an advertisement for the product 20 is displayed on the display device 30. Here, the advertisement related to the product 20 may be an advertisement related to the product 20 itself, or may be an advertisement for something else. In the latter case, the advertisement related to the product 20 may be, for example, an advertisement for another product in the same series as the product 20, or an advertisement related to the manufacturer of the product 20.

[0091] In this case, the data format of the related information 54 is also arbitrary. For example, the related information 54 may represent an advertisement for the product 20 as text data. Alternatively, the related information 54 may represent an advertisement for the product 20 as a single image or as a plurality of time-series images (e.g., a video file). In the latter case, the display device 30 sequentially displays the time-series images representing the advertisement for the product 20.

[0092] <Generation of display 32: S112> The display control unit 2080 uses the related information 54 acquired about the product 20 to generate a display 32 about the product 20 (S112). For example, the display control unit 2080 applies the content of the acquired related information 54 to a prepared template to generate the display 32. The template is stored in advance in a storage device accessible from the display control unit 2080.

[0093] A template is information that defines the display format of information included in related information 54. For example, if related information 54 represents the name and price of a product, the template defines the layout and size ratio of the product name and price.

[0094] The specific method for realizing a template is arbitrary. For example, a template can be defined as an HTML element. If the product name and price are used as related information 54, for example, an HTML element is prepared as a template in which text or images representing the product name and price can be embedded. In other words, this HTML element has a section for embedding the product name and a section for embedding the product price.

[0095] The display control unit 2080 embeds the name and price of the product 20 shown in the related information 54 into the corresponding portion of the HTML element prepared as a template. The HTML element with the name and price of the product 20 embedded therein is rendered by an HTML rendering engine, thereby generating the display 32.

[0096] The same applies when an advertisement for the product 20 is used as the related information 54. That is, an HTML element is prepared as a template into which text and images representing the advertisement for the product 20 can be embedded, and the display control unit 2080 embeds the content of the related information 54 therein, thereby generating the display 32.

[0097] Here, if there is no specific advertisement for the product 20, a separately prepared advertisement (hereinafter referred to as a default advertisement) may be used. The default advertisement is stored in a storage device accessible from the second acquisition unit 2040. For example, if there is no related information 54 corresponding to the product 20, or if the related information 54 corresponding to the product 20 indicates that there is no advertisement for the product 20, the second acquisition unit 2040 acquires the above-mentioned default advertisement and generates the display 32 with this default advertisement. For example, the default advertisement is an advertisement related to the store, such as an advertisement for discounts or points redemption that are common to all products in the store, or an advertisement related to the store's sponsor.

[0098] Note that templates do not necessarily have to be defined as HTML elements, but can be defined as other of It may be defined in any way.

[0099] Furthermore, the display control unit 2080 does not need to use a template to generate the display 32. For example, assume that the related information 54 shows information about the product 20 in the form of an image. In this case, the display control unit 2080 may use the related information 54 as the display 32 as is.

[0100] Alternatively, for example, the related information 54 may include not only the content of the information about the product 20 but also the display format of the content. Specifically, the related information 54 may include an HTML element in which the name and price of the product 20 are embedded. In this way, by including information about the display format in the related information 54, the display format of the information about the product 20 can be determined for each product 20.

[0101] <<About the display size 32>> Here, the size of the display 32 may be fixed, or may be determined according to the display status of the product 20. In the latter case, for example, the display control unit 2080 determines the size of the display 32 of each product 20 based on the size of the area occupied by that product 20 in the display location 40.

[0102] 14A and 14B are diagrams illustrating a case in which the size of the display device 30 is determined depending on the display state of the products 20. In Fig. 14A, the display control unit 2080 sets the width of the display 32 to be the same as the width of the area occupied by the same type of products 20. In Fig. 14B, the display control unit 2080 sets the height of the display 32 to be the same as the height of the area (tier) occupied by the same type of products 20.

[0103] Here, multiple templates may be prepared according to the size of the display 32. In this case, when generating the display 32, the display control unit 2080 uses a template corresponding to the size of the display 32 determined according to the display status of the product 20. In this way, the display 32 is generated in an appropriate display mode according to the size. For example, a template corresponding to a relatively small size of the display 32 (e.g., a width or height less than a predetermined value) generates a display 32 in which the product name and price are displayed alternately, while a template corresponding to a relatively large size of the display 32 (e.g., a width or height equal to or greater than a predetermined value) generates a display 32 in which both the product name and price are displayed at once. Alternatively, for example, a template corresponding to a relatively small size of the display 32 generates a display 32 that includes only the product name and price, while a template corresponding to a relatively large size of the display 32 generates a display 32 that includes other information (e.g., advertisements) in addition to the product name and price.

[0104] <Display 32: S114> The display control unit 2080 displays the display 32 of each product 20 at the display position on the display device 30 determined for that product 20 (S114). Here, in order to actually control the display device 30 detected from the target image 10 by image analysis, the display control unit 2080 needs to be able to identify the display device 30 to be controlled from one or more display devices 30 in the real world. That is, the display control unit 2080 needs to identify identification information of the display device 30. For example, suppose the display control unit 2080 transmits control information (e.g., information including the display 32 and its display position) to the display device 30, thereby causing the display device 30 to display the display 32. In this case, the display control unit 2080 needs to know the destination address (e.g., MAC address) as identification information of the display device 30.

[0105] There are various methods for identifying the identification information of the display devices 30 detected from the target image 10. For example, information indicating the identification information and installation position of each display device 30 (hereinafter referred to as display layout information) is stored in advance in a storage device, and the identification information of each display device 30 is identified using this information.

[0106] As a more specific example, assume that a camera is provided for each display shelf to capture an image of that display shelf. In this case, the identification information of the camera is associated with the display layout information of each display device 30 installed on the display shelf captured by the camera and stored in advance in a storage device. The display control unit 2080 identifies the identification information of the camera that generated the target image 10 acquired by the first acquisition unit 2020, and acquires the display layout information associated with the identification information. The display control unit 2080 compares the position of each display device 30 in the target image 10 with the installation position of the display device 30 indicated by the acquired display layout information, thereby identifying the display layout information corresponding to each display device 30 included in the target image 10. This identifies the identification information of each display device 30 included in the target image 10.

[0107] The camera may be configured to change its angle of view (for example, a PTZ camera). In this case, the display control unit 2080 can determine the imaging range of the camera at the time the target image 10 was generated (i.e., which part of the display shelf is included in the target image 10) by using the target image 10 and the camera parameters (pan value, tilt value, zoom value, etc.) used when the target image 10 was captured. The display control unit 2080 identifies the identification information of each display device 30 included in the target image 10 by using the display layout information and the imaging range of the camera identified using the camera parameters.

[0108] Alternatively, for example, each display device 30 may have a function to display its own identification information. In this case, the display control device 2000 acquires a target image 10 captured by each display device 30 with the identification information displayed thereon, and identifies the identification information displayed on each display device 30 by performing image analysis on the target image 10. This allows the display control device 2000 to ascertain the identification information of each display device 30 included in the target image 10. Note that the technology for "displaying identification information on a display device and analyzing an image including that display device to ascertain the identification information of that display device" can utilize the technology disclosed in Patent Document 2, for example.

[0109] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes. 1. A first acquisition unit that acquires a target image, the target image includes a display device provided in a display area of ​​the product and the product displayed in the display area; a second acquisition unit that recognizes a product included in the target image and acquires related information that is information related to the product; a display position determination unit that determines a display position for displaying a related display showing related information of the product on the display device based on a relative positional relationship between the product and the display device in the target image; a display control unit that displays the related display of the product at the determined display position on the display device. 2. The target image includes a plurality of the display devices; the display position determination unit determines the display device on which the related display of the product is to be displayed and a display position of the related display on the display device, based on a positional relationship between the product and each of the display devices in the target image; 1. The display control device according to 1, wherein the display control unit displays a display related to the product at the determined display position on the determined display device. 3. The display control device according to 2., wherein a plurality of the display devices are provided on one shelf or one side wall. 4. A display control device described in any one of 1 to 3, wherein the related information includes the name and price of the product, or includes an advertisement related to the product. 5. A display control device described in any one of 1 to 4, wherein the display control unit determines the size of the related display of the product based on the size of the area occupied by the product in the target image. 6. A display control device described in any one of 1 to 5, wherein the display control unit obtains a template that defines the display format of the related display, and uses the related information and the template to generate the related display that represents the related information. 7. A plurality of templates are prepared according to the size of the area occupied by the product in the target image; 6. The display control device according to claim 6, wherein the display control unit generates the related display of the product using a template corresponding to the size of the area occupied by the product in the target image. 8. A computer-implemented control method comprising: a first acquisition step of acquiring a target image; the target image includes a display device provided in a display area of ​​the product and the product displayed in the display area; a second acquisition step of recognizing a product included in the target image and acquiring related information related to the product; a display position determination step of determining a display position for displaying a related display showing related information of the product on the display device based on a relative positional relationship between the product and the display device in the target image; a display control step of displaying the related display of the product at the display position on the display device. 9. The target image includes a plurality of the display devices; In the display position determination step, the display device on which the related display of the product is to be displayed and the display position of the related display on the display device are determined based on a positional relationship between the product and each of the display devices in the target image; 8. The control method according to 8, wherein in the display control step, a display related to the product is displayed at the determined display position on the determined display device. 10. The control method according to claim 9, wherein a plurality of the display devices are provided on one shelf or one side wall. 11. A control method described in any one of 8 to 10, wherein the related information includes the name and price of the product, or includes an advertisement related to the product. 12. A control method described in any one of 8 to 11, wherein in the display control step, the size of the related display of the product is determined based on the size of the area occupied by the product in the target image. 13. A control method described in any one of 8 to 12, wherein in the display control step, a template that defines the display format of the related display is obtained, and the related display representing the related information is generated using the related information and the template. 14. A plurality of templates are prepared according to the size of the area occupied by the product in the target image; 14. The control method according to 13, wherein in the display control step, the related display of the product is generated using a template corresponding to the size of the area that the product occupies in the target image. 15. A program that causes a computer to execute the control method described in any one of 8 to 14.

[0110] Further reference examples are given below. 16. A control method executed by a computer that realizes a display control device, comprising: The display control device, Acquire the target image, the target image includes a display device provided in a display area of ​​the product and the product displayed in the display area; Furthermore, the display control device Recognizing a product included in the target image and acquiring related information related to the product; determining a display position for displaying a related display showing related information of the product on the display device based on a relative positional relationship between the product and the display device in the target image; A control method for displaying the related display of the product at the display position on the display device. 17. The target image includes a plurality of the display devices; The display control device, determining the display device on which the related display of the product is to be displayed and the display position of the related display on the display device based on the positional relationship between the product and each of the display devices in the target image; 16. The control method according to claim 16, further comprising displaying a display related to the product at the determined display position on the determined display device. 18. The control method according to claim 17, wherein a plurality of the display devices are provided on one shelf or one side wall. 19. A control method according to any one of claims 16 to 18, wherein the related information includes the name and price of the product, or includes an advertisement relating to the product. 20. The display control device A control method described in any one of 16. to 19., in which the size of the related display of the product is determined based on the size of the area occupied by the product in the target image, and the related display of the determined size is displayed. 21. The display control device A control method described in any one of 16. to 20., which includes obtaining a template that defines the display format of the related display, using the related information and the template to generate the related display that represents the related information, and displaying the generated related display. 22. A plurality of templates are prepared according to the size of the area occupied by the product in the target image; The display control device, 21. The control method described in 21., wherein the related display of the product is generated using a template corresponding to the size of the area occupied by the product in the target image, and the generated related display is displayed.

[0111] 23. To the computer, Execute the procedure to acquire the target image, the target image includes a display device provided in a display area of ​​the product and the product displayed in the display area; On the computer, a step of recognizing a product included in the target image and acquiring related information related to the product; determining a display position for displaying a related display showing related information of the product on the display device based on a relative positional relationship between the product and the display device in the target image; a program for executing a procedure for displaying the related display of the product at the display position on the display device; 24. The target image includes a plurality of the display devices; a step of determining the display device on which the related display of the product is to be displayed and the display position of the related display on the display device based on a positional relationship between the product and each of the display devices in the target image; 23. The program described in 23. for causing a computer to execute a procedure for displaying the related display of the product at the determined display position on the determined display device. 25. The program according to claim 24, wherein a plurality of the display devices are provided on one shelf or one side wall. 26. The program according to any one of 23 to 25, wherein the related information includes the name and price of the product, or includes an advertisement related to the product. 27. A program described in any one of 23. to 26. for causing a computer to execute the steps of determining the size of the related display of the product based on the size of the area occupied by the product in the target image, and displaying the related display of the determined size. 28. A program described in any one of 23. to 27. for causing a computer to execute the steps of obtaining a template that defines the display format of the related display, using the related information and the template to generate the related display that represents the related information, and displaying the generated related display. 29. A plurality of templates are prepared according to the size of the area occupied by the product in the target image; 28. A program for causing a computer to execute a procedure for generating the related display of the product using a template corresponding to the size of the area occupied by the product in the target image, and displaying the generated related display.

[0112] This application claims priority based on Japanese Patent Application No. 2019-223559, filed December 11, 2019, the disclosure of which is incorporated herein in its entirety. [Explanation of symbols]

[0113] 10 Target image 20 items 30 Display device 32 displays 40 Display area 42 Shelf 44 Side wall 50 Related Information Storage Device 52 Identification Information 54 Related Information 56 Image Features 70 Server Machine 80 Camera 1000 calculator 1020 Bus 1040 processor 1060 memory 1080 storage device 1100 Input / Output Interface 1120 Network Interface 2000 Display Control Device 2020 1st Acquisition Division 2040 2nd Acquisition Department 2060 Display position determination section 2080 display control unit

Claims

1. a first acquisition unit that acquires a target image; the target image includes a display device provided in a display area of ​​the product and the product displayed in the display area; a second acquisition unit that recognizes a product included in the target image and acquires related information that is information related to the product; a display position determination unit that determines a display position for displaying a related display showing related information of the product on the display device based on a relative positional relationship between the product and the display device in the target image; a display control unit that displays the related display of the product at the determined display position on the display device, the display control unit acquires a template that defines a display mode of the related information display, and generates the related information display that represents the related information using the related information and the template; a plurality of templates are prepared according to the size of the area occupied by the product in the target image; A template corresponding to a case where the size of the area is less than a predetermined value generates the related display by alternating between the name and price of the product, and a template corresponding to a case where the size of the area is equal to or greater than the predetermined value generates the related display by simultaneously using both the name and price; The display control device is configured to generate the related display of the product using a template corresponding to the size of an area occupied by the product in the target image.

2. the target image includes a plurality of the display devices; the display position determination unit determines the display device on which the related display of the product is to be displayed and a display position of the related display on the display device, based on a positional relationship between the product and each of the display devices in the target image; The display control device according to claim 1 , wherein the display control unit causes the display related to the product to be displayed at the determined display position on the determined display device.

3. The display control device according to claim 2 , wherein a plurality of the display devices are provided on one shelf or one side wall.

4. The display control device according to claim 1 , wherein the related information includes a name and a price of the product, or includes an advertisement related to the product.

5. The display control device according to claim 1 , wherein the display control unit determines a size of the related display of the product based on a size of an area occupied by the product in the target image.

6. A control method executed by a computer that realizes a display control device, comprising: The display control device, Acquire the target image, the target image includes a display device provided in a display area of ​​the product and the product displayed in the display area; Furthermore, the display control device Recognizing a product included in the target image and acquiring related information related to the product; determining a display position for displaying a related display showing related information of the product on the display device based on a relative positional relationship between the product and the display device in the target image; Displaying the related display of the product at the display position on the display device; obtaining a template that defines a display format of the related information display, and generating the related information display using the related information and the template; a plurality of templates are prepared according to the size of the area occupied by the product in the target image; A template corresponding to a case where the size of the area is less than a predetermined value generates the related display by alternating between the name and price of the product, and a template corresponding to a case where the size of the area is equal to or greater than the predetermined value generates the related display by simultaneously using both the name and price; Furthermore, the display control device The control method generates the related display of the product using a template corresponding to the size of the area that the product occupies in the target image.

7. On the computer, Execute the procedure to acquire the target image, the target image includes a display device provided in a display area of ​​the product and the product displayed in the display area; On the computer, a step of recognizing a product included in the target image and acquiring related information related to the product; determining a display position for displaying a related display showing related information of the product on the display device based on a relative positional relationship between the product and the display device in the target image; a step of displaying the related display of the product at the display position on the display device; In the step of displaying, a template that defines a display format of the related information display is acquired, and the related information display representing the related information is generated using the related information and the template; a plurality of templates are prepared according to the size of the area occupied by the product in the target image; A template corresponding to a case where the size of the area is less than a predetermined value generates the related display by alternating between the name and price of the product, and a template corresponding to a case where the size of the area is equal to or greater than the predetermined value generates the related display by simultaneously using both the name and price; In the displaying step, the program generates the related display of the product using a template corresponding to the size of the area that the product occupies in the target image.

Images