SHOPPING SUPPORT SYSTEM, SHOPPING SUPPORT METHOD, AND PROGRAM

Abstract

This shopping assistance system comprises a first output control unit, a commodity detection unit, and a second output control unit. The first output control unit causes a first display to show information relating to sales floors of a store where commodities are displayed. The commodity detection unit detects, on the basis of the action of a customer, a commodity that the customer wants to acquire. The second output control unit causes a second display to show information relating to the detected commodity.

Description

Shopping support system, shopping support method, and recording medium

[0001] The present disclosure relates to a shopping support system and the like.

[0002] Dome-shaped displays are known as a means for displaying virtual spaces and images (see, for example, Patent Documents 1 and 2). Wearable information processing terminals are also known as a means for displaying virtual spaces and images (see, for example, Patent Document 3).

[0003] International Publication No. 2018 / 101279 International Publication No. 2017 / 187821 Japanese Patent Application Laid-Open No. 2020-129356

[0004] For example, when a large display such as a dome-shaped display is used as a display means, a user may need to use a controller to perform operations. However, using a controller can be problematic in that it can be difficult to operate. For example, when shopping in a virtual store or a remote store, some users may find it cumbersome to use a controller to view products displayed in the store.

[0005] An example of an object of the present disclosure is to provide a shopping support system or the like that can improve convenience when using a large display.

[0006] A shopping support system in one aspect of the present disclosure includes a first output control means for causing a first display to display information about the sales area of ​​a store where products are displayed, a product detection means for detecting products that a customer wishes to acquire based on the customer's behavior, and a second output control means for causing a second display to display information about the detected products.

[0007] A shopping support method in one aspect of the present disclosure displays information about a store's sales area where products are displayed on a first display, detects products that the customer wants to purchase based on the customer's behavior, and displays information about the detected products on a second display.

[0008] A program in one aspect of the present disclosure causes a computer to execute a process that displays information about the store's sales area where products are displayed on a first display, detects products that the customer wants to purchase based on the customer's behavior, and displays information about the detected products on a second display.

[0009] The program may be stored in a non-transitory computer-readable recording medium.

[0010] According to the present disclosure, it is possible to improve convenience when using a large display.

[0011] 1 is a block diagram showing an example of a configuration of a shopping support system according to a first embodiment. FIG. 2 is a flowchart showing an example of an operation of the shopping support system according to the first embodiment. FIG. 3 is an explanatory diagram showing an example of a dome-shaped display. FIG. 4 is an explanatory diagram showing an example of an installation of an aerial display and an example of an installation of devices around a customer. FIG. 5 is an explanatory diagram showing an example of a sales floor of a store. FIG. 6 is an explanatory diagram showing a simplified aerial display. FIG. 7 is a block diagram showing an example of a configuration of a shopping support system according to a second embodiment. FIG. 7 is an explanatory diagram showing an example of a sales floor of a store displayed on a dome-shaped display. FIG. 8 is an explanatory diagram showing an example of detecting a product from a customer's hand movements and displaying information about the product on an aerial display. FIG. 9 is an explanatory diagram showing an example of detecting a product from a customer's line of sight and displaying information about the product on an aerial display. FIG. 10 is an explanatory diagram showing an example of detecting a product from a customer's conversation and displaying information about the product on an aerial display. FIG. 11 is an explanatory diagram showing an example of changing the orientation of a displayed product by changing the position and orientation of an imaging device that captures an image of the product. FIG. 12 is an explanatory diagram showing an example of changing the orientation of a displayed product by switching an imaging device. FIG. 13 is an explanatory diagram showing an example of changing the orientation of a displayed product by rotating a platform. FIG. 14 is a flowchart showing an example of an operation of the shopping support system according to the second embodiment. FIG. 15 is an explanatory diagram showing an example of an implementation of the shopping support system. FIG. 16 is an explanatory diagram showing an example of the hardware configuration of a computer.

[0012] Hereinafter, with reference to the drawings, embodiments of a shopping support system, a shopping support method, a program, and a non-transitory recording medium for recording a program according to the present disclosure will be described in detail. The embodiments do not limit the disclosed technology. Here, each embodiment may be described using an airborne display as an example of a non-contact display.

[0013] In addition, in each embodiment, shopping is used as an example, the user of the first display is called a customer, and the system is described as a shopping support system, but use cases in which the first display is combined with a second display such as an aerial display are not limited to shopping.

[0014] (Embodiment 1) First, in embodiment 1, basic functions of a shopping support system will be described. Fig. 1 is a block diagram showing an example configuration of a shopping support system according to embodiment 1. The shopping support system 10 includes a first output control unit 101, a product detection unit 102, and a second output control unit 103.

[0015] The first output control unit 101 controls the display of the first display. The first display is, for example, a large display. Here, "large" means that it is at least larger than the second display. For example, the first display may be a display device configured to cover the customer's field of vision. This allows the customer to shop in an immersive state. Here, a dome-shaped display is used as the first display. For example, the first output control unit 101 causes the dome-shaped display to display information about a sales floor in a store where products are displayed. Here, the information about the sales floor may include video or images of the sales floor. Note that the video here refers to moving images, and the images here refer to still images. Note that the first output control unit 101 may also cause the dome-shaped display to display information about other products, such as the name of the sales floor, as additional information, along with the video or images of the sales floor. Here, the store may be any type of retail store, such as a supermarket, a supercenter, a convenience store, a mass retailer, a home improvement store, a drugstore, a store selling apparel, a department store, or an independent store such as a bakery or a prepared food store, and is not particularly limited.

[0016] The commodity detection unit 102 detects a commodity that a customer wants to acquire based on the customer's behavior. For example, a commodity that a customer wants to acquire is a commodity that the customer wants to check details about. For example, a commodity that a customer wants to acquire is a commodity that the customer is interested in. Here, the customer's behavior is, for example, the customer's movement. Specifically, the customer's behavior may be, for example, a specific body movement such as the customer's hand movement or the customer's eye movement among the customer's movements in more detail. For example, the commodity detection unit 102 detects the customer's movement from an image captured by an imaging device. For example, if the customer's behavior is the customer's hand movement, the commodity detection unit 102 identifies the position where the customer reaches out their hand based on the customer's hand movement and detects a commodity corresponding to the identified position. The commodity corresponding to the identified position is, for example, a commodity displayed on a dome-shaped display that is located on an extension of the identified position or a commodity within a predetermined range from the identified position.

[0017] Furthermore, for example, the customer's behavior may be a conversation between the customer and a store clerk, and the commodity detection unit 102 may detect the customer's behavior by performing voice recognition on the conversation.

[0018] The second output control unit 103 controls the display of the second display. The second display is a display for the customer. The second display is, for example, a display located closer to the customer than the first display. For example, the second display is located between the customer and the first display. Also, for example, the second display is a display installed in a position where the display is easily visible to the customer. For example, the display area of ​​the second display is smaller than the display area of ​​the first display. For example, the first display displays information about the sales floor, such as images and videos of the sales floor, while the second display displays information about products, such as images and videos of products that have attracted attention, such as products selected by the customer. For this reason, for example, it is desirable that the second display be installed closer to the customer than a dome-shaped display and display information about products in an easily visible manner.

[0019] The second display may be a fixed display or a portable display. Here, a non-contact type display is used as the second display, but a contact type display may also be used.

[0020] Furthermore, the objects to be displayed are not particularly limited, and may include, for example, images, videos, letters and numbers, and special codes that can be read by any device.

[0021] For example, the second output control unit 103 displays information about the detected product on a non-contact display. For example, the information about the product may be an image of the product, a video of the product, or information related to the product. For example, the information related to the product may be letters, numbers, or codes related to the product, and is not limited to specific information such as the price, name, place of origin, or manufacturer of the product. The product code may be a code representing letters or numbers related to the product. The video of the product may be real-time video or video captured in advance. The image of the product may be an image newly captured in real time or an image captured in advance. Although the product is also shown in the video or image of the sales floor, the video or image of the product is a video or image that focuses on the product.

[0022] 2 is a flowchart showing an example of the operation of the shopping support system 10 according to the first embodiment. The first output control unit 101 causes the dome display to display information about the sales floor of the store (step S101).

[0023] The product detection unit 102 determines whether a product desired by the customer has been detected (step S102). In step S102, the product detection unit 102 detects a product desired by the customer based on the customer's behavior. If a product desired by the customer has not been detected (step S102: No), the product detection unit 102 returns to step S102.

[0024] If the product desired by the customer is detected (step S102: No), the second output control unit 103 displays information about the detected product on the non-contact display (step S103).Then, the shopping support system 10 ends the process.

[0025] As described above, in the first embodiment, the shopping support system 10 displays, for example, a store on a first display, which is a large display, and displays products detected by customer behavior on a second display. This improves convenience when using a large display such as a dome-shaped display. Therefore, a more realistic shopping experience can be provided. For example, the shopping support system 10 displays information about a store's sales floor on a dome-shaped display. When a customer selects a displayed product, the shopping support system 10 can display a large image or video of the selected product on a handheld display, or present additional information about the selected product. Furthermore, compared to viewing each product or product information on a dome-shaped display, it is easier for customers to check the product and product information when they can see it up close using a handheld display.

[0026] Furthermore, for example, the second display is installed between the customer and the first display. When an aerial display is used as the second display, there is no physical object between the customer and the first display when product information is not displayed on the aerial display. Therefore, when an aerial display is used as the second display, it does not interfere with the customer's hand movements or the customer's field of view when looking at the first display. This provides a shopping experience that is closer to actual shopping, such as being able to reach for products or have an unobstructed conversation with a store clerk. Furthermore, by making the aerial display invisible to surrounding people, information that the customer does not want others to see, such as the customer's personal information, can be presented to the customer via the aerial display. Note that the first display is large, so if various information is displayed, there is a risk that it will be seen by others. Thus, information that is not visible to others is difficult to display on the first display. Furthermore, when an aerial display is used as the second display, the customer may be able to see the products in three dimensions.

[0027] Furthermore, when a wearable information processing terminal such as an HMD (Head Mounted Display) is used as a display means, the user may not be able to see what is in front of them when the virtual space is displayed. On the other hand, when a dome-shaped display is used as a display means, the user has the advantage of being able to see what is in front of them even when the virtual space is displayed. Furthermore, when a dome-shaped display is used as a display means, there is a distance between the dome-shaped display and the user, making it possible to install a second display such as a spatial display.

[0028] Furthermore, when a dome-shaped display is used, general-purpose use is conceivable. For example, it is assumed that a dome-shaped display is installed in a location accessible to everyone and that a variety of users will use the dome-shaped display. In such cases, some users may be unfamiliar with the controller. According to the first embodiment, it is possible to improve operability during shopping. By improving the convenience when using a dome-shaped display as in the first embodiment, it is possible to further enhance versatility. Furthermore, by displaying videos and images of the sales floor on the dome-shaped display 21 that covers the entire field of view, customers can shop in an immersive state. Therefore, it is possible to provide a shopping experience that is closer to actual shopping.

[0029] However, the operation by the controller and the shopping support system 10 may be used in combination as appropriate.

[0030] (Embodiment 2) Next, embodiment 2 will be described in detail with reference to the drawings. In embodiment 2, an example of a dome-shaped display as the first display and an example of installing an aerial display as the second display will be described. Furthermore, in embodiment 2, an example of controlling the display of the dome-shaped display to make the displayed product easier to see while information about the product is being displayed on the aerial display will be described. Below, explanations of content that overlaps with the above explanation will be omitted to the extent that the explanation of embodiment 2 is not unclear.

[0031] Fig. 3 is an explanatory diagram showing an example of a dome-shaped display. Fig. 4 is an explanatory diagram showing an example of an installation of an aerial display and an example of installation of devices around a customer. In Fig. 3, the dome-shaped display 21 includes, for example, a screen 2101, a projection device 2102, and a table 2103.

[0032] When the dome display 21 displays a video or image, the projection device 2102 projects the video or image onto the screen 2101 .

[0033] For example, the aerial display 22 is installed on a table 2103. Also, in FIG. 4 , the screen 2101 and the projection device 2102 of the dome-shaped display 21 are omitted. In FIG. 4 , a customer is sitting in a chair and looking at an apple displayed on the dome-shaped display and the aerial display 22. While FIG. 4 shows an example of viewing the table 2103 from the side, in reality, the aerial image of the aerial display 22, as described below, may not have a thickness. In such cases, the aerial image cannot be seen when viewed from the side in the thickness direction. However, for ease of understanding, the following description will be given using an example in which an apple or the like is displayed when viewed from the side. Also, in FIG. 4 , an imaging device 23 and a recording device 24 may be installed where the dome-shaped display 21 is installed. For example, the imaging device 23 and the recording device 24 are used to detect customer behavior. Although not shown, an audio output device, such as a speaker that outputs the voice of a store clerk, may be installed where the dome-shaped display 21 is installed.

[0034] There is no particular limitation on the number of imaging devices 23, recording devices 24, and aerial displays 22. For example, the imaging devices 23 may transmit captured video or images to a shopping support system or the like. The recording devices 24 may transmit audio to a shopping support system or the like.

[0035] The dome-shaped display 21 is, for example, a device in which at least a portion of the screen 2101 is curved, so as to obscure the customer's field of view. The dome-shaped display 21 may not be a complete dome, such as a 360-degree dome, but may have a portion missing, such as a 180-degree dome. The dome-shaped display 21 does not need to have a portion of the screen 2101 curved. The dome-shaped display 21 is sized to provide a sufficient gap between the user and the screen 2101 to allow for the placement of a table 2103 on which the aerial display 22 is installed. Thus, the size and type of the dome-shaped display 21 are not particularly limited. For example, the dome-shaped display 21 may be a 180-degree dome-shaped display 21 or a 360-degree dome-shaped display 21. The size of the dome-shaped display 21 may be, for example, approximately 1 to 2 meters in length, 1 to 2 meters in width, and 1 to 2 meters in height.

[0036] Furthermore, there is no particular limitation on the installation location of the dome-shaped display 21. For example, the dome-shaped display 21 may be installed in a customer's home, a company, or the like, or may be installed in a location where anyone can use it.

[0037] FIG. 5 is an explanatory diagram showing an example of a sales floor of a store. For example, product shelves and the like are installed in the sales floor of the store. Products are displayed on the installed shelves. The store may be a store in a virtual space or an actual store. The store in the virtual space may be a virtual store that does not actually exist, or a virtual store that imitates an actual store. When an actual store exists and an image of the actual store is displayed on the dome-shaped display 21, for example, an imaging device 25 and a recording device 26 are installed in the sales floor of the actual store. Furthermore, although not shown, an audio output device such as a speaker that outputs the voices of customers may be installed in the sales floor of the actual store.

[0038] 5, two imaging devices 25-1 and 25-2 are installed, but the number of imaging devices 25 is not particularly limited. When imaging device 25 is not particularly limited, it will be referred to as imaging device 25. Imaging device 25 captures images of the sales floor of the store. Imaging device 25 may also capture images of store clerks. Imaging device 25 may also capture images of each product in the store. Here, imaging device 25-1 may capture images of the sales floor of the store including the clerks, and imaging device 25-2 may capture images of specified products. For example, imaging device 25 may transmit the captured images to a shopping support system or the like.

[0039] The recording device 26 also records or collects sounds from the sales floor, and may transmit the sounds to a shopping support system, for example.

[0040] 5 shows an example in which an actual store is used, but a store in a virtual space may also be used. For example, a store in a virtual space may be used when there is no actual store.

[0041] 6 is a simplified explanatory diagram of the aerial display 22. The aerial display 22 includes, for example, an optical element, a display, and a sensor.

[0042] In the aerial display 22, an optical element passes light emitted by the display and forms an identical image on the opposite side of the display. In FIG. 6, this image is the aerial image. A sensor is used to manipulate the aerial image. The sensor is, for example, a motion sensor. In the following description, an operation that can be detected by the sensor is also referred to as an operation on the aerial display. The type of aerial display is not particularly limited.

[0043] 7 is a block diagram showing an example of the configuration of a shopping support system according to embodiment 2. Shopping support system 20 includes first output control unit 201, product detection unit 202, second output control unit 203, acquisition unit 204, video analysis unit 205, audio analysis unit 206, imaging device control unit 207, platform control unit 208, superimposition position identification unit 209, viewing position identification unit 210, and operation reception unit 211.

[0044] The shopping support system 20 includes, in addition to the components of the shopping support system 10 according to the first embodiment, an acquisition unit 204, a video analysis unit 205, an audio analysis unit 206, an imaging device control unit 207, a platform control unit 208, a superimposition position identification unit 209, a viewing position identification unit 210, and an operation reception unit 211. The first output control unit 201 has, as a basic function, the function of the first output control unit 101 according to the first embodiment. The commodity detection unit 202 has, as a basic function, the function of the commodity detection unit 102 according to the first embodiment. The second output control unit 203 has, as a basic function, the function of the second output control unit 103 according to the first embodiment.

[0045] Furthermore, for example, the shopping support system 20 includes a customer database (DB) 2001, a store database 2002, and a product database 2003. Each functional unit of the shopping support system 20 can refer to and update various databases as needed.

[0046] For example, the customer DB 2001 stores images and voices of customers, customer behavior history, etc. The behavior history may be a history of customer movements detected from images, a conversation history detected from voices, or a history of customer operations on the aerial display 22.

[0047] Furthermore, for example, the store DB 2002 stores store videos, store images, store audio, and the like.

[0048] The product DB 2003 stores product identification information and product information for each product in association with each other. Product information includes, for example, product video, product image, product price, product name, product display position in the sales area, and product features. Product features may include product color, product shape, product origin, product ingredients, and product manufacturing company. For example, product features may be used to identify a product from conversations between a customer and a store clerk, so the information is not particularly limited as long as it can identify the product.

[0049] Next, each functional unit will be described.

[0050] For example, the acquisition unit 204 acquires a captured image of a customer from the imaging device 23. For example, the acquired image data is stored in the customer DB 2001. Furthermore, for example, the acquisition unit 204 acquires the customer's voice from the recording device 24. For example, the acquired voice data is stored in the customer DB 2001.

[0051] For example, if there is an actual store, the acquisition unit 204 acquires captured video of the store from the imaging device 25. For example, the data of the acquired video is stored in the store DB 2002, for example. Note that the acquisition unit 204 may also acquire video of the store that has been created in advance. For example, if there is no actual store, a video of the store in a virtual space may be created. For example, the data of the acquired video is stored in the store DB 2002, for example.

[0052] If there is an actual store, the acquisition unit 204 acquires the voice of the store from the recording device 26. The acquired voice data is stored in the store DB 2002, for example.

[0053] The acquisition unit 204 may also acquire audio in the virtual space.

[0054] The first output control unit 201 causes the dome-shaped display 21 to display information about the sales floor of the store where the products are displayed. For example, the information about the sales floor of the store is a video or image of the sales floor. Specifically, if there is an actual store, for example, the first output control unit 201 causes the dome-shaped display 21 to display a video of the sales floor acquired from the imaging device 25. This allows customers to view the video of the sales floor acquired in real time, making them feel as if they are shopping in an actual store.

[0055] Fig. 8 is an explanatory diagram showing an example of a store's sales floor being displayed on the dome-shaped display 21. In Fig. 8, the dome-shaped display 21 displays the sales floor as shown in Fig. 5. For example, a customer can sit in a chair and view the sales floor.

[0056] Returning to the description of FIG. 7 , the first output control unit 201 further displays information about the store clerk on the dome-shaped display 21. The information about the store clerk includes a video of the store clerk, an image of the store clerk, the name of the store clerk, the affiliation of the store clerk, and characteristics of the store clerk. The affiliation of the store clerk refers to the store clerk's affiliation, such as "in charge of the vegetable section." The characteristics of the store clerk may include, for example, a qualification held by the store clerk. For example, if a store clerk is present in the actual store, the imaging device 25 captures an image of the sales floor including the store clerk. Then, the first output control unit 201 displays the captured image on the dome-shaped display 21. In the example of FIG. 8 , the captured image includes the store clerk in the sales floor of FIG. 5 . Furthermore, when displaying information about the store clerk, the first output control unit 201 may also display an avatar of the store clerk. For example, when displaying the avatar of the store clerk, the imaging device 25 captures an image of the sales floor. The first output control unit 201 may then superimpose an avatar of the store clerk onto the captured image and display the superimposed image. Alternatively, for example, the first output control unit 201 may superimpose an avatar of the store clerk onto an image of a sales floor in a virtual space and display the superimposed image.

[0057] If the voice of the store clerk can be acquired from the recording device 24, the acquisition unit 204 may acquire the voices of the customer and the store clerk from the recording device 24.

[0058] The product detection unit 202 detects products that the customer wants to purchase based on the customer's behavior. First, in order for the product detection unit 202 to detect products, for example, the video analysis unit 205 analyzes the customer's behavior from video. Then, the product detection unit 202 detects products based on the analyzed customer behavior. Alternatively, in order for the product detection unit 202 to detect products, the audio analysis unit 206 analyzes the customer's behavior from audio. Then, the product detection unit 202 detects products based on the analyzed customer behavior. Note that product detection examples will be described later for each type of customer behavior.

[0059] Then, the second output control unit 203 displays information about the detected product on the aerial display 22. The information about the product may be a video of the product, an image of the product, or information related to the product such as the product name. The video of the product may be, for example, a captured video or a video of the product reproduced in a virtual space. The captured video may be a video of the product captured in real time by the imaging device 25, or a video of the product captured in advance. Furthermore, the image of the product may be a captured image of the product, an image of the product such as an illustration, or an image of the product reproduced in a virtual space. The captured image may be a video of the product captured in real time by the imaging device 25, or a video of the product captured in advance.

[0060] <Examples of Product Detection> Next, examples of product detection based on customer behavior will be described.

[0061] <Example of Item Detection Based on Hand Movement> First, an example will be described in which the customer's behavior is the customer's hand movement. For example, if the customer's behavior is the customer's hand movement, the video analysis unit 205 may detect the customer's hand movement from the video captured by the imaging device 23. For example, the item detection unit 202 detects the item that the customer wants to acquire based on the customer's hand movement. As a more specific method of detecting the item, for example, the item detection unit 202 detects the item based on the position at which the customer extends their hand. For example, the item detection unit 202 detects the item at the position at which the customer extends their hand or the item near that position. The item near the position at which the customer extends their hand may be the item on an extension of the position at which the customer extends their hand, or the item closest to the position at which the customer extends their hand.

[0062] Here, an example of detecting the customer's hand movement and displaying information about the product on the aerial display 22 will be described with reference to FIG. 9 .

[0063] 9 is an explanatory diagram showing an example of detecting a product from the hand movement of a customer and displaying information about the product on the aerial display 22. In FIG. 9, the screen 2101 of the dome-shaped display 21 and the projection device 2102 are omitted from the illustration. For example, the imaging device 23 captures an image of the customer. Then, the acquisition unit 204 acquires the captured image of the customer in real time. Then, the product detection unit 202 detects the behavior of the customer from the image.

[0064] 9, the video analysis unit 205 detects, based on the video, that a customer is standing up from a seated position and reaching out toward the dome-shaped display 21. Also, in FIG. 9, the product detection unit 202 detects the product at the end of the customer's reaching hand from among the products in the sales area displayed on the dome-shaped display 21. Then, the second output control unit 203 outputs information about the detected product to the aerial display 22. As a result, in FIG. 9, the aerial display 22 displays information about the product, such as an image or video of the product. In the example of FIG. 9, an apple is displayed.

[0065] Returning to the description of Fig. 7, the hand movement is not limited to being detected by a sensor such as the imaging device 23, but may be detected by a dedicated glove or the like. Note that the sensor is not limited to the imaging device 23 and can be variously changed.

[0066] <Example of Product Detection Based on Eye Movement> Next, an example will be described in which the customer's behavior is the customer's eye movement. For example, when the customer's behavior is the customer's eye movement, the video analysis unit 205 may detect the customer's eye movement from an image captured by the imaging device 23.

[0067] For example, a product that a customer is looking at may be a product that the customer wants to check. Therefore, the product detection unit 202 detects the product that the customer is looking at as a product that the customer wants to acquire. Note that examples of a product that a customer is looking at include a product that the customer is looking at, a product that the customer is looking at for a predetermined period of time, and a product that the customer is looking at a predetermined number of times. There is no particular limitation on the method for identifying a product that a customer is looking at.

[0068] Here, an example of detecting the customer's line of sight and displaying information about a product on the aerial display 22 will be described with reference to FIG. 10 .

[0069] 10 is an explanatory diagram showing an example of detecting a product from the movement of a customer's gaze and an example of displaying information about the product on the aerial display 22. In FIG. 10, the screen 2101 of the dome-shaped display 21 and the projection device 2102 are omitted from the illustration. For example, the imaging device 23 captures an image of the customer. Then, the acquisition unit 204 acquires the captured image of the customer in real time. Then, the image analysis unit 205 detects the movement of the customer's gaze from the image.

[0070] 10, the product detection unit 202 detects, based on the video, a product that a customer is looking at for a predetermined period of time or longer from among the products in the sales area displayed on the dome-shaped display 21. Then, the second output control unit 203 displays information about the detected product on the aerial display 22. As a result, in Fig. 10, the aerial display 22 displays information about the product, such as an image or video of the product. In the example of Fig. 10, an apple is displayed.

[0071] Returning to the description of Fig. 7, the movement of the line of sight is not limited to being detected by the imaging device 23, but may be detected by other sensors. Furthermore, regardless of the movement of the line of sight, the direction of the face may be used.

[0072] <Example of Product Detection Based on Conversation> Next, an example will be described in which the customer's behavior is a conversation with a store clerk. For example, if the customer's behavior is a conversation between the customer and a store clerk, the voice analysis unit 206 performs voice recognition on the conversation between the customer and the store clerk. The voice analysis unit 206 then detects keywords that can identify the product, such as the product's name, the product's display position in the sales area, the product's price, and the product's characteristics, from the conversation. The product's characteristics may be the product's color, product's shape, the product's place of origin, the product's raw materials, the product's manufacturer, etc. The product's position in the sales area may be, but is not limited to, the right end, the left end, the upper right, or the position on the shelf. For example, a keyword that includes the product's position may be, for example, "the apple on the far right." For example, the product detection unit 202 detects products that match the detected keywords from the product DB 2003. Note that the keywords may be keywords spoken by the customer or by the store clerk.

[0073] Here, an example of detecting a product from a conversation and displaying information about the product on the aerial display 22 will be described with reference to FIG. 11 .

[0074] 11 is an explanatory diagram showing an example of detecting a product from a conversation and displaying information about the product on the aerial display 22. In FIG. 11, the screen 2101 of the dome-shaped display 21 and the projection device 2102 are omitted from the illustration. For example, the recording device 24 records or collects audio. Then, the acquisition unit 204 acquires the audio from the recording device 24 in real time. Then, the audio analysis unit 206 detects information that can identify the product from the conversation by performing voice recognition on the conversation.

[0075] In Fig. 11 , for example, the product detection unit 202 detects the keyword "apple on the far right" from the conversation. Then, the second output control unit 203 displays information about the detected product on the aerial display 22. As a result, in Fig. 11 , the aerial display 22 displays information about the product, such as an image or video of the product. In the example of Fig. 11 , an apple is displayed.

[0076] Returning to the explanation of Fig. 7, the customer's behavior obtained from the voice is not limited to the conversation between the customer and the store clerk, but may also be, for example, the customer's utterance.

[0077] This concludes the description of the example of detecting a product. As described above, there are various methods for detecting a product, and these methods may be combined as appropriate. For example, the product detection unit 202 may detect a product that a customer wants to acquire based on the analyzed conversation and the analyzed eye movement.

[0078] <Example of Changing Orientation of Merchandise in Image or Video> Here, an example of changing the display orientation of a commodity in an image or video while the commodity image or video is being displayed on the aerial display 22 will be described.

[0079] For example, the video analysis unit 205 analyzes the video of the customer to detect the customer's hand movement relative to the aerial display 22. Specifically, the video analysis unit 205 may detect, for example, a hand movement such as turning a product. For example, the video analysis unit 205 may detect whether the hand movement is such as turning the product to the left or to the right.

[0080] Next, the shopping support system 20 performs control based on the hand movement so that the orientation of the product in the image or video displayed on the aerial display 22 is changed. There are various examples of control methods. Here, we will explain using a video of the product as an example.

[0081] First, an example will be described in which at least one of the imaging position and orientation of the imaging device 25 installed in the store is controlled based on hand movement. For example, if the image of a product is an image of the product captured in real time by the imaging device 25 installed in the store, the imaging device control unit 207 controls at least one of the position and orientation of the imaging device 25 based on the customer's hand movement. This allows the imaging device 25 to capture images of different directions of the product. More specifically, for example, if the imaging device control unit 207 detects a hand movement such as turning the product to the left, it controls at least one of the imaging position and orientation of the imaging device 25 so that the left side of the product currently being captured can be captured. The imaging position and orientation may be determined based on the hand movement. Note that in a store, products may be placed on a stand, and the imaging device 25 may capture images of the product placed on the stand.

[0082] Here, there is no particular limitation on the method for controlling the image capturing position and orientation of the image capturing device 25. For example, if the image capturing device 25 is attached to a robot arm, the image capturing device control unit 207 can control the image capturing position of the image capturing device 25 by controlling the robot arm.

[0083] 12 is an explanatory diagram showing an example of changing the orientation of a displayed product by changing the position and orientation of an imaging device 25 that captures an image of the product. First, in a store, imaging device 25 captures an image of a product placed on a stand 27. In Fig. 12, an apple is placed on stand 27 as a product.

[0084] Then, the second output control unit 203 displays the image of the product on the aerial display 22. In Fig. 12, the aerial display 22 displays an apple as the product.

[0085] In FIG. 12, the video analysis unit 205 analyzes the video of the customer captured by the imaging device 23 to analyze the hand movements of the customer.

[0086] 12, the imaging device control unit 207 controls the position and orientation of the imaging device 25 based on the movement of the customer's hand. In FIG. 12, the position and orientation of the imaging device 25 are changed. As a result, an image of the product with the changed orientation is displayed on the aerial display 22. For example, in FIG. 12, another part of the apple is displayed on the aerial display 22.

[0087] Returning to the explanation of Fig. 7 , for example, when the second output control unit 203 causes the aerial display 22 to display images of products captured in real time by an imaging device 25 selected from a plurality of imaging devices 25 installed in a store, a new imaging device 25 is selected from the plurality of imaging devices 25 based on hand movements. Then, the second output control unit 203 causes the image captured by the newly selected imaging device 25 to be displayed on the aerial display 22. In this way, by switching the imaging device 25, the orientation of the products displayed on the aerial display 22 can be changed.

[0088] 13 is an explanatory diagram showing an example in which the orientation of a displayed commodity is changed by switching the imaging device 25. First, imaging devices 25-1 and 25-2 are installed in a store. Imaging devices 25-1 and 25-2 capture images of a commodity placed on a stand 27 in the store. In FIG. 13, an apple is placed on stand 27 as a commodity.

[0089] For example, the second output control unit 203 displays an image of a commodity captured by the imaging device 25-1 on the aerial display 22. In Fig. 13, an apple is displayed as a commodity on the aerial display 22.

[0090] In FIG. 13, the video analysis unit 205 analyzes the video of the customer captured by the imaging device 23 to analyze the hand movements of the customer.

[0091] 13, the second output control unit 203 selects the image capturing device 25-2 from the image capturing devices 25-1 and 25-2 based on the customer's hand movement, and displays the image captured by the selected image capturing device 25-2. As a result, the image of the product with its orientation changed is displayed on the aerial display 22. For example, in FIG. 13, other parts of the apple are displayed on the aerial display 22.

[0092] Returning to the description of FIG. 7 , for example, when the image of a product is an image of the product captured in real time by the imaging device 25 installed in the store, the product to be displayed on the aerial display 22 may be placed on a platform 27 by a store clerk, and the imaging device 25 may capture an image of the product placed on the platform 27. For example, the platform 27 may be rotatable. In such a case, the platform control unit 208 controls the rotation of the platform 27. This allows the orientation of the captured product to be changed. The platform control unit 208 controls the rotation of the platform 27 based on the customer's hand movement. This allows the imaging device 25 to capture images of different orientations of the product. More specifically, for example, when a hand movement to rotate the product to the left is detected, the imaging device control unit 207 rotates the platform 27 to the left so that the left side of the currently captured product can be captured. The amount of rotation may be determined according to the hand movement. Furthermore, in addition to the example of rotation of the platform 27, the platform control unit 208 may also control the height of the platform 27. If the position of the platform 27 is changeable, the platform control unit 208 changes the position of the platform 27. As a result, the table control unit 208 can control the height, position, and rotation of the table 27, thereby changing the orientation of the product imaged by the imaging device 25, etc.

[0093] Fig. 14 is an explanatory diagram showing an example in which the orientation of a displayed product is changed by rotating the stand 27. First, the imaging device 25 captures an image of a product placed on the stand 27 in a store. In Fig. 14, an apple is placed on the stand 27 as a product.

[0094] Then, the second output control unit 203 displays the image of the product on the aerial display 22. In Fig. 14, an apple is displayed as the product on the aerial display 22.

[0095] In FIG. 14, the video analysis unit 205 analyzes the video of the customer captured by the imaging device 23 to analyze the hand movements of the customer.

[0096] 14, the table control unit 208 rotates the table 27 based on the movement of the customer's hand. As a result, an image of the product with its orientation changed is displayed on the aerial display 22. For example, in FIG. 14, another part of the apple is displayed on the aerial display 22.

[0097] Returning to the description of Fig. 7, for example, if the image of the product is an image captured in advance, the second output control unit 203 displays the image of the product captured in advance such that the orientation of the product corresponds to the movement of the hand.

[0098] In the above example, an example has been described in which control is performed to change the orientation of a product in the image displayed on the aerial display 22 based on the customer's hand movement, but control may also be performed to change the orientation of a product in the image displayed on the aerial display 22. For example, first, the second output control unit 203 displays one of multiple images of the product captured from different orientations. Then, when the customer's hand movement is detected, the second output control unit 203 displays one of the multiple images captured with an orientation that corresponds to the customer's hand movement.

[0099] Furthermore, when a new image of a product to be displayed is captured, as in the example of video, at least one of the image capturing position and orientation of the image capturing device 25 may be controlled by the image capturing device control unit 207. For example, the image capturing device control unit 207 controls at least one of the position and orientation of the image capturing device 25, and after the position and orientation are controlled, the image capturing device 25 captures a new image of the product. Then, the second output control unit 203 displays the newly captured image of the product.

[0100] Furthermore, when a new image of a product to be displayed is captured, similar to the example of video, the second output control unit 203 may switch the imaging device 25. When the second output control unit 203 is to display on the aerial display 22 an image of a product captured by an imaging device 25 selected from multiple imaging devices 25 installed in the store, a new imaging device 25 is selected from the multiple imaging devices 25 based on hand movements. The newly selected imaging device 25 then captures the image of the product. The second output control unit 203 then displays on the aerial display 22 the image captured by the newly selected imaging device 25.

[0101] Furthermore, when a new image of a product to be displayed is captured, the platform 27 may be controlled by the platform control unit 208, as in the example of video. The platform control unit 208 controls the rotation of the platform 27 based on the movement of the customer's hand. The imaging device 25 then captures a new image of the product in a different orientation. The second output control unit 203 then displays the new image captured by the imaging device 25 on the aerial display 22.

[0102] Furthermore, in the above example, an example was described in which control was performed to change the orientation of the product in the image displayed on the aerial display 22 based on the customer's hand movements, but control may also be performed to change the orientation of the product in the image or picture displayed on the aerial display 22 in accordance with operations on the aerial display 22, not limited to hand movements.

[0103] For example, the aerial display 22 can receive operations from a user. Therefore, the operation receiving unit 211 may receive operations on the aerial display 22.

[0104] Then, similar to the example of hand movements, the imaging device control unit 207 may control at least one of the imaging position and orientation of the imaging device 25 based on the received operation. Alternatively, similar to the example of hand movements, the second output control unit 203 selects a new imaging device 25 from the multiple imaging devices 25 based on the hand movement based on the received operation. Then, the second output control unit 203 displays the image captured by the newly selected imaging device 25 on the aerial display 22. Alternatively, similar to the example of hand movements, the platform control unit 208 controls the rotation of the platform 27 based on the received operation. Alternatively, similar to the example of hand movements, if the image of the product is an image captured in advance, the first output control unit 201 displays the image from the images captured in advance such that the image of the product is oriented in accordance with the received operation.

[0105] Note that specific examples of control of each functional unit based on the received operation are similar to the example of hand movements, and therefore detailed explanations will be omitted.

[0106] <Controlling the display of the dome-shaped display 21 while the aerial display 22 is displaying information about a product> For example, the background may be visible through the aerial display 22. When a customer looks at a product displayed by the aerial display 22, the dome-shaped display 21 is in the customer's line of sight. The customer sees the display on the dome-shaped display 21 behind the product and beyond the product through the aerial display 22. In this way, while information about a product is being displayed on the aerial display 22, the display on the dome-shaped display 21 may be difficult for the customer to see.

[0107] Therefore, the first output control unit 201 changes the display of the dome-shaped display 21 so that the products displayed on the aerial display 22 are not difficult to see. This makes it possible to prevent the products from becoming difficult to see when the aerial display 22 and the dome-shaped display 21 are combined. To change the display of the dome-shaped display 21, the first output control unit 201 may change the display of the entire dome-shaped display 21, or may change the display of a portion of the dome-shaped display 21. As an example of changing the entire display, the first output control unit 201 may turn off the display of the video of the sales floor, or display a video corresponding to the product instead of the video of the sales floor. For example, if the product is a food product, the video corresponding to the product may be a plate, etc.

[0108] Furthermore, when changing the display of a portion of the dome-shaped display 21, the first output control unit 201 may erase or change the color of the display of a portion of the dome-shaped display 21 that overlaps with the product when the customer views the product in the video or image displayed on the aerial display 22. This makes it possible to prevent the product from becoming difficult to see when the aerial display 22 and the dome-shaped display 21 are combined.

[0109] As an example of changing a portion of the display, for example, while the aerial display 22 is displaying information about a product, the first output control unit 201 changes the display at a fixed position on the dome-shaped display 21. The fixed position is a position corresponding to the display area where the aerial display 22 displays information about the product. As a method of changing the display at the fixed position, the first output control unit 201 may erase the display at the fixed position. Alternatively, the first output control unit 201 may change the color of the display at the fixed position to a predetermined color. The predetermined color may be a color that makes the product easy to see, such as black or white, or a color that corresponds to the product.

[0110] Although a fixed position has been used as an example, the position may be specified in more detail. The superimposition position specifying unit 209 specifies the relative position of the dome-shaped display 21 that overlaps with the display area in which the aerial display 22 displays product-related information when the customer views the product-related information displayed in the display area, based on the positional relationship between the customer's position and the aerial display 22, and the positional relationship between the display area in which the aerial display 22 displays product-related information and the dome-shaped display 21. For example, the display area is the aerial imaging area described in FIG. 6 . The customer's position may be specified using a three-dimensional (3D) sensor or the like. The 3D sensor may be installed on the dome-shaped display 21, for example. While the aerial display 22 is displaying product-related information, the first output control unit 201 changes the display of the specified position on the dome-shaped display 21. The display change method is as described above.

[0111] Alternatively, while the aerial display 22 is displaying information about a product, the first output control unit 201 causes the dome-shaped display 21 to display an image different from the image of the sales floor. For example, the image may be different depending on the product, such as a plate or another shelf. Here, an image is used as an example, but an image may also be used.

[0112] Although an example in which one customer is present in one dome-type display 21 has been given here, multiple customers may be shopping together. When multiple customers are present in the dome-type display 21, the first output control unit 201 may control the display on the dome-type display 21 for each of the multiple customers.

[0113] In addition, when multiple aerial displays 22 are installed, the first output control unit 201 may control the display of the dome-shaped display 21 for each of the multiple aerial displays 22.

[0114] This concludes the description of the example of controlling the display on the dome-shaped display 21 while information about a product is being displayed on the aerial display 22.

[0115] <Control of Display Size and Color of Merchandise in Images and Videos> When an image or video of a product is displayed on the aerial display 22, the size of the product may appear different depending on the position from which the customer views the displayed product.

[0116] Therefore, the second output control unit 203 may display a measuring tape that allows the dimensions to be checked together with the image of the product on the aerial display 22. This allows the customer to check the actual size of the product by comparing the size of the product with the measuring tape.

[0117] Furthermore, the customer's viewing position may be identified so that a life-size image or video of the product can be displayed to the customer. For example, a 3D sensor may be installed on the dome-shaped display 21. The 3D sensor measures the position of the customer's head and hands. The acquisition unit 204 then acquires the measurement results of the 3D sensor. The viewing position identification unit 210 identifies the customer's viewing position based on the positional relationship between the head position and the hand position based on the measurement results. The second output control unit 203 displays either an image or video of the product at a size based on the life-size size of the product and the viewing position. As a result, the image or video of the product is displayed so that the product appears life-size.

[0118] Furthermore, the second output control unit 203 may cause the aerial display 22 to display an enlarged version of the product. Alternatively, the second output control unit 203 may cause the aerial display 22 to display a reduced version of the product. Here, the operation receiving unit 211 may receive an instruction to enlarge or reduce the product by receiving an operation on the aerial display 22. Then, based on the received operation, the second output control unit 203 may cause the product in the image or video to be displayed enlarged or reduced.

[0119] Furthermore, for example, when products are fresh foods such as vegetables and fruits, it is expected that the color and shape of each product will vary. For example, when images such as product illustrations or images of products reproduced in a virtual space are displayed on the aerial display 22, if the same image is displayed each time, it is expected that the customer will feel that the shopping experience is different from a realistic one. Here, product images are used as an example, but the same applies to product videos. It is desirable to achieve a more realistic shopping experience. Therefore, for example, the second output control unit 203 displays product images on the aerial display 22 based on at least one of information regarding the product's color and shape. The product's color and shape may be obtained from a conversation between a store clerk and a customer. For example, the voice analysis unit 206 detects at least one of information regarding the product's color and information regarding the product's shape by performing voice recognition on the conversation between the store clerk and the customer. Then, the second output control unit 203 displays the product images in at least one of the colors and shapes corresponding to the conversation. Specifically, for example, when a conversation between a store clerk and a customer such as "Today's apple is redder than usual" is detected, the second output control unit 203 displays the image of the apple in a color redder than a predetermined standard. For example, when a conversation between a store clerk and a customer such as "Today's apple is bigger than usual" is detected, the second output control unit 203 displays the image of the apple so that the apple appears to be a predetermined size larger than the predetermined standard. This allows the image of the product to be displayed on the aerial display 22 in a color and shape that is closer to the actual product.

[0120] However, if any of the images and videos of all individual products is available, the second output control unit 203 may display any of the images and videos corresponding to the individual products.

[0121] Furthermore, for example, when video of an actual store is displayed in real time, if a product shelf tag or the like is captured together with the product, the customer can check the product price and name. However, if there is no shelf tag or if the shelf tag is not captured, the second output control unit 203 may display the product name and price on the aerial display 22 together with the video or image of the product.

[0122] <Example of Ending Display of Product-Related Information on Aerial Display 22> The timing of ending display of product-related information on the aerial display 22 can be changed in various ways depending on how the application program is used.

[0123] For example, when the customer is interested in another product or a store clerk, the display of product-related information on the aerial display 22 may be terminated. For example, the video analysis unit 205 may detect the customer's line of sight from the video of the customer. For example, the video analysis unit 205 detects that the customer's eyes are looking up and not looking at the aerial display 22. Then, when the second output control unit 203 detects that the customer is not looking at the aerial display 22, it terminates the display of product-related information on the aerial display 22. Furthermore, when the product detection unit 202 detects a new product, such as when the customer reaches out for another product, the second output control unit 203 terminates the display of product-related information on the aerial display 22. However, in this case, the second output control unit 203 may cause the aerial display 22 to display information about the newly detected product.

[0124] For example, the display of product information on the aerial display 22 may be terminated by a customer operation. For example, the customer operation may be an operation indicating the end of the display, or an operation to add the product to a shopping cart. The customer operation can be detected by the customer's voice, gestures, or the like. Specifically, for example, the video analysis unit 205 detects a predetermined gesture of the customer from a video of the customer. Then, when the predetermined gesture is detected, the second output control unit 203 terminates the display of product information on the aerial display 22. Alternatively, for example, the audio analysis unit 206 detects a predetermined keyword from the customer's voice. Then, when the predetermined keyword is detected, the second output control unit 203 terminates the display of product information on the aerial display 22.

[0125] Furthermore, when a customer leaves the dome-type display 21, the display of product-related information on the aerial display 22 may be terminated. A customer leaving the dome-type display 21 means, for example, that the customer leaves a location where the customer can view or operate the dome-type display 21. Specifically, a customer leaving the dome-type display 21 may mean, for example, that the customer leaves a chair placed in front of the dome-type display 21. For example, the video analysis unit 205 may detect that a customer has left by detecting whether the customer has been captured on video captured by the imaging device 23. Then, when it is detected that the customer has left, the second output control unit 203 terminates the display of product-related information on the aerial display 22.

[0126] 15 is a flowchart showing an example of the operation of the shopping support system 20 according to the second embodiment. Although not specifically shown in the flowchart, the acquisition unit 204 acquires video captured by the imaging device 23 and audio captured by the recording device 24. The video analysis unit 205 analyzes customer behavior from the video captured by the imaging device 23. The audio analysis unit 206 analyzes customer behavior from the audio captured by the recording device 24.

[0127] The first output control unit 201 causes the dome display 21 to display information about the sales floor of the store (step S201).

[0128] Next, the product detection unit 202 determines whether the product desired by the customer has been detected based on the customer's behavior (step S202). If the product has not been detected (step S202: No), the product detection unit 202 returns to step S202.

[0129] On the other hand, if a product is detected (step S202: Yes), the second output control unit 203 causes the aerial display 22 to display information about the product (step S203). The first output control unit 201 also changes the display on the dome-shaped display 21 (step S204). In step S204, the first output control unit 201 may change the display of a fixed position on the dome-shaped display 21, or may change the display of a specified position on the dome-shaped display 21. The order of the processing of steps S203 and S204 is not particularly limited. For example, steps S203 and S204 may be performed at the same time.

[0130] Next, the second output control unit 203 changes the display of information about the products based on the customer's behavior (step S205). In step S205, the second output control unit 203 may change the display orientation of the products in the video or image based on the customer's behavior. Furthermore, step S205 is not limited to control by the second output control unit 203; control of the platform 27 by the platform control unit 208 or control of the imaging device 25 by the imaging device control unit 207 may also be performed. For example, the imaging device control unit 207 may change the orientation of the products displayed on the aerial display 22 by changing the imaging position or orientation of the imaging device 25. Alternatively, for example, the platform control unit 208 may control the platform 27 to change the orientation of the products displayed on the aerial display 22. Note that the control of the display by the second output control unit 203, the control of the platform 27 by the platform control unit 208, and the control of the imaging device 25 by the imaging device control unit 207 may be combined as appropriate.

[0131] The second output control unit 203 determines whether to end the display of the information about the product (step S206). If the display of the information about the product is not to be ended (step S206: No), the second output control unit 203 returns to step S205. If the display of the information about the product is to be ended (step S206: Yes), the first output control unit 201 returns to step S201.

[0132] The flowchart may be terminated as appropriate.

[0133] As described above, in the second embodiment, the shopping support system 20 detects the customer's hand movements from a video or image of the customer and detects the product the customer wants to acquire based on the detected hand movements. More specifically, for example, the shopping support system 20 detects the product the customer wants to acquire based on the position of the customer's outstretched hand. For example, the customer can simply make an intuitive movement, such as picking up a product from a shelf, and the product the customer wants to check will be displayed on the aerial display 22. Furthermore, when a customer visits a real store to shop, it is expected that the customer will grab a product from a shelf and bring it close to their face to check it. Therefore, the shopping support system 20, in which information about the product detected from hand movements is displayed on the aerial display 22, can provide a shopping experience that is closer to actual shopping.

[0134] Furthermore, the shopping support system 20 analyzes the conversation between the customer and the store clerk from the voice and detects the product that the customer wants to purchase based on the conversation. As a result, the customer only needs to talk to the store clerk, and information about the product that the customer wants to check is displayed on the aerial display 22. Furthermore, for example, when a customer visits a real store to shop, it is expected that the customer will talk to the store clerk, and the store clerk will recommend and show products to the customer, or hand the products over to the customer. Therefore, the shopping support system 20, in which information about products detected from the conversation is displayed on the aerial display 22, can provide a shopping experience that is closer to actual shopping.

[0135] Furthermore, the shopping support system 20 displays an image or video of the product in at least one of a color and a shape according to the conversation on the aerial display 22. This allows the image or video of the product to be displayed on the aerial display 22 in a color and shape that is close to the actual product.

[0136] The shopping support system 20 also displays information about store staff on the dome display 21. This allows customers to shop in a manner that is closer to that of an actual store.

[0137] Furthermore, the shopping support system 20 changes the display of products in the video based on the customer's hand movements. For example, the shopping support system 20 controls the imaging device 25 installed in the store based on the customer's hand movements. When products are placed on a rotating platform 27 and the imaging device 25 captures an image of the product placed on the platform 27, the shopping support system 20 controls the rotation of the platform 27 based on the customer's hand movements. This allows the customer to change the display orientation of the product by moving their hand. In this way, the orientation of the product can be intuitively changed.

[0138] Furthermore, as described above, the product-related information displayed on the aerial display 22 is transparent. This can make it difficult for customers to see the products. Therefore, the shopping support system 20 changes the display on the dome-shaped display 21 while the aerial display 22 is displaying product-related information. Specifically, for example, the shopping support system 20 may change the display at a fixed position on the dome-shaped display 21. Alternatively, the shopping support system 20 may identify a position on the dome-shaped display 21 that is in an extension of the customer's line of sight when the customer views the product, and change the display at the identified position. This can prevent the customer from having difficulty seeing the product.

[0139] The store may be a store in a virtual space or a real store. If there is a real store, the shopping support system 20 may display an image of the store's sales floor captured in real time by the imaging device 25 on the dome display 21. This allows customers to have the experience of being in the real store even when they are not actually in the store.

[0140] This concludes the description of each embodiment. Each embodiment may be modified or used in combination as appropriate. In each embodiment, the shopping support system 20 may be configured to include some of the functional units and information.

[0141] Furthermore, the embodiments are not limited to the above examples and can be modified in various ways. Furthermore, the configuration of the shopping support system 20 in each embodiment is not particularly limited. For example, the shopping support system 20 may be implemented by a single device, such as a single server. When each functional unit of the shopping support system 20 is implemented by a single device, the single device may be called, for example, a shopping support device, an information processing device, or the like, and is not particularly limited. Alternatively, the shopping support system 20 in each embodiment may be implemented by different devices for different functions or data. For example, each functional unit of the shopping support system 20 may be configured by multiple servers, thereby realizing a shopping support system 20 including multiple servers. For example, the shopping support system 20 may be realized by a database server including each DB and a server having each functional unit.

[0142] 16 is an explanatory diagram showing an example of implementation of the shopping support system 20. The following describes an example in which there is an actual store.

[0143] In FIG. 16, the shopping support system 20 includes, for example, an edge terminal device 31 and a server 32 .

[0144] For example, the edge terminal device 31, the aerial display 22, the dome-shaped display 21, the imaging device 23, and the sound recording device 24 are installed in a house, a shared space, etc. The edge terminal device 31, the aerial display 22, the dome-shaped display 21, the imaging device 23, and the sound recording device 24 are connected via a communication network.

[0145] For example, the imaging device 25 is installed in a store. The imaging device 25, the edge terminal device 31, and the server 32 are connected via a communication network.

[0146] In addition, the shopping support system 20 may be configured as an entire system including an edge terminal device 31 , a server 32 , an aerial display 22 , a dome-shaped display 21 , an imaging device 23 , a sound recording device 24 , and an imaging device 25 .

[0147] 16 , for example, each functional unit in each embodiment is realized by an edge terminal device 31 and a server 32. For example, the edge terminal device 31 includes an acquisition unit 204, a video analysis unit 205, an audio analysis unit 206, and an operation reception unit 211. For example, because analyzing video and audio imposes a large load, the edge terminal device may be configured to perform these analyses. On the other hand, for example, the server 32 includes a first output control unit 201, a product detection unit 202, a second output control unit 203, an acquisition unit 204, an imaging device control unit 207, a platform control unit 208, a superimposition position identification unit 209, and a viewing position identification unit 210. Furthermore, the server 32 may be a plurality of servers. In this way, each functional unit of the shopping support systems 10 and 20 may be realized by a plurality of devices, and the plurality of devices may be installed in different locations.

[0148] In each embodiment, each piece of information or each DB may include a portion of the information described above. Each piece of information or each DB may also include information other than the information described above. Each piece of information or each DB may be divided into multiple DBs or multiple pieces of information in more detail. Thus, the method for realizing each piece of information or each DB is not particularly limited.

[0149] Furthermore, each screen is merely an example and is not particularly limited. Buttons, lists, check boxes, information display fields, input fields, etc. (not shown) may be added to each screen. Furthermore, the background color of the screen may be changed.

[0150] Furthermore, for example, in each embodiment, the process of generating information to be displayed on the aerial display 22 may be performed by the second output control unit 203 of the shopping support system 20. This process may also be performed by the aerial display 22.

[0151] Furthermore, for example, in each embodiment, the process of generating information to be displayed on the dome-shaped display 21 may be performed by the first output control unit 201 of the shopping support system 20. This process may also be performed by the dome-shaped display 21.

[0152] Furthermore, in each embodiment, a display in which at least a portion of the screen is curved has been given as an example of the first display, and a dome-shaped display 21 has been given as a specific example of the first display. The first display is not limited to a display in which at least a portion of the screen is curved, as long as it is a display intended to display a virtual space, video, images, etc. Alternatively, for example, as described above, the first display may be a device configured to cover the user's field of vision.

[0153] In addition, in each embodiment, a non-contact type display has been described as an example of the second display. As described above, the second display may be a contact type display, and is not particularly limited.

[0154] (Example of Computer Hardware Configuration) Next, an example of a hardware configuration when each device of the shopping support systems 10, 20, etc. described in each embodiment is implemented by a computer will be described. Fig. 17 is an explanatory diagram showing an example of a computer hardware configuration. For example, some or all of each device can be implemented using any combination of a computer 80 and a program as shown in Fig. 17.

[0155] The computer 80 includes, for example, a processor 801, a ROM (Read Only Memory) 802, a RAM (Random Access Memory) 803, and a storage device 804. The computer 80 also includes a communication interface 805 and an input / output interface 806. The components are connected to each other, for example, via a bus 807. The number of each component is not particularly limited, and there may be one or more of each component.

[0156] The processor 801 controls the entire computer 80. Examples of the processor 801 include a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). The computer 80 includes a storage unit such as a ROM 802, a RAM 803, and a storage device 804. Examples of the storage device 804 include a semiconductor memory such as a flash memory, a hard disk drive (HDD), and a solid state drive (SSD). For example, the storage device 804 stores an operating system (OS) program, application programs, and programs according to the embodiments. Alternatively, the ROM 802 stores application programs and programs according to the embodiments. The RAM 803 is used as a work area for the processor 801 .

[0157] The processor 801 also loads programs stored in the storage device 804, ROM 802, etc. The processor 801 then executes each process coded in the program. The processor 801 may also download various programs via the communication network NT. The processor 801 also functions as a part or all of the computer 80. The processor 801 may then execute the processes or instructions in the illustrated flowchart based on the program.

[0158] The communication interface 805 is connected to a communication network NT such as a LAN (Local Area Network) or a WAN (Wide Area Network) via a wireless or wired communication line. The communication network NT may be composed of multiple communication networks NT. As a result, the computer 80 is connected to external devices and external computers 80 via the communication networks NT. The communication interface 805 serves as an interface between the communication network NT and the inside of the computer 80. The communication interface 805 also controls the input and output of data from external devices and external computers 80.

[0159] Furthermore, the input / output interface 806 is connected to at least one of an input device, an output device, and an input / output device. The connection method may be wireless or wired. Examples of the input device include a keyboard, a mouse, and a microphone. Examples of the output device include a display device, a lighting device, and an audio output device that outputs audio. Examples of the input / output device include a touch panel display. Note that the input device, output device, and input / output device may be built into the computer 80 or may be external.

[0160] The hardware configuration of the computer 80 is an example. The computer 80 may have some of the components shown in FIG. 17 . The computer 80 may have components other than those shown in FIG. 17 . For example, the computer 80 may have a drive device or the like. The processor 801 may then read programs and data stored on a recording medium attached to the drive device or the like into the RAM 803. Examples of non-transitory tangible recording media include optical disks, flexible disks, magneto-optical disks, and USB (Universal Serial Bus) memories. As described above, the computer 80 may have input devices such as a keyboard and a mouse. The computer 80 may have an output device such as a display. The computer 80 may also have an input device, an output device, and an input / output device.

[0161] The computer 80 may also include various sensors (not shown). The types of sensors are not particularly limited. The computer 80 may also include an imaging device capable of capturing images or videos.

[0162] This concludes the description of the hardware configuration of each device. There are various variations in the method of realizing each device. For example, each device may be realized by any combination of a different computer and program for each component. Furthermore, multiple components of each device may be realized by any combination of a single computer and program.

[0163] Furthermore, some or all of the components of each device may be realized by circuits for specific applications. Furthermore, some or all of the components of each device may be realized by general-purpose circuits including a processor such as an FPGA (Field Programmable Gate Array). Furthermore, some or all of the components of each device may be realized by a combination of circuits for specific applications and general-purpose circuits. Furthermore, these circuits may be a single integrated circuit. Alternatively, these circuits may be divided into multiple integrated circuits. The multiple integrated circuits may be connected via a bus or the like.

[0164] Furthermore, when some or all of the components of each device are realized by a plurality of computers, circuits, etc., the plurality of computers, circuits, etc. may be centrally located or distributed.

[0165] The shopping support method described in each embodiment is realized by being executed by a shopping support system. Furthermore, for example, the shopping support method is realized by a computer such as a server or a terminal device executing a prepared program. The program described in each embodiment is recorded on a computer-readable recording medium such as an HDD, SSD, flexible disk, optical disk, magnetic optical disk, or USB memory. The program is then read from the recording medium and executed by the computer. Furthermore, the program may be distributed via a communications network NT.

[0166] Each component of the shopping support system in each embodiment described above may be realized by dedicated hardware, such as a computer. Alternatively, each component may be realized by software. Alternatively, each component may be realized by a combination of hardware and software.

[0167] Although the present disclosure has been described above with reference to various embodiments, the present disclosure is not limited to the above embodiments. The configuration and details of each of the present disclosures may include embodiments to which various modifications that would be apparent to those skilled in the art are applied within the scope of the present disclosure. The present disclosure may also include embodiments in which the details described herein are appropriately combined or substituted as necessary. For example, details described using a particular embodiment may also be applied to other embodiments to the extent that no contradiction occurs. For example, although multiple operations are described in sequence in the form of a flowchart, the order of the descriptions does not limit the order in which the multiple operations are performed. Therefore, when implementing each embodiment, the order of the multiple operations may be changed as long as it does not interfere with the content.

[0168] Some or all of the above-described embodiments can be described as follows: However, some or all of the above-described embodiments are not limited to the following.

[0169] (Supplementary Note 1) A shopping support system comprising: first output control means for causing a first display to display information about a sales floor of a store where products are displayed; product detection means for detecting a product that a customer wishes to acquire based on the customer's behavior; and second output control means for causing a second display to display information about the detected product. (Supplementary Note 2) The shopping support system described in Supplementary Note 1, wherein the customer's behavior is the movement of the customer's hand. (Supplementary Note 3) The shopping support system described in Supplementary Note 2, wherein the product detection means detects the product based on the position at which the customer extends their hand. (Supplementary Note 4) The shopping support system described in any of Supplements 1 to 3, wherein the customer's behavior is a conversation between the customer and a store clerk. (Supplementary Note 5) The shopping support system described in Supplementary Note 4, wherein the customer's behavior is a conversation between the customer and a store clerk. (Supplementary Note 6) The shopping support system of any of Supplements 1 to 5, wherein the information about the product is a video or an image of the product, and the second output control means displays the information about the product in at least one of a color and a shape that corresponds to the conversation between the customer and a store clerk. (Supplementary Note 7) The shopping support system of any of Supplements 1 to 5, wherein the information about the product is a video of the product. (Supplementary Note 8) The shopping support system of Supplementary Note 7, wherein the second output control means changes the display of the product in the video of the product based on the movement of the customer's hand. (Supplementary Note 9) The shopping support system of Supplementary Note 7, wherein the video of the product is a video of the product captured by an imaging device installed in the store, and includes imaging device control means that controls at least one of an imaging position and an imaging direction of the imaging device based on the movement of the customer's hand. (Supplementary Note 10) The shopping support system described in Supplementary Note 7 includes: the image of the product is an image of the product captured by an imaging device installed in the store; and a platform control means for controlling the rotation of a rotating platform on which the product can be placed in the store based on the customer's hand movements.(Supplementary Note 11) The shopping support system of any of Supplements 1 to 10, wherein the first output control means changes the display of a fixed position on the first display when the second display displays the information about the product. (Supplementary Note 12) The shopping support system of any of Supplements 1 to 10, further comprising: an identification means for identifying a position on the first display that overlaps with a display area when the customer views information about the product displayed in the display area, based on a positional relationship between the customer's position and the position of the second display, and a positional relationship between a display area of ​​the information about the product on the second display and the position of the first display, wherein the first output control means changes the display of the identified position on the first display when the second display displays the information about the product. (Supplementary Note 13) The shopping support system of any of Supplements 1 to 12, wherein the store is a store in a virtual space. (Supplementary Note 14) The shopping support system of any of Supplements 1 to 12, wherein the information about the sales floor is an image of the sales floor of the store captured by an imaging device installed in the store. (Supplementary Note 15) The shopping support system of any of Supplements 1 to 14, wherein the first display is a device structured to cover the customer's field of vision. (Supplementary Note 16) The shopping support system of any of Supplements 1 to 15, wherein the first display is a dome-shaped display. (Supplementary Note 17) The shopping support system of any of Supplements 1 to 16, wherein the second display is a contactless display. (Supplementary Note 18) The shopping support system of Supplementary Note 17, wherein the second display is an aerial display. (Supplementary Note 19) The shopping support system of any of Supplements 1 to 18, wherein the second display is installed between the customer and the first display. (Supplementary Note 20) The shopping support system of any of Supplements 1 to 18, wherein the second display is installed closer to the customer than the first display. (Supplementary Note 21) The shopping support system of any of Supplements 1 to 20, wherein the first display is larger than the second display.(Supplementary Note 22) A shopping support method comprising: displaying, on a first display, information about a sales floor in a store where products are displayed; detecting products that the customer wants to acquire based on customer behavior; and displaying information about the detected products on a second display. (Supplementary Note 23) A computer-readable non-transitory recording medium having recorded thereon a program that causes a computer to execute the following processes: displaying, on a first display, information about the sales floor in a store where products are displayed; detecting products that the customer wants to acquire based on customer behavior; and displaying information about the detected products on a second display. (Supplementary Note 24) A program that causes a computer to execute the following processes: displaying, on a first display, information about the sales floor in a store where products are displayed; detecting products that the customer wants to acquire based on customer behavior; and displaying information about the detected products on a second display.

[0170] 10, 20 Shopping support system 21 Dome-shaped display 22 Aerial display 23 Imaging device 24 Sound recording device 25, 25-1, 25-2 Imaging device 26 Sound recording device 27 Unit 31 Edge terminal device 32 Server 80 Computer 101, 201 First output control unit 102, 202 Product detection unit 103, 203 Second output control unit 204 Acquisition unit 205 Video analysis unit 206 Audio analysis unit 207 Imaging device control unit 208 Unit control unit 209 Superimposition position identification unit 210 Viewing position identification unit 211 Operation reception unit 801 Processor 802 ROM 803 RAM 804 Storage device 805 Communication interface 806 Input / output interface 807 Bus 2101 Screen 2102 Projection device 2103 Table NT Communication network 2001 Customer DB 2002 Store DB 2003 Product DB

Claims

1. a first output control means for causing a first display to display information about a sales floor of the store in which the product is displayed; A product detection means for detecting a product that the customer wants to acquire based on the customer's behavior; A second output control means for causing a second display to display information relating to the detected product; A shopping support system comprising:

2. The customer's behavior is the customer's hand movement. The shopping support system according to claim 1 .

3. the commodity detection means detects the commodity based on the position where the customer extends his / her hand; The shopping support system according to claim 2 .

4. The customer's behavior is a conversation between the customer and a store clerk.

4. The shopping support system according to claim 1.

5. The first output control means further causes the first display to display information about the store clerk. The shopping support system according to claim 4.

6. The information about the product is a video of the product or an image of the product, the second output control means displays the information about the product in at least one of a color and a shape according to the conversation between the customer and the store clerk; 4. The shopping support system according to claim 1.

7. The information about the product is a picture of the product.

4. The shopping support system according to claim 1.

8. The second output control means changes the display of the product in the image of the product based on the movement of the customer's hand. The shopping support system according to claim 7.

9. displaying on the first display information relating to the sales area of ​​the store in which the product is displayed; Detecting products that the customer wants to acquire based on the customer's behavior; Displaying information about the detected product on a second display; How to help with shopping.

10. On the computer, displaying on the first display information relating to the sales area of ​​the store in which the product is displayed; Detecting products that the customer wants to acquire based on the customer's behavior; Displaying information about the detected product on a second display; A program that executes a process.