Control apparatus, control method, and program

Abstract

To provide more appropriately a state of a viewer in real time against a distributor who distributes an image.SOLUTION: A control apparatus controls a first display device of a distributor who can view a specific article via the first display device, and distributes an image obtained by imaging the specific article. The control apparatus includes: acquisition means for acquiring a position where each of N (N≥1) users who view the image of the specific article, and each of the N users having a specific attribute views the specific article; generation means for generating a first display item indicating the position where each of the N users in the specific article views it; and control means for displaying the first display item on the first display device so that a position indicated by the first display item in the specific article and the first display item are seen in an overlapped manner.SELECTED DRAWING: Figure 2B

Description

[Technical field]

[0001] The present invention relates to a control device, a control method, and a program. [Background technology]

[0002] In recent years, an electronic commerce (EC) model known as "live commerce" has become popular, where streamers and viewers can purchase products while engaging in two-way communication. In live commerce, streamers deliver live videos explaining products, and viewers can ask questions and make comments in real time. Streamers can also understand viewers' reactions to products by referring to questions and comments from viewers, as well as product purchase status.

[0003] In Patent Document 1, the system displays information about the product purchasing status of each viewer on the display screen of a distributor who distributes real-time video related to a product. The system changes the display of the avatar of a viewer who has purchased a product among the avatars linked to each viewer, so that the distributor can grasp the purchasing status of each viewer.

[0004] In Patent Document 2, a video distribution system acquires reaction information indicating the reactions of multiple viewers during distribution of video data stored in a video library, and determines a second video data to be distributed after a first video data based on the reaction information. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2016-197411 A [Patent Document 2] JP 2020-120303 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology of Patent Document 1 only distinguishes whether the viewer is a non-purchaser or a purchaser from the display of the avatar. The technology of Patent Document 2 cannot reflect the viewer's reaction in live video such as live commerce. Therefore, neither technology can provide the viewer's real-time status to the distributor who distributes the image.

[0007] Therefore, an object of the present invention is to provide a technique that enables a distributor distributing images to more appropriately provide real-time information on the state of viewers. [Means for solving the problem]

[0008] One aspect of the present invention is a method for producing a composition comprising the steps of: A control device for controlling a first display device of a distributor who is capable of viewing a specific object through the first display device and who distributes an image capturing the specific object, An acquisition means for acquiring a position where each of N users (N≧1) who view an image of the specific object and have a specific attribute is viewing the specific object; a generating means for generating a first display item indicating a position on the particular object at which each of the N users is looking; A control is performed to display the first display item on the first display device so that the first display item appears to overlap with a position indicated by the first display item on the specific object. With your means, The control device is characterized by having the following features.

[0009] One aspect of the present invention is a method for producing a composition comprising the steps of: A control method for controlling a first display device of a distributor who is able to view a specific object through a first display device and who distributes an image capturing the specific object, comprising: An acquisition step of acquiring a position where each of N users (N≧1) who view an image of the specific object and have a specific attribute is looking at the specific object; A generating step of generating a first display item indicating a position at which each of the N users is looking at the particular object; a control step of displaying the first display item on the first display device so that the first display item appears to overlap a position on the specific object indicated by the first display item; The control method is characterized by having the following features. Effect of the Invention

[0010] According to the present invention, it is possible to provide a distributor distributing images with more appropriate information about the real-time state of viewers. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a functional configuration diagram of each HMD according to the first embodiment. [Figure 2A] 3A to 3C are diagrams illustrating displays of the HMDs according to the first embodiment. [Figure 2B] 3A to 3C are diagrams illustrating displays of the HMDs according to the first embodiment. [Figure 2C] 3A to 3C are diagrams illustrating displays of the HMDs according to the first embodiment. [Diagram 3] 4 is a flowchart of processing of each HMD according to the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating auxiliary display items according to the first embodiment. [Diagram 5] FIG. 11 is a configuration diagram of a server device and an HMD according to a second embodiment. [Figure 6A] 11A to 11C are diagrams illustrating a display of an HMD according to a second embodiment. [Figure 6B] 11A to 11C are diagrams illustrating a display of an HMD according to a second embodiment. [Figure 7] 10 is a flowchart of the processing of an HMD and a server device according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0013] In live commerce, broadcasters often take pictures of products using a camera mounted on a smartphone. Viewers often use their smartphones to view the broadcasted images and send comments. In this situation, the broadcaster needs to read the comments sent by the viewers on the small screen of the smartphone, and the visibility of the screen is low. This makes it difficult for the broadcaster to grasp in real time the explanation desired by the viewer who sent the comment. In addition, since the camera viewpoint of the broadcasted image is the viewpoint selected by the broadcaster, the viewer is forced to see the range of the product as seen from the viewpoint specified by the broadcaster. For this reason, the viewer cannot freely see the range of the product that he or she wants to see, which can be stressful. In consideration of such problems, each embodiment will be described below.

[0014] <Embodiment 1> 1A and 1B, a functional configuration of an HMD according to embodiment 1 will be described below. In embodiment 1, an HMD worn by a first user (hereinafter referred to as a "distributor") acquires information on positions where multiple second users (hereinafter referred to as "viewers") are looking, and generates a heat map.

[0015] A heat map is a display item for visualizing the position of an image (objects displayed in the image) that each viewer is looking at. A heat map is a display item that colors the position of each viewer with a shade according to the number of viewers looking at that position (see Figures 2B and 2C).

[0016] 1A and 1B are diagrams showing the configuration of an HMD (head mounted display) 100 according to embodiment 1. Fig. 1A is a diagram showing the configuration of an AR (Augmented Reality) type HMD 100A worn by a distributor. Fig. 1B is a diagram showing the configuration of a VR (Virtual Reality) type HMD 100B worn by a viewer.

[0017] The HMD 100A includes an imaging unit 101, a communication unit 102, a viewpoint position acquisition unit 103, an attribute acquisition unit 104, a control unit 105, a ROM 106, a RAM 107, a heat map generation unit 108, and a display unit 109. These components are connected to each other via a bus 110.

[0018] The HMD 100B includes a communication unit 102, a viewpoint position acquisition unit 111, an attribute acquisition unit 112, a control unit 105, a ROM 106, a RAM 107, and a display unit 109. These components are connected to each other via a bus 110.

[0019] The imaging unit 101 captures an image of the real space in the line of sight of the distributor, and acquires a moving image (hereinafter, referred to as a "distributor viewpoint image").

[0020] The communication unit 102 connects (communicates) with external devices via a network. Therefore, the communication unit 102 of the HMD 100A and the communication unit 102 of the HMD 100B can communicate with each other. The communication unit 102 executes communication control processing using the network. The communication unit 102 can perform, for example, Internet communication using TCP / IP.

[0021] The viewpoint position acquisition unit 111 detects the position where the viewer is looking (hereinafter referred to as the "viewpoint position"). Furthermore, the viewpoint position acquisition unit 103 acquires information on the viewer's viewpoint position detected by the viewpoint position acquisition unit 111. Note that a method for detecting the viewpoint position is a well-known technique, and therefore a detailed description thereof will be omitted.

[0022] The attribute acquisition unit 112 acquires attribute information of the viewer in response to the operation of the viewer. The attribute acquisition unit 104 acquires attribute information of the viewer from the HMD 100B. The attribute information is, for example, information on the purchase status of "a product (object) that a distributor is introducing in real time using an image" and the image viewing status. Specifically, in the first embodiment, the attribute information is one of three types of information: "product not purchased", "product purchased", and "viewing completed".

[0023] "Product not purchased" indicates that the viewer has not purchased the product. "Product purchased" indicates that the viewer has purchased the product. "Viewing completed" indicates that the viewer has completed viewing.

[0024] The control unit 105 performs overall control of each component connected to the bus 110. Furthermore, in the HMD 100A, the control unit 105 selects information on a viewpoint position (one or more viewpoint positions) according to a purpose from information on a plurality of viewpoint positions acquired by the viewpoint position acquisition unit 103. In the first embodiment, the control unit 105 selects information on the viewpoint position of a viewer whose attribute information indicates "product not purchased".

[0025] The ROM 106 stores a control program, an operating system (OS), and Stores various other programs.

[0026] The RAM 107 functions as a main memory and a work area for the control unit 105. The control unit 105 loads programs and the like required for executing processes into the RAM 107, and then executes the programs to realize various operations.

[0027] The heat map generating unit 108 generates a heat map based on the viewpoint positions and the number of viewers (number of viewpoints) using one or more viewpoint positions selected by the control unit 105 as a population. When a viewer is viewing a CG (Computer Graphics) image different from the distributor viewpoint image, the heat map generating unit 108 converts the viewpoint position in the CG image into the coordinates of the product in the distributor viewpoint image and generates a heat map according to that position. The CG image is an image in which the product is rendered into a virtual object.

[0028] The display unit 109 is attached to the HMD 100 so as to be located in front of both eyes of a user wearing the HMD 100 on his / her head. The display unit 109 of the HDM 100A realizes optical see-through. The display unit 109 of the HDM 100A has a see-through lens so that the distributor can see the product through the display unit 109 (so that the product is visible). For this reason, the display unit 109 of the HDM 100A displays a heat map so that the heat map is seen superimposed on the product in real space. The display unit 109 of the HDM 100B displays a distributor's viewpoint image or a CG image.

[0029] It is not necessary for the HMD 100A to have all of the components shown in Fig. 1A. For example, the HMD 100A may have only the imaging unit 101 and the display unit 109, and the other components may be included in a control device (a control device that controls the display of the HMD 100A). In other words, the HMD 100A may have the imaging unit 101, the display unit 109, and the control device.

[0030] The exchange of data between the HMD 100A and the HMD 100B and the displayed images will be described with reference to FIGS. 2A to 2C.

[0031] FIG. 2A shows a viewer viewing either a "distributor's viewpoint image capturing a camera, which is a product held by the distributor," or a "CG image in which the camera is a virtual object." FIG. 2B shows a heat map being superimposed on a product based on the viewpoint positions of multiple viewers. FIG. 2C shows the heat map when the attribute information of one of the multiple viewers changes. Each viewer can choose whether to view the distributor's viewpoint image or the CG image.

[0032] In FIG. 2A, a distributor 201 holds a camera, which is a product, in his / her hand and wears an HMD 100A with a see-through lens on his / her head. A distributor perception display 205 is a display that the distributor 201 perceives through a display unit 109. An imaging unit 101 of the HMD 100A captures an image of the product (camera) to obtain an image that is the same as the distributor perception display 205. Then, the image obtained by the imaging unit 101 is distributed to an HMD 100B of a viewer 202-204 as a distributor viewpoint image.

[0033] Viewers 202 to 204 wear VR-type HMDs 100B on their heads. A viewer display image 206 is displayed on the display unit 109 of the HMD 100B worn by viewer 202. A viewer display image 207 is displayed on the display unit 109 of the HMD 100B worn by viewer 203. A viewer display image 208 is displayed on the display unit 109 of the HMD 100B worn by viewer 204.

[0034] In Fig. 2A, viewers 202 and 204 view a distributor's viewpoint image distributed by HMD 100A as a viewer display image. In contrast, viewer 203 views a CG image different from the distributor's viewpoint image as a viewer display image. This CG image represents a virtual object of a product shown in the distributor's viewpoint image. Viewer 203 can freely rotate the virtual object (change the attitude of the virtual object) by operating it, and can check any range of the product.

[0035] 2B, viewpoint position 209 indicates the dial part of the camera as the position where viewer 202 looks in viewer display image 206. Viewpoint position 210 indicates the liquid crystal panel part of the camera as the position where viewer 203 looks in viewer display image 207. Similarly, viewpoint position 211 indicates the liquid crystal panel part as the position where viewer 204 looks in viewer display image 208.

[0036] Here, it is assumed that the attribute information of all three viewers 202-204 indicates "no product purchase", and that the HMD 100B of the viewers 202-204 stores information on the product purchase status as the attribute information. The HMD 100B of the viewers 202-204 transmits the attribute information indicating the product purchase status and the information on the viewpoint position to the HMD 100A of the distributor. After that, the HMD 100A selects viewers who have "not purchased the product" based on the attribute information received from the HMD 100B of the viewers 202-204, and generates a heat map using the viewpoint positions of the viewers who have "not purchased the product" as a population. In the distributor perception display 205, a heat map 212 for one person is superimposed on the dial portion, and a heat map 213 for two people with a darker color is superimposed on the liquid crystal panel portion.

[0037] Also, in the state of FIG. 2B, assume that the viewer 202 purchases the product described by the distributor. In response to this purchase, the attribute acquisition unit 112 of the HMD 100B of the viewer 202 updates the attribute information from "product not purchased" to "product purchased". The HMD 100B of the viewer 202 then transmits the updated attribute information and information on the viewpoint position to the HMD 100A of the distributor. The heat map generation unit 108 of the HMD 100A receives this information and ends the generation (display) of the heat map 212 corresponding to the viewpoint position of the viewer 202. As a result, only the heat map 213 corresponding to the viewpoint positions of the two people is displayed on the distributor perception display 205 (HMD 100A).

[0038] The flow of the transmission and reception process between the HMD 100A of the distributor and the HMD 100B of the viewer according to the first embodiment will be described with reference to FIGS. 3A, 3B, and 4. FIG.

[0039] The flowchart in FIG. 3A illustrates the processing executed by the HMD 100A.

[0040] In step S301, the imaging unit 101 captures an image of a product (object) to obtain a distributor's viewpoint image. The control unit 105 distributes the distributor's viewpoint image to an external network (the viewer's HMD 100B via the external network). Here, the distributor's viewpoint image shows a product described by the distributor. The control unit 105 may also obtain a CG image of the product shown in the distributor's viewpoint image. In this case, the control unit 105 distributes the CG image together with the distributor's viewpoint image to the external network (the viewer's HMD 100B via the external network).

[0041] In step S302, the attribute acquisition unit 104 and the viewpoint position acquisition unit 103 judge whether or not information (attribute information and viewpoint position information) of at least one viewer has been acquired from the external network. If it is judged that information (attribute information and viewpoint position information) of at least one viewer has been acquired, the process proceeds to step S303. If it is judged that information (attribute information and viewpoint position information) of any viewer has not been acquired, the process proceeds to step S304. Process 02 is repeated.

[0042] In step S303, the heat map generating unit 108 selects a population of the heat map based on the attribute information and the information of the viewpoint position acquired in step S302. In the first embodiment, there are three types of information in the attribute information: "Product not purchased", "Product purchased", and "Watching completed". Therefore, the heat map generating unit 108 determines viewers (M viewers) who have attribute information of "Product not purchased" among the viewers (N viewers) who have transmitted attribute information (received the distributor viewpoint image). Then, the heat map generating unit 108 selects the information of the viewpoint positions of the M viewers who have not purchased the product as a population for generating the heat map.

[0043] In step S304, the heat map generating unit 108 calculates the viewpoint positions (coordinates) of the M viewers who have "not purchased the product" selected in step S303. Then, the heat map generating unit 108 calculates the number of viewers who are looking at each viewpoint position, and generates a heat map.

[0044] As described above, there are cases where the viewer is viewing a CG image instead of a distributor's viewpoint image. In this case, the heat map generating unit 108 converts the viewpoint position into coordinates within the product in the distributor's viewpoint image and uses the converted coordinates to generate a heat map. Here, a technique such as "using the position of the product in the distributor's viewpoint image as a reference to perform coordinate conversion based on three-dimensional rotation information and shift amount of a virtual object in a CG image" may be used, or a known technique may be used.

[0045] In step S305, the display unit 109 displays the heat map generated in step S304 so that the heat map and the product (the product's position indicated by the heat map) are seen to be superimposed on each other, as shown in FIG. 2B. If the product's position indicated by the heat map is in a blind spot as seen by the distributor, the heat map generation unit 108 generates an auxiliary display item 400 indicating that there is a viewer viewing a position that the distributor cannot view, as shown in FIG. 4. For such a position, the display unit 109 displays the auxiliary display item 400 instead of the heat map. The auxiliary display item 400 indicates the general position (product's position) where the viewer is viewing by using an arrow shape.

[0046] 4, viewer 202 has already "purchased the product," while viewer 203 has "not purchased the product," as in the case of FIG.

[0047] In step S306, the control unit 105 judges whether the distributor has finished explaining the product. If it is judged that the distributor has finished explaining the product, the process proceeds to step S307. If it is judged that the distributor has not finished explaining the product, the process proceeds to step S301.

[0048] In step S307, the control unit 105 stops the distribution of the distributor viewpoint image, and transmits a distribution stop notification to the external network.

[0049] The flowchart in Fig. 3B explains the process executed by the HMD 100B. The process of this flowchart is executed by each of the M HMDs 100B of the M viewers who receive the delivery of the distributor viewpoint image (including the N HMDs 100B of the N viewers who have "not purchased the product").

[0050] In step S321, the attribute acquisition unit 112 acquires attribute information from the current state of the viewer. In step S321, the attribute acquisition unit 112 sets the attribute information to indicate "product not purchased."

[0051] In step S322, the control unit 105 determines whether or not a distributor's viewpoint image has been received. If it is determined that the distributor's viewpoint image has been received, the process proceeds to step S323. If it is determined that the distributor's viewpoint image has not been received, the process of step S322 is repeated.

[0052] In step S323, the control unit 105 receives a CG image in which the product shown in the distributor viewpoint image is made into a virtual object. Note that the control unit 105 does not necessarily have to receive the CG image from the HMD 100A. For example, the control unit 105 may receive the CG image from a specific cloud server that stores the CG image after receiving the distributor viewpoint image from the HMD 100A.

[0053] In step S324, the control unit 105 selects either the distributor's viewpoint image or the CG image based on the viewer's instruction (selection). The control unit 105 displays the selected image on the display unit 109 as a viewer display image. The distributor's viewpoint image does not necessarily display the range of the product that the viewer wants to see. On the other hand, the distributor can check any range of the product by rotating the virtual object of the product in the CG image.

[0054] In step S325, the control unit 105 transmits the attribute information and the viewer's viewpoint position information to the HMD 100A via the external network. The viewpoint position information includes not only the coordinate information of the viewpoint position, but also information that can identify the image that the distributor is viewing (whether the distributor's viewpoint image or the CG image is being viewed).

[0055] In step S326, the control unit 105 determines whether or not a purchase action has occurred by the viewer for the product described in the distribution. The control unit 105 determines whether or not a purchase action has occurred based on operations on the viewer display images 206-208 or operations on a device connected to the HMD 100B. If it is determined that a purchase action has occurred, the process proceeds to step S327. If it is determined that a purchase action has not occurred, the process proceeds to step S328.

[0056] In step S327, the attribute acquiring unit 112 sets the attribute information to indicate "product purchased".

[0057] In step S328, the control unit 105 determines whether or not a delivery stop notification has been received. If it is determined that a delivery stop notification has been received, the process proceeds to step S332. If it is determined that a delivery stop notification has not been received, the process proceeds to step S329.

[0058] In step S329, the control unit 105 determines whether or not the viewer has performed a viewing end operation while receiving the distributor viewpoint image. If it is determined that the viewer has performed a viewing end operation, the process proceeds to step S330. If it is determined that the viewer has not performed a viewing end operation, the process proceeds to step S324.

[0059] In step S330, the attribute acquisition unit 112 sets the attribute information to indicate "viewing ended."

[0060] In step S331, the control unit 105 transmits attribute information and information on the viewer's viewpoint position to the HMD 100A via an external network.

[0061] In step S332, the control unit 105 stops receiving the images (the distributor's viewpoint image and the CG image) that have been received via the external network.

[0062] According to the first embodiment, the distributor can grasp the range (range of the product) in which the viewer wants an explanation (the viewer is interested) by looking at a heat map showing the position of the product that the viewer with a specific attribute is looking at. Therefore, the distributor can provide an explanation (product explanation) that satisfies the viewer.

[0063] Although viewpoint information of viewers who "have not purchased the product" was used as the population for generating the heat map, viewpoint information of viewers who "have purchased the product" may also be used as the population for generating the heat map. This allows the broadcaster to understand "what parts of the product viewers who have already purchased the product looked at." In other words, the broadcaster can use the heat map to understand the range of products that have product appeal (attractiveness that leads to purchasing desire).

[0064] The attribute information may also include personal information of the viewer (such as age, sex, place of residence, or place of origin) along with information on the product purchase status. In other words, the HMD 100A may determine the viewer to be the population of the viewpoint information according to the viewer's personal information (such as age, sex, place of residence, or place of origin) along with the product purchase status. According to this, for example, viewpoint information of viewers who have "not purchased the product" and are "males under 20 years old" can be used as the population for generating a heat map. By limiting the viewers used to generate a heat map in this way, for example, a distributor can understand, by the heat map, points of interest of men under 20 years old in a product. In other words, by including personal information of the viewer in the attribute information, a distributor can narrow down the target for selling products and understand the viewer's interest in the product.

[0065] Furthermore, in the above, the number of viewers viewing each product position is expressed by a heat map. However, the positions of products viewed by viewers (or the number of viewers viewing each product position) may be expressed by any expression method (display item) other than a heat map. For example, an expression method (such as a color map) may be used that changes the "color" or "color transmittance" applied to each product position depending on the number of viewers viewing each product position. Alternatively, an expression method may be used that displays a number indicating the number of viewers viewing each product position at each product position.

[0066] <Embodiment 2> In the first embodiment, the HMD 100A acquires information (attribute information and information on viewpoint position) of each viewer and generates a heat map. However, in a situation where the HMD 100A is required to be small and lightweight and to generate a heat map quickly, the generation of a heat map by the HMD 100A with low processing power may not be appropriate.

[0067] In the second embodiment, as shown in Fig. 6A, an HMD 500 worn by a distributor and an HMD 100B worn by a viewer are connected via a server device 510. The server device 510 is a control device that controls the display of the HMD 500. Note that the configuration of the HMD 100B according to the second embodiment is the same as the configuration of the HMD 100B according to the first embodiment, and therefore a description thereof will be omitted below.

[0068] Hereinafter, the functional configurations of the HMD 500 and the server device 510 according to the second embodiment will be described with reference to Fig. 5A and Fig. 5B. Note that in Fig. 5A and Fig. 5B, components with the same names (functions) as those in the first embodiment are given the same reference numerals as those in Fig. 1A and Fig. 1B, and detailed description of these components will be omitted.

[0069] FIG. 5A is a configuration diagram of an HMD 500 worn by a broadcaster. The HMD 500 includes an imaging unit 1 The device includes a communication unit 101, a communication unit 102, a control unit 105, a ROM 106, a RAM 107, a display unit 109, and a heat map receiving unit 501. These components are connected to each other via a bus 110.

[0070] The heat map receiving unit 501 receives a heat map from the server device 510 via the communication unit 102 .

[0071] 5B is a diagram showing the configuration of a server device 510 according to embodiment 2. The server device 510 includes a communication unit 102, a viewpoint position acquisition unit 103, an attribute acquisition unit 104, a control unit 105, a ROM 106, a RAM 107, a heat map generation unit 108, and a heat map transmission unit 511. These components are connected to each other via a bus 110.

[0072] The heat map transmission unit 511 transmits the heat map generated by the heat map generation unit 108 to the HMD 500 via the communication unit 102.

[0073] With reference to FIGS. 6A and 6B, data exchange between the distributor's HMD 500, the server device 510, and the viewer's HMD 100B will be described.

[0074] Fig. 6A shows a state where multiple viewers are viewing a distributor's viewpoint image and a CG image. Fig. 6B shows a state where a heat map is displayed on the display unit 109 of the HMD 500 based on the viewer's viewpoint position so that the heat map appears to be superimposed on the product.

[0075] 6A, a distributor's viewpoint image (an image of a product captured by the imaging unit 101 of the HMD 500) and a CG image are transmitted to a server device 510. After that, the distributor's viewpoint image and the CG image are transmitted (distributed) from the server device 510 to the HMD 100B of each viewer via an external network. The distributor's viewpoint image is displayed as a viewer display image 206 on the HMD 100B of the viewer 202. The CG image is displayed as a viewer display image 207 on the HMD 100B of the viewer 203.

[0076] 6B, viewpoint position 209 is the position where viewer 202 looks in viewer display image 206, and points to the dial part of the camera. Viewpoint position 210 is the position where viewer 203 looks in viewer display image 207, and points to the liquid crystal panel part. Here, it is assumed that the attribute information of viewer 202 is "purchased", and the attribute information of viewer 203 is "product not purchased". Furthermore, product purchase status information is stored as attribute information in HMD 100B of each of viewer 202 and viewer 203.

[0077] The attribute information and information on the viewpoint position are transmitted to the server device 510 from the HMD 100B of each of the viewers 202 and 203. After that, the server device 510 selects those who have "not purchased the product" based on the attribute information received from the HMD 100B of the viewer 202 and the attribute information received from the HMD 100B of the viewer 203. The server device 510 also generates a heat map using the information on the viewpoint positions of the selected people who have "not purchased the product" as a population. When the heat map is transmitted to the HMD 500, the HMD 500 displays the heat map as shown in the broadcaster perception display 205 so that the broadcaster perceives that the heat map is superimposed on the position (range) of the product that the viewer 203 is looking at. As a result, in FIG. 6B, only the heat map 613 corresponding to the viewpoint position of the viewer 203 is displayed on the broadcaster perception display 205 (HMD 100A).

[0078] With reference to the flowcharts of FIG. 7A and FIG. 7B, the flow of the transmission and reception process between the HMD 500 of the distributor, the server device 510, and the HMD 100B of the viewer in the second embodiment will be described. Note that in the second embodiment, the process of the HMD 100B is performed when "the destination of information is H Since the embodiment is the same as the first embodiment except for the change from MD 100A to server device 510, the description will be omitted.

[0079] Fig. 7A is a diagram for explaining the processing executed by the HMD 500. Note that among the steps in the flowchart of Fig. 7A, the steps in which the same processing is performed as the steps in the flowchart of Fig. 3A are assigned the same numbers. Therefore, detailed explanations of the processing of these steps are omitted.

[0080] In step S700, the display unit 109 receives the heat map from the server device 510 and displays the heat map so that it appears to be superimposed on the product (see FIG. 6B). At this time, if the position indicated by the heat map is in a blind spot as seen by the distributor, the display unit 109 displays the auxiliary display item 400 indicating that "there is a viewer viewing a position that the distributor cannot see."

[0081] FIG. 7B is a diagram illustrating the process executed by server device 510. As shown in FIG.

[0082] In step S701, the control unit 105 determines whether or not a distributor's viewpoint image has been received. If it is determined that the distributor's viewpoint image has been received, the process proceeds to step S702. If it is determined that the distributor's viewpoint image has not been received, the process of step S701 is repeated.

[0083] In step S702, the control unit 105 receives a CG image (a CG image in which a product shown in the distributor viewpoint image is made into a virtual object).

[0084] In step S703, the attribute acquisition unit 104 and the viewpoint position acquisition unit 103 determine whether or not information (attribute information and viewpoint position information) of at least one viewer has been acquired from the external network. If it is determined that information (attribute information and viewpoint position information) of at least one viewer has been acquired, the process proceeds to step S704. If it is determined that information (attribute information and viewpoint position information) of any viewer has not been acquired, the process of step S703 is repeated.

[0085] In step S704, similarly to step S303, the heat map generating unit 108 selects a population of the heat map based on the attribute information and the information on the viewpoint position acquired in step S703.

[0086] In step S705, similarly to step S304, the heat map generating unit 108 calculates the viewpoint positions (coordinates) of all viewers (N viewers) whose attribute information selected in step S303 indicates “product not purchased.” Then, the heat map generating unit 108 calculates the number of viewers looking at each viewpoint position, and generates a heat map.

[0087] In step S706, the control unit 105 transmits the heat map to the HMD 500.

[0088] In step S707, the control unit 105 determines whether or not a distribution stop notification has been received from the HMD 500. If it is determined that the distribution stop notification has been received, the process proceeds to step S708. If it is determined that the distribution stop notification has not been received, the process proceeds to step S701.

[0089] In step S708, the control unit 105 stops receiving the distributor viewpoint image and the CG image.

[0090] According to the second embodiment, even when a distributor's HMD needs to be made smaller and lighter and heat map generation needs to be made faster, the distributor can see the heat map that shows the position on the product where the viewer is looking, and can grasp the part of the product that the viewer wants to be explained about. This allows the distributor to provide a satisfactory explanation (product explanation) to the viewer.

[0091] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0092] Also, in the above, "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2." Conversely, "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2." Therefore, unless a contradiction occurs, "equal to or greater than A" may be read as "equal to or greater than A (high; long; many)," and "equal to or less than A" may be read as "equal to or less than A (low; short; few)." And, "equal to or greater than A" may be read as "equal to or greater than A," and "equal to or less than A" may be read as "equal to or less than A."

[0093] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0094] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0095] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) A control device for controlling a first display device of a distributor who is capable of viewing a specific object through the first display device and who distributes an image capturing the specific object, An acquisition means for acquiring a position where each of N users (N≧1) who view an image of the specific object and have a specific attribute is looking at the specific object; a generating means for generating a first display item indicating a position in the particular object at which each of the N users is looking; a control means for displaying the first display item on the first display device so that the first display item appears to overlap with a position on the specific object indicated by the first display item; A control device comprising: (Configuration 2) The first display item is a heat map that visualizes the position where each user is looking by coloring the position with a shade corresponding to the number of users looking at the position. 2. The control device according to configuration 1, (Configuration 3) The generating means determines, from among M users (M≧N) who view the specific object displayed in the image of the specific object, the N users who have the specific attribute based on attribute information of the M users. 3. The control device according to configuration 1 or 2. (Configuration 4) The attribute information includes information indicating a purchase status of the specific object. 4. The control device according to configuration 3. (Configuration 5) The specific attribute is an attribute of being a person who has not purchased the specific object. 5. The control device according to any one of configurations 1 to 4. (Configuration 6) and when any of the N users is viewing a position that cannot be seen by the distributor via the first display device, the control means displays, on the first display device, a second display item indicating that there is a user viewing a position that cannot be seen by the distributor. 6. The control device according to any one of configurations 1 to 5. (Configuration 7) each of the N users can select an image of the specific object to be viewed by the user from an image of the specific object captured by the first display device and a CG (Computer Graphics) image of the specific object; Each of the N users is capable of changing the posture of the specific object in the CG image by performing an operation when viewing the CG image. 7. The control device according to any one of configurations 1 to 6. (Configuration 8) the control device is included in the first display device, The image distributed by the distributor is distributed to each of the N second display devices of the N users; The N second display devices are communicatively connected to the first display device. 8. The control device according to any one of configurations 1 to 7. (method) A control method for controlling a first display device of a distributor who is able to view a specific object through a first display device and who distributes an image capturing the specific object, comprising: An acquisition step of acquiring a position where each of N users (N≧1) who view an image of the specific object and have a specific attribute is looking at the specific object; A generating step of generating a first display item indicating a position at which each of the N users is looking at the particular object; a control step of displaying the first display item on the first display device so that the first display item appears to overlap a position on the specific object indicated by the first display item; A control method comprising the steps of: (program) A program for causing a computer to function as each of the means of the control device according to any one of configurations 1 to 8. [Explanation of symbols]

[0096] 100A: HMD, 100B: HMD, 101: imaging unit, 103: viewpoint position acquisition unit, 105: control unit, 108: heat map generating unit, 109: display unit

Claims

1. A control device for controlling a first display device of a distributor that can view a specific object through the first display device and distributes an image of the specific object, An acquisition means for acquiring a position where each of N users (N≧1) who view an image of the specific object and have a specific attribute is viewing the specific object; a generating means for generating a first display item indicating a position on the particular object at which each of the N users is looking; a control means for displaying the first display item on the first display device so that the first display item appears to overlap with a position on the specific object indicated by the first display item; A control device comprising:

2. The first display item is a heat map that visualizes the position where each user is looking by coloring the position with a shade corresponding to the number of users looking at the position. The control device according to claim 1 .

3. The generation means determines, from among M users (M≧N) who view the specific object displayed in the image of the specific object, N users who have the specific attribute based on attribute information of the M users.

3. The control device according to claim 1 or 2.

4. The attribute information includes information indicating a purchase status of the specific object.

4. The control device according to claim 3.

5. The specific attribute is an attribute of being a person who has not purchased the specific object.

3. The control device according to claim 1 or 2.

6. and when any of the N users is viewing a position that cannot be seen by the distributor via the first display device, the control means displays, on the first display device, a second display item indicating that there is a user viewing a position that cannot be seen by the distributor.

3. The control device according to claim 1 or 2.

7. each of the N users can select an image of the specific object to be viewed by the user from an image of the specific object captured by the first display device and a CG (Computer Graphics) image of the specific object; Each of the N users is capable of changing the posture of the specific object in the CG image by performing an operation when viewing the CG image.

3. The control device according to claim 1 or 2.

8. The control device is included in the first display device, The image distributed by the distributor is distributed to each of the N second display devices of the N users; The N second display devices are communicatively connected to the first display device.

3. The control device according to claim 1 or 2.

9. A control method for controlling a first display device of a distributor who is able to view a specific object through a first display device and who distributes an image capturing the specific object, comprising: An acquisition step of acquiring a position where each of N users (N≧1) who view an image of the specific object and have a specific attribute is looking at the specific object; generating a first display item that indicates a position in the particular object that each of the N users is looking at; a control step of displaying the first display item on the first display device such that the first display item appears to overlap a position on the specific object indicated by the first display item; A control method comprising the steps of:

10. A program for causing a computer to function as each of the means of the control device according to claim 1 or 2.

Images