Information processing method, information processing device, and program

Abstract

The present invention generates a synthesized user focus region map obtained by synthesizing a focus region of a real-space user and a focus region of an image-observing user, and analyzes the level of focus in real space in terms of region units. The present invention involves generating a real-space user focus region map obtained by analyzing focus regions of a real-space user who is directly observing real space, an image-observing user focus region map obtained by analyzing a focus region of an image-observing user who is observing a three-dimensional image in real space, and a synthesized user focus region map obtained by synthesizing said two focus region maps. Any one or more of these focus region maps are used to detect the region of the real-space region where there is a high level of focus or the region of the real-space region where there is a low level of focus, and effective advertisement display processing, warning output processing, etc. are carried out.

Description

Information processing method, information processing device, and program

[0001] The present disclosure relates to an information processing method, an information processing device, and a program. More specifically, the present disclosure relates to an information processing method, an information processing device, and a program that analyzes the gaze points of real users in real space and image viewing users who view images distributed online, and performs effective advertisement display processing, warning output processing, and the like based on the analysis results.

[0002] In recent years, with the rapid spread of the Internet, online product sales have surpassed those in brick-and-mortar stores. In addition, in such online product sales, the use of configurations that use three-dimensional images that reproduce real spaces such as stores, such as metaverse spaces, is increasing.

[0003] One example of a 3D image that recreates a real space, such as a store, is a digital twin. A digital twin is data composed of 3D image data that is an exact copy of a real-world building or store. For example, based on information acquired by cameras and sensors installed in a real-world store, 3D image data that recreates the ever-changing state of the store in real time is generated and distributed via the Internet.

[0004] The user displays the distributed image data, i.e., digital twin data, on the display of the user's device, such as a head-mounted display (HMD), PC, smartphone, etc. By viewing the digital twin data, which is 3D image data that recreates the interior of a store, via the HMD or PC, the user can shop online with the same feeling as if they were purchasing products in a physical store, without having to go to the actual store.

[0005] Image data such as digital twins can be distributed to a large number of users that cannot be accommodated in a real store, and it is expected that more users will purchase products.

[0006] As mentioned above, a digital twin is a copy of image data of a real space. For example, a user wearing a head-mounted display (HMD) can observe the state of a real store as it is, and can observe the products, store clerks, advertisements, etc. present in the real store through images just as if they were in the real store.

[0007] In order to effectively display advertisements to these users, it is important to display advertisements in positions where many users can see them.

[0008] For example, Patent Document 1 (Patent Publication No. 6932224) discloses a configuration that aims to achieve greater advertising effectiveness by analyzing the viewpoint direction of an avatar equivalent to a user wearing a head-mounted display (HMD) from the image displayed by the HMD and displaying advertisements in that viewpoint direction.

[0009] However, the above-mentioned Patent Document 1 (Patent Publication No. 6932224) is merely a configuration for controlling the position of advertisements on images distributed online. The above-mentioned digital twin data is three-dimensional image data that reproduces a real space, for example, the interior of a real store, as it is, and the position of advertisements inside the real store becomes the position of advertisements on images distributed online as is, so there is a problem that the processing described in the above document cannot be applied.

[0010] Patent No. 6932224

[0011] The present disclosure has been made in consideration of, for example, the above-mentioned problems, and provides an information processing method, an information processing device, and a program that analyze the gaze points of real users in real space and image observing users who observe the online distributed images in a configuration in which reproduced image data of real space, such as a digital twin, is distributed online, and that performs effective advertisement display processing, warning output processing, etc. based on the analysis results.

[0012] A first aspect of the present disclosure is an information processing method executed in an information processing device, in which a data processing unit generates a real-space user gaze area map, which is analysis data of the gaze area of ​​a real-space user who directly observes real space, and an image-observing user gaze area map, which is analysis data of the gaze area of ​​an image-observing user who observes a three-dimensional image of the real space, and detects at least one of areas of high gaze or areas of low gaze in the real space using at least one of the real-space user gaze area map and the image-observing user gaze area map.

[0013] Furthermore, a second aspect of the present disclosure resides in an information processing device that has a data processing unit that performs analysis processing of gaze areas, and that generates a real-space user gaze area map, which is analysis data of the gaze area of ​​a real-space user who directly observes real space, and an image-observing user gaze area map, which is analysis data of the gaze area of ​​an image-observing user who observes a three-dimensional image of the real space, and that detects at least one of areas in the real space that are highly gazed upon or areas that are low in gaze, using at least one of the real-space user gaze area map and the image-observing user gaze area map.

[0014] Furthermore, a third aspect of the present disclosure is a program for executing information processing in an information processing device, which program causes a data processing unit to execute the following processes: generating a real-space user gaze area map, which is analysis data of the gaze area of ​​a real-space user who directly observes real space; and an image-observing user gaze area map, which is analysis data of the gaze area of ​​an image-observing user who observes a three-dimensional image of the real space; and detecting at least one of areas of high gaze or low gaze in the real space using at least one of the real-space user gaze area map and the image-observing user gaze area map.

[0015] The program of the present disclosure is, for example, a program that can be provided in a computer-readable format via a storage medium or a communication medium to an information processing device or a computer system capable of executing various program codes. By providing such a program in a computer-readable format, processing according to the program is realized on the information processing device or the computer system.

[0016] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description of the embodiments of the present disclosure and the accompanying drawings. Note that in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices that are located within the same housing.

[0017] According to one embodiment of the present disclosure, a configuration is realized in which a composite user gaze area map is generated by combining the gaze areas of a real-space user and an image-observing user, and the degree of attention is analyzed for each area in the real space. Specifically, for example, a real-space user gaze area map is generated by analyzing the gaze areas of a real-space user who directly observes the real space, an image-observing user gaze area map is generated by analyzing the gaze areas of an image-observing user who observes a three-dimensional image of the real space, and a composite user gaze area map is generated by combining these two gaze area maps. Using at least one of these gaze area maps, areas of high or low attention in the real space are detected, and effective advertisement display processing, warning output processing, etc. are performed. This configuration is realized in which a composite user gaze area map is generated by combining the gaze areas of a real-space user and an image-observing user, and the degree of attention is analyzed for each area in the real space. Note that the effects described in this specification are merely exemplary and not limiting, and additional effects may also be present.

[0018] 1 is a diagram illustrating an event venue, which is an example of an environment to which the processing of the present disclosure can be applied; FIG. 2 is a diagram illustrating an example of the gaze direction of a real-space user who is a visitor (customer) at the event venue; FIG. 3 is a diagram illustrating an example configuration of an event venue, which is an example of an environment to which the processing of the present disclosure can be applied; FIG. 4 is a diagram illustrating an example of a gaze area analysis result of a real-space user at the event venue; FIG. 5 is a diagram illustrating a process of generating digital twin data including digital twin image data, which is three-dimensional image data that reproduces an event venue, and distributing the data via a network; FIG. 6 is a diagram illustrating an example of the gaze area of ​​a certain image observing user who is observing image data (digital twin image data) distributed by an information processing device; FIG. 7 is a diagram illustrating an example of the gaze area of ​​a certain image observing user who is observing image data (digital twin image data) distributed by an information processing device; FIG. 8 is a diagram illustrating an example of an image observing user gaze area analysis process executed by an information processing device; FIG. 9 is a diagram illustrating an example of a gaze area map generated by an information processing device; FIG. 10 is a diagram illustrating an example of a process of generating a combined user gaze area map (real-space user & image observing user gaze area map) by integrating gaze area maps corresponding to two different types of users; FIG. 10 is a diagram illustrating an example of a change over time in the gaze scores of a plurality of advertisement displayable areas set up at an event venue. ｓｔａｒｔ ~T ｅｎｄ) is a diagram illustrating an example of a calculation result of the gaze score S of an advertisement displayed on a display device. It is a diagram illustrating a specific example of a calculation process of an advertising price P for each area based on the value of the gaze score S for each area, which is executed by an information processing device. It is a diagram illustrating a specific example of data provided to an advertisement provider, an advertisement management company, or the like. It is a diagram illustrating example data of an advertising price P for each advertisement displayable area unit determined based on the gaze score S for each advertisement displayable area unit calculated by an information processing device. It is a diagram illustrating an advertising image display process by a robot equipped with an advertisement output function such as a projector. It is a diagram illustrating a flowchart illustrating a sequence of a process executed by an information processing device of the present disclosure. It is a diagram illustrating a flowchart illustrating a sequence of a process executed by an information processing device of the present disclosure. It is a diagram illustrating an example in which an information processing device that executes a generation process of a user gaze area map and an information processing device that generates and distributes digital twin data are configured separately. It is a diagram illustrating a factory as an example of an environment to which the process of the present disclosure can be applied. It is a diagram illustrating a configuration in which a factory terminal installed in a factory and an information processing device that is a gaze area analysis server are connected via a communication network. It is a diagram illustrating an example of a gaze area analysis result of a real-space user who is a worker at the factory. It is a diagram illustrating an example of a gaze area analysis process of each worker based on a real-space user gaze area map by an information processing device. FIG. 10 is a diagram illustrating a specific example of monitoring processing in a factory monitoring center. FIG. 11 is a diagram illustrating processing, etc., for transmitting digital twin data including three-dimensional image data that reproduces the current state of a factory generated by an information processing device to a factory monitoring center. FIG. 12 is a diagram illustrating an example of an image observing user gaze area map generated as a result of gaze area analysis processing of an image observing user (monitoring user) executed by an information processing device. FIG. 13 is a diagram illustrating an example of outputting a warning message such as "The suspension area is misaligned" via a speaker of the user terminal (HMD) of the image observing user. FIG. 14 is a diagram illustrating an example of a composite user gaze area map (real space user & image observing user gaze area map) generated by an information processing device. FIG. 15 is a diagram illustrating a flowchart illustrating the sequence of processing of Example 2 executed by the information processing device of the present disclosure.Fig. 1 is a diagram illustrating a flowchart for describing a processing sequence of Example 2 executed by an information processing device of the present disclosure. Fig. 2 is a diagram illustrating a flowchart for describing a processing sequence of Example 2 executed by an information processing device of the present disclosure. Fig. 3 is a diagram illustrating a configuration example of devices constituting an information processing system of the present disclosure. Fig. 4 is a diagram illustrating a configuration example of devices constituting the information processing system of the present disclosure. Fig. 5 is a diagram illustrating a hardware configuration example of a local terminal, an information processing device, a user terminal, etc.

[0019] The information processing method, information processing device, and program of the present disclosure will be described in detail below with reference to the drawings. The description will be made according to the following items: 1. (Example 1) Example of analyzing a user's gaze area to detect an area with high gaze level 2. Regarding the processing sequence of Example 1 executed by the information processing device of the present disclosure 3. (Example 2) Example of analyzing a user's gaze area to detect a problem and execute warning processing 4. Regarding the processing sequence of Example 2 executed by the information processing device of the present disclosure 5. Regarding example configurations of devices constituting the information processing system of the present disclosure 6. Regarding example hardware configurations of the information processing device, etc. 7. Summary of the configuration of the present disclosure

[0020] 1. Example 1 Example of Analyzing User's Gaze Area and Detecting Area with High Gaze Degree] First, as Example 1 of the present disclosure, an example of analyzing a user's gaze area and detecting an area with high gaze degree will be described.

[0021] 1 is a diagram showing an event venue 10 such as a store selling various products. The event venue 10 is an event venue such as a store that exists in the real world, where various products are sold, for example.

[0022] Note that the event venue 10 shown in Figure 1 is merely an example of an environment to which the processing of the present disclosure can be applied. The processing of the present disclosure is not limited to the event venue shown in Figure 1 and can be applied to various locations. It can be used both indoors and outdoors, and can be used in various spaces where people are present, such as stores selling products, downtown areas, train stations, airports, parks, etc. Here, an example of use at the event venue 10 shown in Figure 1 will be described as an example of an environment to which the processing of the present disclosure can be applied.

[0023] At the event venue 10, many products for sale are displayed, there are sales staff selling the products face-to-face, and there are also customers who are visitors to the event venue 10. Here, the visitors to the event venue 10 are shown as real-space users 30. The real-space users 30 are users who directly observe the real space. In other words, they are not users who observe images of the real space, such as two-dimensional images or three-dimensional images, but users who directly observe the real space with their naked eyes.

[0024] As shown in the figure, many advertising display areas 20 are provided on the surrounding walls of an event venue 10. Various advertisements are displayed in the advertising display areas 20.

[0025] The advertisement display area 20 can be set as an area that is gazed upon by many of the real-space users 30 who are visitors to the event venue 10, i.e., as a gaze area, thereby increasing the advertising effectiveness of the displayed advertisements.

[0026] A specific example of the gaze area of ​​the real-space users 30 will be described with reference to Fig. 2. Fig. 2 is a diagram showing the gaze direction of the real-space users 30, who are visitors (customers) at the event venue 10, with dotted arrows. The tip area of ​​each arrow is the gaze area of ​​each user.

[0027] The information processing device disclosed herein executes processing to analyze the gaze areas of these real-space users 30. As shown in Fig. 3, a large number of cameras 41 are installed at the event venue 10, and they capture images of the event venue 10 from various angles. The cameras 41 also capture images of the real-space users 30 who are visitors to the event venue 10. Furthermore, a robot 45 equipped with a camera also moves around and captures images of the interior of the event venue 10 and images of the real-space users 30.

[0028] Although the example shown in Fig. 3 shows only one robot 45 equipped with a camera, multiple robots may be used. Furthermore, an air vehicle such as a drone equipped with a camera may be used to capture images from above the venue. While the example shown in Fig. 3 shows the event venue 10 as an indoor venue, the event venue may be set outdoors, and multiple drones may be flown to capture images from various angles.

[0029] Images taken by cameras 41 installed at various locations in the event venue 10 and images taken by a camera of robot 45 are transmitted to an event venue terminal 40, which is a local terminal of the event venue 10. The event venue terminal 40 transmits the images taken by each camera together with attribute information including the image capture time, camera position information, image capture direction, etc. to an information processing device 50. The information processing device 50 is, for example, a cloud server, and is a gaze area analysis server that executes gaze area analysis processing.

[0030] The information processing device 50 analyzes the images captured by each camera within the event venue 10 received from the event venue terminal 40 and analyzes the gaze direction and gaze area of ​​the real-space users 30 who are visitors to the event venue 10.

[0031] Figure 4 shows an example of the results of analyzing the gaze areas of real-space users 30 at the event venue 10. The information processing device 50 generates a real-space user gaze area map 60 as shown in Figure 4. The real-space user gaze area map 60 shown in Figure 4 is a map that indicates which area of ​​the side wall surface of the event venue 10, i.e., the side wall surface on which the advertisement display area 20 shown in Figure 1 can be set, the real-space users 30 are gazing at.

[0032] The gaze area map shown in Fig. 4 is a map in which an area that is gazed at by more users is set to be brighter (whiter), and an area that is gazed at by fewer users is set to be darker (blacker). Note that the gaze area map shown in Fig. 4 is just an example, and various settings are possible, such as a map in which areas with high and low gaze levels are shown in different colors. The gaze area map is a map that attempts to detect areas with high and low gaze levels in real space.

[0033] The information processing device 50 analyzes the images captured by each camera within the event venue 10 received from the event venue terminal 40, analyzes the gaze areas of real-space users 30 who are visitors to the event venue 10, and generates a real-space user gaze area map 60 as shown in Figure 4.

[0034] Note that the real-space user gaze area map 60 shown in Fig. 4 is only a part of the map, and gaze area maps similar to that shown in Fig. 4 can be generated for side wall surfaces other than those shown in Fig. 4, other product arrangement shelves, etc. However, in order to speed up processing, it is desirable to configure the map generation by predefining only the advertisement displayable area as the map generation area.

[0035] 4 is a real-space user gaze area map 60 at a certain time, for example, time t1, generated by analyzing the gaze area of ​​the real-space user 30 at time t1. As the gaze area of ​​the real-space user 30 moves over time, the real-space user gaze area map 60 is also updated over time.

[0036] The information processing device 50 generates a gaze area map for a certain period of time (tx to ty) that reflects the cumulative information of the gaze area of ​​the real-space user 30 over a certain period of time (tx to ty), for example, and uses this map to calculate a gaze score S for each area of ​​the event venue 10.

[0037] The gaze score S is a value that reflects the degree of gaze per area in real space. The gaze score S is a score for each area of ​​the event venue 10, for example, per advertisement display area, and is determined according to the number of users gazing at that area and the gaze duration.

[0038] For example, the more users who gaze at an area, and the longer the gaze time, the higher the score value. For example, a score value ranging from 0 to 100 is calculated for each area. Furthermore, the information processing device 50 calculates an advertising price P for each area based on the value of the gaze score S for each area. Specific examples of the calculation process for the gaze score S for each area and the advertising price P for each area will be described in detail later.

[0039] The information processing device 50 not only performs the above-described analysis process of the gaze area, but also performs a process of generating digital twin data including three-dimensional image data that reproduces the current state of the event venue 10, so-called digital twin image data, and distributing the data via a network. A specific example of this image distribution configuration will be described with reference to FIG. 5.

[0040] 5, the information processing device 50 distributes three-dimensional image data that reproduces the current state of the event venue 10. The distributed image data is viewed by many users accessing the internet.

[0041] The event venue terminal 40 at the event venue 10 receives not only camera images but also audio information acquired by a microphone installed at the event venue 10 and transmits it to the information processing device 50. The information processing device 50 generates digital twin data including audio information together with three-dimensional image data that reproduces the current state of the event venue 10, and distributes it to many users.

[0042] Furthermore, the event venue terminal 40 may be configured to acquire information from various other sensors installed at the event venue, in addition to images and audio, and transmit the information to the information processing device 50. By using the various sensor information input from the event venue terminal 40, the information processing device 50 can generate and distribute highly accurate digital twin data that is closer to the actual event venue 10.

[0043] The digital twin data generated by the information processing device 50 is distributed to a large number of users via a network. Specifically, as shown in Fig. 5, the digital twin data is transmitted to various user terminals such as a head-mounted display (HMD) 70, a smartphone 71, and a PC 72.

[0044] Image observing users a to c, 80 shown in Figure 5 use various user terminals such as a head-mounted display (HMD) 70, a smartphone 71, and a PC 72 to observe three-dimensional image data, i.e., digital twin image data, that reproduces the current state of the event venue 10.

[0045] As explained earlier, digital twin image data is a three-dimensional copy image data that reproduces the real space exactly as it is, and for example, an image observing user 80 wearing a head-mounted display (HMD) 70 can observe the current state of the event venue 10 in real time.

[0046] That is, by viewing three-dimensional image data (digital twin image data) that recreates the interior of the event venue 10 via a user terminal such as an HMD, smartphone, or PC, the image observing users a to c, 80 can shop with the same sensation as if they were purchasing products at the actual event venue 10, without actually going to the event venue 10. Note that a configuration may also be adopted in which a dedicated operator who can handle distributed images is assigned to explain products to the image observing users 80, enabling dialogue (chat) with the image observing users 80.

[0047] 5 shows only three image observing users 80, many other users can observe the image data (digital twin image data) distributed by the information processing device 50. Each of the many image observing users 80 freely observes the event venue 10, and the area that each user is looking at, i.e., the area of ​​gaze, is different.

[0048] 6 is a diagram showing an example of the gaze area of ​​one image observing user a, 80, who is observing image data (digital twin image data) distributed by the information processing device 50. The image observing user a, 80 shown in FIG. 6 can freely observe in any direction from any position in the event venue 10.

[0049] The image shown in the lower left of FIG. 6 (image observing user gaze area a, 90a) is image data corresponding to the gaze area that the image observing user a, 80 is viewing using the HMD 70, that is, the image observing user gaze area a, 90a.

[0050] In addition, the image observing user a, 80 can freely control the user's position within the event venue 10 and the user's line of sight by controlling the HMD 70, for example by performing directional control, and an image corresponding to this control, i.e., the gaze area image, will be displayed on the display unit of the HMD 70.

[0051] Therefore, the image observing user's gaze area a, 90a shown in the lower left of FIG. 6 is an image showing the gaze area of ​​the image observing user a, 80.

[0052] 7 is a diagram showing an example of the gaze area of ​​another image observing user c, 80, who is observing image data (digital twin image data) distributed by the information processing device 50. The image observing user c, 80 shown in FIG. 7 can also freely observe in any direction from any position in the event venue 10.

[0053] The image shown in the lower left of Figure 7 (image observing user gaze area a, 90a) is image data corresponding to the gaze area that image observing user c, 80 is viewing using the HMD 70, that is, image observing user gaze area c, 90c.

[0054] In this way, the numerous image observation users 80 who observe the image data (digital twin image data) distributed by the information processing device 50 each freely observe the event venue 10, and the area that each user is looking at, i.e., the area of ​​gaze, is different.

[0055] The information processing device 50 acquires image gaze area information from the user terminals (HMDs, smartphones, PCs, etc.) of a large number of image observing users 80 who observe image data (digital twin image data).

[0056] This gaze area information corresponds to the display information of the user terminal (HMD, smartphone, PC, etc.) of the image observing user 80, i.e., the image area information of the image observing user gaze area 90 shown in Figures 6 and 7, and this display image area information is transmitted to the information processing device 50.

[0057] Alternatively, a configuration may be adopted in which user gaze information detected by a sensor such as a camera attached to a user terminal (HMD, smartphone, PC, etc.) is transmitted to the information processing device 50. For example, a sensor (camera, etc.) for analyzing the user's gaze area is attached to the user terminal (HMD, smartphone, PC, etc.), and the user terminal (HMD, smartphone, PC, etc.) transmits the analyzed gaze information and gaze area information of the image observing user 80 to the information processing device 50 via a network.

[0058] The information processing device 50 receives gaze area information of a large number of image observing users 80 from a large number of user terminals (HMDs, smartphones, PCs, etc.) to which image data (digital twin image data) is distributed, and performs analysis processing of the gaze areas of the large number of image observing users 80 based on this received information.

[0059] Fig. 8 is a diagram showing an example of the gaze area analysis process of the image observing user 80 executed by the information processing device 50. The information processing device 50 executes the gaze area analysis process of the image observing user 80 to generate an image observing user gaze area map 95 as shown in Fig. 8. The image observing user gaze area map 95 shown in Fig. 8 is a map showing which area of ​​the side wall surface of the event venue 10, i.e., the side wall surface on which the advertisement display area 20 shown in Fig. 1 can be set, the image observing user 80 is gazing at.

[0060] The gaze area map shown in Fig. 8 is a map in which an area that is gazed at by more users is set to be brighter (whiter), and an area that is gazed at by fewer users is set to be darker (blacker). Note that, as described above with reference to Fig. 4, the gaze area map can be set in various ways, such as a map in which areas with high and low gaze levels are shown in different colors.

[0061] The information processing device 50 analyzes the gaze area information of the image observing user 80 received from a large number of user terminals (HMDs, smartphones, PCs, etc.) to which the image data (digital twin image data) is distributed, and generates an image observing user gaze area map 95 as shown in Figure 8.

[0062] Note that the image observation user gaze area map 95 shown in Fig. 8 is only a part of the map, and gaze area maps similar to that shown in Fig. 8 can be generated for side wall surfaces other than those shown in Fig. 8, other product arrangement shelves, etc. However, in order to speed up processing, it is desirable to configure the map generation by predefining only the advertisement displayable area as the map generation area.

[0063] 8 is an image observing user gaze area map 95 at a certain time, for example, time t1, generated by analyzing the gaze area of ​​an image observing user 80 at time t1. Since the gaze area of ​​the image observing user 80 moves over time, the image observing user gaze area map 95 is also updated over time.

[0064] As previously described with reference to Fig. 4, the information processing device 50 analyzes the gaze areas of real-space users 30 who are actual visitors to the event venue 10, and generates the real-space user gaze area map 60 shown in Fig. 4. Furthermore, as described with reference to Fig. 8, the information processing device 50 executes a gaze area analysis process for image observing users 80 who observe image data distributed by the information processing device 50 on their user terminals, and generates an image observing user gaze area map 95 as shown in Fig. 8.

[0065] That is, as shown in Fig. 9, the following two types of gaze area maps are generated: (a) a real-space user gaze area map 60 which is gaze area analysis data of the real-space user 30; and (b) an image-observing user gaze area map 95 which is gaze area analysis data of the image-observing user 80.

[0066] Furthermore, the information processing device 50 generates a combined user gaze area map (real-space user and image-observing user gaze area map) 100 by integrating these two types of gaze area maps corresponding to different users, as shown in FIG.

[0067] The information processing device 50 generates a composite user gaze area map (real-space user & image-observing user gaze area map) 100 shown in Fig. 10 by combining (adding) gaze area maps corresponding to the following two different users: (a) a real-space user gaze area map 60 which is gaze area analysis data of the real-space user 30; and (b) an image-observing user gaze area map 95 which is gaze area analysis data of the image-observing user 80.

[0068] The composite user gaze area map (real-space user & image-observing user gaze area map) 100 is a map, i.e., a composite map, that combines the gaze areas of the real-space users 30 and the image-observing users 80. In other words, it is a map generated by combining the gaze areas of the real-space users 30 who are viewing the event venue 10 in real space with the gaze areas of the image-observing users 80 who are viewing digital twin image data, which is copy image data of the event venue 10.

[0069] 10 is a gaze area map generated by analyzing the gaze areas of the real-space user 30 and the image-observing user 80 at a certain time, for example, time t1. Because the gaze areas of the real-space user 30 and the image-observing user 80 move over time, the gaze areas of the real-space user 30 and the image-observing user 80 also move over time, and the synthetic user gaze area map 100 is also updated over time.

[0070] The information processing device 50 uses the synthetic user gaze area map 100 for a certain period of time (tx to ty), for example, to calculate the gaze score S for each area of ​​the event venue 10. The gaze score S is a value that reflects the gaze level for each area in the real space. The gaze score S is a score for each area of ​​the event venue 10, for example, for each advertisement display area, and is determined based on the total number of real-space users 30 and image observing users 80 gazing at that area, and the total gaze time.

[0071] For example, the greater the total number of users (real space users 30 and image observing users 80) gazing at the area, and the longer the total gazing time, the higher the score value.

[0072] A specific example of the calculation process of the gaze score S executed by the information processing device 50 will be described with reference to Fig. 11. The information processing device 50 calculates the gaze score S for each advertisement displayable area 110 set in advance in the event venue 10.

[0073] The information processing device 50 calculates the gaze score S for each advertisement displayable area unit according to the following calculation formula (Formula 1).

[0074]

[0075] The parameters included in the above (Equation 1) are as follows: T ｓｔａｒｔ : Start time (seconds) of the gaze score S calculation period T ｅｎｄ t: End time (seconds) of the gaze score S calculation period ｉ : gaze time (seconds) of user i during the gaze score S calculation period U: number of users during the gaze score S calculation period (number of real-space users in the event venue + number of image-observing users observing images of the event venue) C: specified gaze score S initial value (for example, initial setting score value = (any value between 0 and 100)

[0076] As described above, the parameter C is a predetermined initial value of the gaze score S. For example, for all advertisement displayable areas, it is possible to set C (initial score S) = 0, C (initial score S) = 20, or C (initial score S) = 50.

[0077] The gaze score S changes over time. FIG. 12 shows an example of how the gaze scores of multiple advertisement displayable areas 110 installed at the event venue 10 change over time. FIG. 12 shows an example of how the gaze score S changes over the following times: (S01) Initial state (time t0, gaze score S is initialized to 0) (S02) After the event starts (time t1) (S03) After the event starts (time t2)

[0078] As shown in FIG. 12, the gaze score S is a value that changes over time. When the gaze score S is calculated using the above (Equation 1), the start time of the gaze score S calculation period (T ｓｔａｒｔ ) and the end time of the gaze score S calculation period (Tｅｎｄ ) to set an arbitrary period (T ｓｔａｒｔ ~T ｅｎｄ ) can be calculated as a gaze score S.

[0079] The value of the gaze score S for each advertisement displayable area calculated by the above calculation formula (Formula 1) is a value between 0 and 100, with 0 being the minimum score and 100 being the maximum score. The larger the value of the gaze score S calculated by the above calculation formula (Formula 1) for an area, the more users have been gazing at that area for a longer period of time.

[0080] In this way, the information processing device 50 calculates the gaze score S for each advertisement displayable area unit according to the above calculation formula (Formula 1).

[0081] A specific period (T ｓｔａｒｔ ~T ｅｎｄ An example of the calculation result of the gaze score S of the above-mentioned item is shown in FIG.

[0082] An example of the calculation result of the gaze score S shown in Figure 13 is as follows: The advertisement displayable area a, 110a has a gaze score S = 95, and is an advertisement displayable area with a high gaze score S value. The advertisement displayable area b, 110b has a gaze score S = 60, and is an advertisement displayable area with a medium gaze score S value. The advertisement displayable area c, 110c has a gaze score S = 98, and is an advertisement displayable area with a high gaze score S value. The advertisement displayable area d, 110d has a gaze score S = 32, and is an advertisement displayable area with a low gaze score S value. In this way, the information processing device 50 calculates the gaze score S for each advertisement displayable area according to the above calculation formula (Formula 1), as shown in Figure 13.

[0083] Furthermore, the information processing device 50 calculates an area-unit advertising price P based on the value of the area-unit gaze score S. A specific example of the calculation process of the area-unit advertising price P based on the value of the area-unit gaze score S executed by the information processing device 50 will be described with reference to Fig. 14 .

[0084] As shown in FIG. 14, the information processing device 50 displays advertisements for a specific period (Tｓｔａｒｔ ~T ｅｎｄ Based on the value of the gaze score S during that period (T ｓｔａｒｔ ~T ｅｎｄ ) is calculated according to the following formula (Formula 2).

[0085]

[0086] The parameters included in the above (Equation 2) are as follows: P ｌｏｗ : The minimum price within the predefined advertising price tolerance range P ｈｉｇｈ : The highest price in the predefined advertising price tolerance range

[0087] In this way, the information processing device 50 calculates the advertising price P for each area based on the value of the gaze score S for each advertisement displayable area 110 in accordance with the above calculation formula (Formula 2).

[0088] The analysis result data of the gaze score S for each advertisement displayable area 110 calculated by the information processing device 50 and the data of the advertisement price P for each advertisement displayable area 110 are provided to advertisement providers, advertisement management companies, etc. Specific examples of the data provided to advertisement providers, advertisement management companies, etc. will be described with reference to Figs. 15 and 16 .

[0089] 15 shows an example of analysis result data of the gaze score S for each advertisement displayable area 110 calculated by the information processing device 50. The gaze score S calculated according to the calculation formula (Formula 1) described above for each advertisement displayable area 110 set on the side wall surface of the event venue 10 is shown in shades of gray. The example shown in FIG. 15 is gaze score analysis result data indicating that dark area portions (areas close to black) have high gaze score S values ​​and bright area portions (areas close to white) have low gaze score S values.

[0090] By checking the analysis result data of such gaze score S, advertising providers, advertising management companies, etc. can determine which areas of the event venue 10 are being gazed at by many users.

[0091] 16 shows an example of data on the advertisement price P for each advertisement displayable area 110 determined based on the gaze score S for each advertisement displayable area 110 calculated by the information processing device 50. ｓｔａｒｔ ~T ｅｎｄ ) the specific period (T ｓｔａｒｔ ~T ｅｎｄ ) is the advertising price P per unit of the advertisement displayable area 110.

[0092] For each advertisement displayable area 110 set on the side wall surface of the event venue 10, the advertising price P calculated according to the calculation formula (Formula 2) explained above is shown.

[0093] For example, the advertisement displayable area a, 110a shown in Fig. 16 has a gaze score S of 95, which is an area with a high value of the gaze score S. This advertisement displayable area a, 110a indicates that the advertisement price is set to 5,000 yen / s, that is, the advertisement display price per second is set to 5,000 yen.

[0094] The advertisement displayable area b, 110b has a gaze score S = 40 and an advertising price = ¥2,000 / s, i.e., an area where the advertisement display price per second is set to ¥2,000. The advertisement displayable area c, 110c has a gaze score S = 80 and an advertising price = ¥4,000 / s, i.e., an area where the advertisement display price per second is set to ¥4,000. The advertisement displayable area d, 110d has a gaze score S = 65 and an advertising price = ¥3,000 / s, i.e., an area where the advertisement display price per second is set to ¥3,000.

[0095] For example, during the period when an advertisement is displayed by an advertisement provider or an advertisement management company, it is possible to calculate the gaze point score S of the users (real-space users 30 and image observing users 80) and the advertisement price P, and perform an advertisement price control process that dynamically varies the advertisement price according to the calculated advertisement price P. In other words, it is possible to perform a dynamic advertisement price variation type advertisement display in which the advertisement price varies according to the degree of user gaze.

[0096] In addition, when determining which advertising displayable area of ​​the event venue 10 to display the advertisement in before the advertisement is displayed by an advertising provider, advertising management company, etc., it is also possible to use data on the gaze point score S and advertising price P of users (real-space users 30 and image-observing users 80) per advertising displayable area in the past.

[0097] In addition, advertising display at the event venue 10 may be carried out using a display installed on the side wall of the event venue 10, or advertising images may be projected onto a screen installed on the side wall of the event venue 10.

[0098] When projecting an advertising image onto a screen installed on a side wall of the event venue 10, it is possible to use a robot 45, for example, as shown in Figure 17. The robot 45 is equipped with an advertising output function such as a projector, and moves around the event venue 10 to project an advertising image onto a screen installed on a side wall of the event venue 10.

[0099] In addition, instead of a robot 45 as shown in Figure 17, a configuration may be used in which a drone equipped with an advertising output function such as a projector is used to project advertising images from above the event venue 10 onto a screen installed on the side wall of the event venue 10.

[0100] 2. Processing Sequence of First Embodiment Executed by Information Processing Apparatus 50 of the Present Disclosure Next, a processing sequence of the first embodiment executed by the information processing apparatus 50 of the present disclosure will be described.

[0101] 9 and 10 , the information processing device 50 generates a composite user gaze area map (real-space user & image-observing user gaze area map) 100 by adding together the gaze area maps corresponding to the following two different users: (a) a real-space user gaze area map 60 which is gaze area analysis data of the real-space user 30; and (b) an image-observing user gaze area map 95 which is gaze area analysis data of the image-observing user 80.

[0102] The composite user gaze area map (real-space user & image observation user gaze area map) 100 is a map generated by combining the gaze area of ​​a real-space user 30 viewing the event venue 10 in real space and the gaze area of ​​an image observation user 80 viewing digital twin image data, which is copy image data of the event venue 10.

[0103] The information processing device 50 also performs a process of generating digital twin data including three-dimensional image data that reproduces the current state of the event venue 10 and distributing it via a network.

[0104] The processing sequence executed by the information processing device 50 of the present disclosure will be described below with reference to the flowcharts shown in Figures 18 and 19. Note that the processing according to the flowcharts shown in Figures 18 and 19 can be executed under the control of a control unit (data processing unit) configured by a CPU or the like having a program execution function in accordance with a program stored in a storage unit of the information processing device 50. The processing of each step of the flowcharts shown in Figures 18 and 19 will be described below in order.

[0105] (Step S101) First, in step S101, the information processing device 50 inputs sensor detection information such as a camera-captured image from the event venue terminal 40 at the event venue 10.

[0106] 3, a large number of cameras 41 are installed in the event venue 10, and they capture images of the event venue 10 from various angles. In addition, a robot 45 equipped with a camera also moves around, capturing images of the interior of the event venue 10 and images of real-space users 30.

[0107] The images taken by camera 41 and the images taken by the camera of robot 45 are transmitted to event venue terminal 40, and event venue terminal 40 transmits the images taken by each camera to information processing device 50 together with attribute information including the time the image was taken, camera position information, image taking direction, etc.

[0108] In step S101, the information processing device 50 receives from the event venue terminal 40 images captured by a camera, audio information picked up by a microphone, and information from various other sensors installed in the event venue.

[0109] (Steps S102 to S103) The processes of steps S102 to S103 and steps S121 to S123 shown in FIG. 18 can be executed in parallel by the information processing device 50.

[0110] The processing of steps S102 to S103 is processing for generating the real-space user gaze area map 60 previously described with reference to Figure 4, i.e., the real-space user gaze area map 60 which is gaze area analysis data of real-space users 30 who are visitors to the event venue 10.

[0111] The processing of steps S121 to S123 is a process of generating digital twin data including three-dimensional image data that reproduces the current state of the event venue 10 and distributing it via the network, and a process of generating the image observation user gaze area map 95 described above with reference to Figure 8, i.e., the image observation user gaze area map 95, which is gaze area analysis data of the image observation user 80 who observes the distributed data.

[0112] First, the processing of steps S102 to S103 will be described. In steps S102 to S103, the data processing unit of the information processing device 50 uses sensor detection information, such as camera-captured images, received from the event venue terminal 40 at the event venue 10 to analyze the gaze areas of real-space users 30 who are visitors to the event venue 10, and generates a real-space user gaze area map 60 as shown in Figure 4, which has been described above. As previously described with reference to Figure 4, the real-space user gaze area map 60 shown in Figure 4 is a map in which areas that are gazed at by more users are set to be brighter (whiter), and areas that are gazed at by fewer users are set to be darker (blacker).

[0113] (Steps S121 to S123) Next, the processing of steps S121 to S123 will be described.

[0114] In step S121, the data processing unit of the information processing device 50 generates digital twin data including three-dimensional image data that reproduces the current state of the event venue 10, so-called digital twin image data, using sensor detection information such as camera images received from the event venue terminal 40, and distributes it via the network.

[0115] Furthermore, in steps S122 to S123, the data processing unit of the information processing device 50 receives gaze area information of the image observing user 80 observing the distributed image from the user terminal (HMD, smartphone, PC, etc.) to which the digital twin data is distributed, and uses this received information to analyze the gaze area of ​​the image observing user 80 and generate an image observing user gaze area map 95 such as that shown in Figure 8 described above.

[0116] (Step S131) ​​Next, in step S131, the data processing unit of the information processing device 50 generates a composite user gaze area map (real-space user & image-observing user gaze area map) 100 by integrating the real-space user gaze area map 60 generated in step S103 and the image-observing user gaze area map 95 generated in step S123.

[0117] This process is the process described above with reference to Fig. 10. As described above with reference to Fig. 10, the information processing device 50 generates a composite user gaze area map (real-space user & image-observing user gaze area map) 100 shown in Fig. 10 by adding together the gaze area maps corresponding to the following two different users: (a) a real-space user gaze area map 60 which is gaze area analysis data of the real-space user 30; and (b) an image-observing user gaze area map 95 which is gaze area analysis data of the image-observing user 80.

[0118] The composite user gaze area map (real-space user & image-observing user gaze area map) 100 is a map generated by combining the gaze area of ​​the real-space user 30 and the gaze area of ​​the image-observing user 80, and is a map generated by combining the gaze area of ​​the real-space user 30 viewing the event venue 10 in real space and the gaze area of ​​the image-observing user 80 viewing digital twin image data, which is copy image data of the event venue 10.

[0119] (Step S151) Next, in step S151, the data processing unit of the information processing device 50 calculates the gaze score S for each advertisement displayable area using the composite user gaze area map (real-space user & image observation user gaze area map) 100 generated in step S131.

[0120] This process is the process described above with reference to Fig. 11. The data processing unit of the information processing device 50 calculates the gaze score S for each advertisement displayable area 110 set in advance in the event venue 10. As described above with reference to Fig. 11, the gaze score S is calculated according to the following calculation formula (Formula 1).

[0121]

[0122] The parameters included in the above (Equation 1) are as follows: T ｓｔａｒｔ : Start time (seconds) of the gaze score S calculation period T ｅｎｄ t: End time (seconds) of the gaze score S calculation period ｉ : gaze time (seconds) of user i during the gaze score S calculation period U: number of users during the gaze score S calculation period (number of real-space users in the event venue + number of image-observing users observing images of the event venue) C: specified gaze score S initial value (for example, initial setting score value = (any value between 0 and 100)

[0123] (Step S152) Next, in step S152, the data processing unit of the information processing device 50 calculates an advertising price P for each area based on the value of the gaze score S for each area calculated in step S151.

[0124] This process is the process described above with reference to Fig. 14. As described with reference to Fig. 14, the information processing device 50 performs the process of displaying advertisements for a specific period (T ｓｔａｒｔ ~T ｅｎｄ Based on the value of the gaze score S during that period (T ｓｔａｒｔ ~T ｅｎｄ ) is calculated according to the following formula (Formula 2).

[0125]

[0126] The parameters included in the above (Equation 2) are as follows: P ｌｏｗ : The minimum price within the predefined advertising price tolerance range P ｈｉｇｈ : The highest price in the predefined advertising price tolerance range

[0127] In this way, in step S152, the information processing device 50 calculates the advertising price P for each area based on the value of the gaze score S for each advertisement displayable area 110 in accordance with the above calculation formula (Formula 2).

[0128] (Steps S153 to S154) The next steps S153 to S154 are processes for using the analysis result data of the gaze score S for each advertisement displayable area 110 calculated in steps S151 to S152 and the data of the advertisement price P for each advertisement displayable area 110.

[0129] In step S153, the analysis result data of the gaze score S for each advertisement displayable area 110 calculated in steps S151 and S152 and the data of the advertisement price P for each advertisement displayable area 110 are provided to the advertisement provider or advertisement management company.

[0130] In step S154, the advertisement fee is collected from the advertisement provider or the advertisement management company according to the calculated advertisement price P. Alternatively, in response to a request from the advertisement provider or the advertisement management company, a process of displaying an advertisement in a designated advertisement displayable area is performed.

[0131] In this way, the information processing device 50 of the present disclosure generates a composite user gaze area map (real-space user & image-observing user gaze area map) 100 that integrates these two types of gaze area maps: (a) a real-space user gaze area map 60, which is gaze area analysis data of the real-space user 30; and (b) an image-observing user gaze area map 95, which is gaze area analysis data of the image-observing user 80.

[0132] The information processing device 50 uses this composite user gaze area map (real-space user & image observation user gaze area map) 100 to calculate a gaze score S according to the gaze level of each advertisement displayable area 110 of the event venue 10, and further determines an advertising price P for each advertisement displayable area 110 based on the calculated gaze score S.

[0133] The advertising price P determined in this way is a price calculated based on the gaze score S, which is calculated taking into consideration not only the gaze areas of the real-space users 30 who are visitors to the event venue 10, but also the gaze areas of the image observing users 80 who are viewing the digital twin image data, and is a price that reliably reflects the gaze levels of more users with high accuracy. In other words, it becomes possible to apply an advertising price that reflects the advertising effect with high accuracy.

[0134] In addition, the gaze point score S and the advertising price P are calculated over an arbitrary period (T ｓｔａｒｔ ~T ｅｎｄ ) unit, making it possible to display advertisements with dynamic advertising price fluctuations that change the advertising price depending on the user's level of attention during any set period.

[0135] In the above explanation, a single information processing device 50 is described as performing the process of generating a user gaze area map and the process of generating and distributing digital twin data, but these processes may also be performed on individual information processing devices (cloud servers).

[0136] That is, as shown in FIG. 20, an information processing device 50a that executes the process of generating the user gaze area map and an information processing device 50b that executes the process of generating and distributing digital twin data may be configured as separate devices.

[0137] The information processing device a, 50a generates a composite user gaze area map (real space user & image observing user gaze area map) 100 by integrating these two types of gaze area maps: (a) a real space user gaze area map 60 which is gaze area analysis data of the real space user 30; and (b) an image observing user gaze area map 95 which is gaze area analysis data of the image observing user 80.

[0138] Meanwhile, information processing devices b and 50b execute a process of generating digital twin data including so-called digital twin image data, which is three-dimensional image data that recreates the current state of the event venue 10, using sensor detection information such as camera-captured images received from the event venue terminal 40, and distributing the generated digital twin data via the network. In this way, each process may be executed by an individual information processing device.

[0139] 3. (Second embodiment) Example of analyzing a user's gaze area to detect a problem and execute a warning process Next, as a second embodiment of the present disclosure, an example of analyzing a user's gaze area to detect a problem and execute a warning process will be described.

[0140] FIG. 21 is a diagram showing a factory 120 that manufactures a certain product. In this factory 120, products in the manufacturing process are moved on a conveyer belt, and multiple workers (= real-space users 131) each perform tasks such as attaching their respective parts to the products on the conveyer belt. In this example, the workers in the factory 120 are users who directly observe the real space, i.e., the real-space users 131. The real-space users 131 are users who directly observe the real space. In other words, they are users who directly observe the real space with their naked eyes, rather than users who observe images of the real space, such as two-dimensional images or three-dimensional images.

[0141] 21 , a large number of cameras 121 are installed in a factory 120, and they capture images of workers (= real-space users 131) and others in the factory 121 from various angles. Furthermore, a robot 122 equipped with a camera also moves around and captures images of workers (= real-space users 131) and others in the factory 120.

[0142] In this embodiment, a process is executed to determine whether real-space users 131, who are workers in the factory 120, are working while reliably looking at the product to be manufactured (gaze target), i.e., the work area, and to detect workers (real-space users 131) who are working without looking at the work area and to issue a warning. That is, in this embodiment, a process is executed to determine whether the workers (real-space users 131) are looking at the area they should be looking at based on the gaze area map, and to output a warning if they are not looking. Note that the area to be looked at does not only include the work area, but also various other areas, such as an area where there is a risk of collision with a moving object such as a robot.

[0143] 21, real-space user a 131a, who is a worker, is working while looking at the product to be manufactured, i.e., the work area, while real-space user b 131b is working without looking at the product in his / her work area. In this embodiment, if real-space user b 131b is detected working without looking at the product in his / her work area, a warning is output.

[0144] The determination of whether the real-space users 131 who are workers are looking at the product to be manufactured is made by analyzing the gaze area of ​​each real-space user 131.

[0145] The gaze area analysis process for each real-space user 131 is executed by an external information processing device 50, as in the previously described Example 1. As shown in Fig. 22 , a factory terminal 125, which is a local terminal installed in a factory 120, and the information processing device 50, which is a gaze area analysis server, are connected via a communication network.

[0146] Images captured by cameras 121 installed at various positions within the factory 120 and images captured by a camera of the robot 122 are transmitted to a factory terminal 125. The factory terminal 125 transmits the images captured by each camera together with attribute information including the image capture time, camera position information, image capture direction, etc. to the information processing device 50. The information processing device 50 is, for example, a cloud server, and is a gaze area analysis server that executes gaze area analysis processing.

[0147] The information processing device 50 analyzes the images captured by each camera in the factory 120 received from the factory terminal 125 and analyzes the line of sight and gaze area of ​​the real-space user 131 who is a worker in the factory 120 .

[0148] 23 shows an example of the gaze area analysis results of real-space users 131 who are workers in the factory 120. The information processing device 50 generates a real-space user gaze area map 135 as shown in Fig. 23. The real-space user gaze area map 135 shown in Fig. 23 is a map that shows the gaze areas of each real-space user 131 who is a worker in the factory 120.

[0149] 23 is a map in which areas that are being gazed at by each of the real-space users 131 who are workers in the factory 120 are set as bright (white) areas, and areas that are not being gazed at are set as dark (black) areas. Note that the gaze area map shown in Fig. 23 is just one example, and various settings are possible, such as a map in which areas with high and low gaze levels are shown in different colors.

[0150] 23 is the real-space user gaze area map 135 at time t1, which is generated by analyzing the gaze area of ​​the real-space user 131, who is the worker, at a certain time, for example, time t1. Because the gaze area of ​​the real-space user 131 moves over time, the real-space user gaze area map 135 is also updated over time.

[0151] 24 , the information processing device 50 analyzes the gaze area of ​​each worker based on the generated real-space user gaze area map 135. Specifically, for example, a worker who is working without looking at the product to be manufactured is detected, and the detection result is transmitted to the factory terminal 125. For example, in the example shown in FIG. 24 , real-space user b 131b who is working without looking at the product to be manufactured is detected, and the detection result is transmitted to the factory terminal 125.

[0152] In addition to the analysis results, the real-space user gaze area map 135 generated by the information processing device 50 may also be transmitted to the factory terminal 125 .

[0153] The factory terminal 125 inputs the gaze area analysis results generated by the information processing device 50, i.e., detection information of workers who are working without looking at the product to be manufactured, and outputs a warning based on the detection information. In the example shown in Fig. 24, a warning is output to real-space user b 131b who is working without looking at the product to be manufactured. For example, as shown in Fig. 24, a warning output device 126 installed in front of real-space user b 131b is activated to output a warning light or a warning sound.

[0154] This warning output process allows the real-space user b, 131b, who is a worker in the factory 120, to realize that he or she has not been looking at the product, and can change his or her line of sight and work while looking at the product.

[0155] Note that the data transmitted from the information processing device 50 to the factory terminal 125 may be the real-space user gaze area map 135 only, and the processing for identifying the person to whom a warning is to be issued may be performed by the factory terminal 125 .

[0156] In this case, the factory terminal 125 detects real-space users b, 131b who are working without looking at the product being manufactured based on the real-space user gaze area map 135 received from the information processing device 50, and outputs a warning to the real-space users b, 131b based on this detection result.

[0157] The monitoring of workers in the factory 120 may be performed by an external factory monitoring center 140. A specific example of the monitoring process in the factory monitoring center 140 will be described with reference to FIG.

[0158] 25, the factory terminal 125, the information processing device 50, and the factory monitoring center 140 (=user terminal in the factory monitoring center 140) are configured to be able to communicate via a network. The information processing device 50 generates digital twin data including three-dimensional image data that reproduces the current state of the factory 120, and transmits it via the network to the factory monitoring center 140 (=user terminal in the factory monitoring center 140).

[0159] A plurality of monitoring users (image observing users 141) are stationed at the factory monitoring center 140 to monitor the inside of the factory 120. Fig. 25 shows an image observing user a (monitoring user a) 141a and an image observing user b (monitoring user b) 141b.

[0160] Image observing user a (monitoring user a) 141a displays the digital twin image data received from the information processing device 50 on a PC 72, which is a user terminal, and monitors the work status of workers in the factory 120. Also, image observing user b (monitoring user b) 141b displays the digital twin image data received from the information processing device 50 on an HMD 70, which is a user terminal, and monitors the work status of workers in the factory 120.

[0161] Each of the multiple image observing users (monitoring users) 141 monitors the work status of the worker, but if their concentration drops, they may stop observing the work of the worker. In this embodiment, the occurrence of such a monitoring error is detected and a warning is issued.

[0162] As shown in Figure 26, first, in step S21, digital twin data including three-dimensional image data that reproduces the current state of the factory 120 generated by the information processing device 50 is transmitted to a user terminal (PC, HMD, etc.) in the factory monitoring center 140.

[0163] Each of image observing user a (monitoring user a) 141a and image observing user b (monitoring user b) 141b displays the digital twin image data received from the information processing device 50 on a user terminal (PC, HMD, etc.) to monitor the work status of workers in the factory 120.

[0164] Next, in step S22, the factory monitoring center 140 transmits gaze area information of each image observing user (monitoring user) to the information processing device 50. That is, the gaze area information of each image observing user (monitoring user) acquired by the user terminal (PC, HMD, etc.) of each image observing user (monitoring user) 141 in the factory monitoring center 140 is transmitted.

[0165] This gaze area information is transmitted to the information processing device 50 by utilizing display information of the user terminal (PC, HMD, etc.) of the image observing user (monitoring user) 141. Alternatively, a configuration may be adopted in which user gaze information detected by a sensor such as a camera attached to the user terminal (PC, HMD, etc.) is transmitted to the information processing device 50. For example, a sensor (camera, etc.) for analyzing the user's gaze area is attached to the user terminal (PC, HMD, etc.), and the user terminal (PC, HMD, etc.) transmits the analyzed gaze information and gaze area information of the image observing user 141 to the information processing device 50 via a network.

[0166] The information processing device 50 receives gaze area information of each of the multiple image observing users (monitoring users) 141 from the user terminal (PC, HMD, etc.) of the factory monitoring center 140, which is the destination of the image data (digital twin image data), and performs analysis processing of the gaze areas of each of the multiple image observing users (monitoring users) 141 based on this received information.

[0167] In step S23, the information processing device 50 transmits the analysis results of the gaze areas of each of the multiple image observing users (monitoring users) 141 and issues a warning as necessary. That is, a warning is output to the image observing users (monitoring users) 141 who are not looking at the area they should be watching.

[0168] FIG. 27 shows an example of an image observing user gaze area map generated as a result of the gaze area analysis process of the image observing user (monitoring user) 141 executed by the information processing device 50.

[0169] The upper part of Fig. 27 shows image observing user gaze area maps a, 145a generated by analyzing the gaze area of ​​image observing user a (monitoring user a) 141a, and the lower part of Fig. 27 shows image observing user gaze area maps b, 145b generated by analyzing the gaze area of ​​image observing user b (monitoring user b) 141b.

[0170] The gaze area map shown in Fig. 27 is a map in which an area that is being gazed at by the image observing user (monitoring user) 141 is set to be brighter (whiter) and an area that is not being gazed at is set to be darker (blacker). Note that the gaze area map shown in Fig. 27 is just an example, and various settings are possible, such as a map in which areas with high and low gaze levels are shown in different colors.

[0171] Of the two gaze area maps shown in Figure 27, in the upper image observation user gaze area map a, 145a, a gaze area is set at the hand of a worker in factory 120, and it can be seen that image observation user a (monitoring user a) 141a is closely monitoring the worker.

[0172] On the other hand, in the lower image observing user gaze area map b, 145b, the gaze area is located in a location other than the worker's hands within the factory 120, and it can be seen that image observing user b (monitoring user b) 141b is not watching the worker's work.

[0173] In such a case, the information processing device a 50 transmits warning information to the user terminal (HMD 70) of the image observing user b (monitoring user b) 141b and executes warning output.

[0174] For example, as shown in FIG. 28, a warning message such as "The stop area is misaligned" is output via the speaker of the user terminal (HMD 70) of image observing user b (monitoring user b) 141b.

[0175] By carrying out such processing, the image observing user b (monitoring user b) 141b can prevent monitoring errors by the observer.

[0176] The information processing device 50 that executes the gaze area analysis process further generates a combined user gaze area map (real-space user and image-observing user gaze area map) 150 as shown in FIG.

[0177] The information processing device 50 generates a composite user gaze area map (real-space user & image observing user gaze area map) 150 shown in Fig. 29 by adding together the gaze area maps corresponding to the following two different users: (a) a real-space user gaze area map 135 which is gaze area analysis data of a real-space user (worker) 131; and (b) an image observing user gaze area map 145 which is gaze area analysis data of an image observing user (monitor) 141.

[0178] The composite user gaze area map (real-space user & image observing user gaze area map) 150 is a map generated by combining the gaze area of ​​the real-space user (worker) 131 and the gaze area of ​​the image observing user (monitor) 141. In other words, it is a map generated by combining the gaze area of ​​the real-space user 131, who is a worker in the factory 120, and the gaze area of ​​the image observing user 141, who is a monitor at the factory monitoring center 140 who is viewing the digital twin image data, which is copy image data of the factory 120.

[0179] 29 is a gaze area map generated by analyzing the gaze area of ​​the real space user 131 and the gaze area of ​​the image observing user 141 at a certain time, for example, time t1. Because the gaze area of ​​the real space user 131 and the gaze area of ​​the image observing user 141 move over time, the gaze area of ​​the synthetic user gaze area map 150 is also updated over time.

[0180] 29, and if no stop area is set in an area to be watched, for example, in a worker's work area, the information processing device 50 performs a warning output process to all real-space users 131 who are workers in the factory 120 and image observing users 141 who are monitors in the factory monitoring center 140. This warning output process enables all users (workers and monitors) to pay attention to the worker's work area.

[0181] 4. Processing Sequence of Second Embodiment Executed by Information Processing Apparatus 50 of the Present Disclosure Next, a processing sequence of the second embodiment executed by the information processing apparatus 50 of the present disclosure will be described.

[0182] In the second embodiment, similarly to the first embodiment described above, the information processing device 50 generates gaze area maps corresponding to the following two different users: (a) a real-space user gaze area map 135 which is gaze area analysis data of a real-space user (worker) 131; and (b) an image observing user gaze area map 145 which is gaze area analysis data of an image observing user (monitor) 141.

[0183] Furthermore, these two maps are added together to generate a combined user gaze area map (real space user & image observing user gaze area map) 150 .

[0184] The information processing device 50 also performs a process of generating digital twin data including three-dimensional image data that reproduces the current state of the factory 120 and distributing it via a network.

[0185] The processing sequence of Example 2 executed by the information processing device 50 of the present disclosure will be described below with reference to the flowcharts shown in Figures 30 to 33. Note that the processing according to the flowcharts shown in Figures 30 to 33 can be executed under the control of a control unit (data processing unit) configured by a CPU or the like having a program execution function in accordance with a program stored in a storage unit of the information processing device 50. The processing of each step of the flowcharts shown in Figures 30 to 33 will be described below in order.

[0186] (Step S201) First, in step S201, the information processing device 50 inputs sensor detection information such as a camera-captured image from the factory terminal 125 of the factory 120.

[0187] 21 and 22, a large number of cameras 121 are installed in the factory 120, and they capture images of the factory 120 from various angles. Furthermore, a robot 122 equipped with a camera also moves around and captures images of the interior of the factory 120 and images of real-space users 131 who are workers.

[0188] The images taken by the camera 121 and the images taken by the camera of the robot 122 are transmitted to the factory terminal 125, and the factory terminal 125 transmits the images taken by each camera to the information processing device 50 together with attribute information including the time the image was taken, camera position information, image taking direction, etc.

[0189] In step S201, the information processing device 50 receives from the factory terminal 125 images captured by a camera, audio information acquired by a microphone, and information from various other sensors installed in the factory.

[0190] (Steps S202 to S203) In steps S202 to S203, the data processing unit of the information processing device 50 analyzes the gaze areas of real-space users 131 who are workers at the factory 120, using sensor detection information such as camera-captured images received from the factory terminal 125 of the factory 120, and generates the real-space user gaze area map 135 as previously described and shown in Fig. 23. As previously described with reference to Fig. 23, the real-space user gaze area map 135 shown in Fig. 23 is a map in which areas that are gazed at by more users are set brighter (whiter) and areas that are gazed at by fewer users are set darker (blacker).

[0191] (Steps S204 to S206) The data processing unit of the information processing device 50 executes the following processes in steps S204 to S206.

[0192] In step S204, the gaze area of ​​each real-space user 131 who is a worker in the factory 120 and the amount of deviation from the gaze-requiring area are analyzed based on the real-space user gaze area map 135 generated in step S203.

[0193] In step S205, it is determined whether or not there is a user whose gaze area is greater than or equal to a threshold value and whose gaze-requiring area is greater than or equal to a threshold value. If a user whose gaze area is greater than or equal to a threshold value and whose gaze-requiring area is greater than or equal to a threshold value is detected, the process proceeds to step S206. If no user is detected, the process returns to step S201, and a new camera-captured image or the like is input, and the processes from step S201 onward are repeated.

[0194] In step S205, if a real-space user 131 is detected whose gaze area and gaze-requiring area are displaced by a threshold or more, a warning is output in step S206 to notify the user that the gaze area is displaced. Alternatively, a warning output request is sent to the factory terminal 125 to cause the factory terminal 125 to output the warning. This process corresponds to the process described above with reference to FIG. 24.

[0195] After these processes, the process returns to step S201, new camera-captured images and the like are input, and the processes from step S201 onwards are repeated.

[0196] Next, referring to the flowchart shown in Figure 31, we will explain the sequence of the process of generating three-dimensional image data, so-called digital twin image data, executed by the information processing device 50, the process of transmitting it to the factory monitoring center 140, and the process of analyzing the gaze area of ​​the image observing user 141, who is the monitor.

[0197] The information processing device 50 can execute the process according to the flow shown in Fig. 31 in parallel with the process of the flow shown in Fig. 30. The process of each step of the flow shown in Fig. 31 will be described below in order.

[0198] (Steps S251 to S254) In step S251, the data processing unit of the information processing device 50 receives sensor detection information such as a camera-captured image from the factory terminal 125.

[0199] Next, in step S252, the data processing unit of the information processing device 50 generates digital twin data including three-dimensional image data that reproduces the current state of the factory 120, so-called digital twin image data, using sensor detection information such as camera images received from the factory terminal 125, and distributes the data to the factory monitoring center 140 via the network.

[0200] Furthermore, in steps S253 to S254, the data processing unit of the information processing device 50 receives gaze area information of the image observing user 141 observing the distributed image from the user terminal (PC, HMD, etc.) of the image observing user 141, who is a monitor within the factory monitoring center 140 to which the digital twin data is distributed, and uses this received information to analyze the gaze area of ​​the image observing user 141 and generate an image observing user gaze area map 145 such as that shown in Figure 27 described above.

[0201] (Steps S255 to S257) Next, the data processing unit of the information processing device 50 executes the following processes in steps S255 to S257.

[0202] In step S255, based on the image observing user gaze area map 145 generated in step S254, the gaze area of ​​each image observing user 141 who is a monitor at the factory monitoring center 140 and the amount of deviation from the gaze-requiring area are analyzed.

[0203] In step S256, it is determined whether or not there is a user for whom the deviation between the gaze area of ​​the image observing user 141, who is a monitor at the factory monitoring center 140, and the area requiring attention is greater than or equal to a threshold value. If a user for whom the deviation between the gaze area of ​​the image observing user 141 and the area requiring attention is greater than or equal to the threshold value is detected, the process proceeds to step S257. If no user is detected, the process returns to step S251, a new camera-captured image or the like is input, and the processes from step S251 onwards are repeated.

[0204] In step S256, if a user is detected whose gaze area of ​​the image observing user 141 is displaced from the gaze-requiring area by a threshold value or more, a warning is output in step S257 to notify the user that the gaze area is displaced. Alternatively, a warning output request is sent to the factory monitoring center 140 to have the user terminal output a warning. This process corresponds to the process described above with reference to FIG. 28.

[0205] After these processes, the process returns to step S251, new images captured by the camera are input, and the processes from step S251 onwards are repeated.

[0206] Next, with reference to the flow shown in Fig. 32 , another process executed by the information processing device 50 will be described, namely, a process sequence for generating a composite user gaze area map (real space user & image observing user gaze area map) 150 by adding together (a) a real space user gaze area map 135 which is gaze area analysis data of the real space user (worker) 131, and (b) an image observing user gaze area map 145 which is gaze area analysis data of the image observing user (monitor) 141, and then performing gaze area analysis processing using the generated composite user gaze area map (real space user & image observing user gaze area map) 150. The processing of each step of the flow shown in Fig. 32 will be described below.

[0207] (Step S281) The processing of step S281 is executed after the processing of generating the real-space user gaze area map 135 in step S203 of the flow shown in FIG. 30 and the processing of generating the image observation user gaze area map 145 in step S254 of the flow shown in FIG. 31 are completed.

[0208] In step S281, the data processing unit of the information processing device 50 generates a composite user gaze area map (real-space user & image observation user gaze area map) 150 by integrating the real-space user gaze area map 135 generated in step S203 and the image observation user gaze area map 145 generated in step S254.

[0209] This process is the process described above with reference to Fig. 29. As described above with reference to Fig. 29, the information processing device 50 generates a composite user gaze area map (real-space user & image observing user gaze area map) 150 shown in Fig. 29 by adding together the gaze area maps corresponding to the following two different users: (a) a real-space user gaze area map 135 which is gaze area analysis data of a real-space user 131 who is a worker in the factory 120; and (b) an image observing user gaze area map 145 which is gaze area analysis data of an image observing user 141 who is a supervisor at the factory monitoring center 140.

[0210] The composite user gaze area map (real-space user & image observation user gaze area map) 150 is a map generated by combining the gaze area of ​​a real-space user 131 who is a worker in the factory 120 and the gaze area of ​​an image observation user 141 who is a supervisor at the factory monitoring center 140. The composite user gaze area map (real-space user & image observation user gaze area map) 150 is a map generated by combining the gaze area of ​​a real-space user 131 who is a worker in the factory 120 and the gaze area of ​​an image observation user 141 who is a supervisor at the factory monitoring center 140 who is viewing digital twin image data, which is copy image data of the factory 120.

[0211] (Steps S282 to S284) Next, the data processing unit of the information processing device 50 executes the following processes in steps S282 to S284.

[0212] In step S282, based on the composite user gaze area map (real-space user & image observation user gaze area map) 150 generated in step S281, the amount of deviation from the area requiring gaze is analyzed for the gaze area of ​​the real-space user 131 who is a worker in the factory 120 and the gaze area of ​​the image observation user 141 who is a supervisor at the factory monitoring center 140.

[0213] In step S283, it is determined whether there are any users for whom the deviation between the gaze area of ​​the real-space user 131 who is a worker in the factory 120 and the gaze area of ​​the image observing user 141 who is a monitor at the factory monitoring center 140 and the gaze-required area is equal to or greater than a threshold value. If a user for whom the deviation is equal to or greater than the threshold value is detected, the process proceeds to step S284. If no users are detected, the process returns to step S201 and step S251, where a new camera-captured image or the like is input, and the processes from step S201 to step S251 are repeated.

[0214] In step S283, if a user is detected whose gaze area of ​​the real-space user 131 who is a worker in the factory 120 and the gaze area of ​​the image observing user 141 who is a monitor at the factory monitoring center 140 deviates from the gaze-required area by a threshold value or more, a warning is output in step S283 to notify the user that the gaze areas are misaligned. Alternatively, a warning output request is sent to the factory monitoring center 140 to have the user terminal output a warning.

[0215] After these processes, the process returns to step S201 and step S251, new images captured by the camera are input, and the processes from step S201 to step S251 onwards are repeated.

[0216] By executing the processing according to the flow shown in Figures 30 to 32 in this way, it is possible to immediately detect and warn workers or supervisors who are not paying attention to areas that require attention, thereby realizing safe and reliable work in the factory.

[0217] 5. Configuration Examples of Devices Constituting the Information Processing System of the Present Disclosure Next, configuration examples of devices constituting the information processing system of the present disclosure will be described.

[0218] A configuration example of an apparatus that constitutes an information processing system according to the present disclosure will be described with reference to FIGS. 33 and 34 .

[0219] Fig. 33 shows a camera 210, a robot 220, and a local terminal 230, which are local components of a real space where a real-space user exists and which configures the information processing system of the present disclosure, such as the event venue 10 described with reference to Fig. 1 or the factory 120 described with reference to Fig. 21. The local terminal 230 corresponds to the event venue terminal 40 shown in Fig. 3 and the like or the factory terminal 125 shown in Fig. 22 and the like.

[0220] The camera 210 has a control unit 211, an imaging unit 212, and a communication unit 213. The control unit 211 controls imaging, such as starting and stopping image capture, controlling the camera direction, controlling the imaging area, and zooming, and further controls communication with the local terminal 230 and analyzes commands received from the local terminal 230.

[0221] The imaging unit 212 captures an image under the control of the control unit 211. The captured image is transmitted to the local terminal 230 via the communication unit 213. The communication unit 213 communicates with the local terminal 230. The image captured by the imaging unit 212 is transmitted to the local terminal 230 under the control of the control unit 211. The communication unit 213 also receives control commands and the like from the local terminal 230 and inputs them to the control unit 211.

[0222] Next, a description will be given of the configuration of the robot 220. The robot 220 includes a control unit 221, a driving unit 222, an imaging unit 223, an output unit 224, a communication unit 225, and a storage unit 226.

[0223] The control unit 221 executes drive control, image capture control, advertisement output control, communication control processing, etc. of the robot 220. The drive unit 222 executes travel control, etc. of the robot 220 under the control of the control unit 221. The imaging unit 223 executes image capture processing under the control of the control unit 221.

[0224] The output unit 224 executes advertisement output processing and the like under the control of the control unit 221. The communication unit 225 executes communication with the local terminal 230. The image captured by the imaging unit 223 is transmitted to the local terminal 230 under the control of the control unit 221. The communication unit 225 also executes processing such as receiving control commands and the like from the local terminal 230 and inputting them to the control unit 221. The memory unit 226 stores control programs and parameters required for travel control of the robot 220, data output control such as advertisements, communication control, and the like, and further stores advertisement data and the like to be output via the output unit 224.

[0225] Next, a description will be given of the configuration of the local terminal 230. As described above, the local terminal 230 corresponds to the event site terminal 40 shown in Fig. 3 and the like, or the factory terminal 125 shown in Fig. 22 and the like.

[0226] As shown in FIG. 33, the local terminal 230 includes a control unit 231 , a communication unit 232 , an input unit 233 , an output unit 234 , and a storage unit 235 .

[0227] The control unit 231 controls communication with the camera 210 and the robot 220, and also controls communication with a device such as a cloud server that performs analysis processing of the gaze area, that is, the information processing device 50 shown in Fig. 3 or 22. Furthermore, it also performs analysis processing of data received from the information processing device 50 shown in Fig. 3 or 22, advertisement output control processing based on the analysis results, warning output processing, and the like.

[0228] The communication unit 232 performs communication processing with the camera 210 and the robot 220, and further performs communication processing with a device such as a cloud server that performs analysis processing of the gaze area, i.e., the information processing device 50 shown in Figures 3 and 22.

[0229] The input unit 233 is an input unit such as a keyboard, mouse, button, switch, etc., which can be operated by a user such as an operator, and receives information such as information to start, stop, and set up processing. The output unit 234 is composed of a display unit, speaker, etc., which are used to output warning messages, various images, text, etc. The storage unit 235 is used as a storage area for programs and parameters of processing executed on the local terminal 230. It is also used as a storage area for advertisements to be output and warning messages.

[0230] Next, the configurations of the information processing device 250 and the user terminal 280 will be described with reference to Fig. 34. The information processing device 250 shown in Fig. 34 is a device such as a cloud server that executes analysis processing of the gaze area, and is a device equivalent to the information processing device 50 shown in Fig. 3 or Fig. 22.

[0231] The user terminal 280 is a terminal having a display unit that displays three-dimensional image data, i.e., digital twin image data, that reproduces the real space generated by the information processing device 250, i.e., the information processing device 50 shown in Figures 3 and 22, and is composed of an HMD, PC, smartphone, etc.

[0232] The information processing device 250 has a communication unit 251, a control unit (data processing unit) 252, and a storage unit 253. The control unit (data processing unit) 252 has a data generation unit (digital twin data etc. generation unit) 261, a gaze area analysis unit 262, a gaze score calculation unit 263, an advertisement price calculation unit 264, an advertisement output control unit 265, and a warning output control unit 266.

[0233] The communication unit 251 communicates with the local terminal 230 and the user terminal (HMD, PC, smartphone, etc.) 280. Sensor detection information such as camera-captured images is input from the local terminal 230 via the communication unit 251. The sensor detection information such as camera-captured images is used for analyzing the gaze area and for generating digital twin data including three-dimensional image data that is reproduction data of the event venue 10 shown in FIG. 1 and the factory shown in FIG. 21 , which are local areas.

[0234] Furthermore, the information processing device 250 outputs advertisement data, advertisement output control information, warning output control information, and the like to the local terminal 230 via the communication unit 251 .

[0235] Furthermore, digital twin data including three-dimensional image data that is reproduction data of the event venue 10 shown in Fig. 1 and the factory shown in Fig. 21 is transmitted to the user terminal (HMD, PC, smartphone, etc.) 280. Also, information necessary for analyzing the user's gaze area, such as line of sight information of the user viewing the digital twin data, is transmitted from the user terminal (HMD, PC, smartphone, etc.) 280.

[0236] The data generation unit (digital twin data generation unit) 261 of the control unit (data processing unit) 252 generates digital twin data including three-dimensional image data that is reproduction data of the event venue 10 shown in Fig. 1 and the factory shown in Fig. 21. The generated digital twin data is transmitted to a user terminal (HMD, PC, smartphone, etc.) 280 via the communication unit 251.

[0237] The gaze area analysis unit 262 inputs sensor detection information such as camera-captured images from the local terminal 230 and analyzes the gaze area of ​​the real-space user. As a result of this analysis, for example, a real-space user gaze area map is generated. For example, the unit executes processing to generate the real-space user gaze area map 60 shown in FIG. 4 or the real-space user gaze area map 135 shown in FIG. 23.

[0238] Furthermore, the gaze area analysis unit 262 inputs gaze area analysis data, such as line of sight information of the image observing user viewing the digital twin data, from the user terminal (HMD, PC, smartphone, etc.) 280, and executes analysis processing of the gaze area of ​​the image observing user. As a result of this analysis, for example, an image observing user gaze area map is generated. For example, processing is executed to generate the image observing user gaze area map 95 shown in FIG. 8 or the image observing user gaze area map 145 shown in FIG. 27.

[0239] The gaze score calculation unit 263 of the control unit (data processing unit) 252 calculates the gaze score S for each predetermined area, for example, for each advertisement displayable area, using the analysis result of the gaze area analysis unit 262. The gaze score calculation unit 263 calculates the gaze score S for each predetermined area, for example, for each advertisement displayable area. Specifically, the gaze score calculation unit 263 executes the calculation process of the gaze score S described above with reference to FIG. 11 .

[0240] The advertising price calculation unit 264 calculates the advertising price P for each predetermined area, for example, for each advertisement displayable area, by using the gaze score S for each predetermined area calculated by the gaze score calculation unit 263. Specifically, the advertising price calculation unit 264 executes the calculation process for the advertising price P described above with reference to FIG.

[0241] In response to a request from the advertising provider, the advertising output control unit 265 transmits control information and advertising data to the local terminal 230 to execute advertising display processing for an area determined based on the gaze score S and advertising price P, thereby executing advertising display control.

[0242] The warning output control unit 266 executes control to output a warning to a user who is not gazing at an area that should be gazed at, based on the gaze area analysis result of the gaze area analysis unit 262. The warning output is executed for a real-space user using the local terminal 230, and for an image-observing user using the user terminal 280.

[0243] The storage unit 253 is used as a storage area for programs and parameters for processes executed in the information processing device 250. It is also used as a storage area for advertisements to be output and a storage area for warning messages.

[0244] Next, we will explain the configuration of the user terminal 280. As described above, the user terminal 280 is a terminal having a display unit that displays three-dimensional image data that reproduces the real space generated by the information processing device 250, i.e., digital twin image data, and is configured using an HMD, PC, smartphone, etc.

[0245] As shown in FIG. 34, the user terminal 280 includes a control unit 281 , an input unit 282 , an output unit 283 , a gaze area information acquisition unit 284 , a communication unit 285 , and a storage unit 286 .

[0246] The control unit 281 performs display control processing of the three-dimensional image data, i.e., digital twin image data, received from the information processing device 250. It also controls the acquisition processing of the user's gaze area information executed by the gaze area information acquisition unit 284. It also controls communication with the information processing device 250, analyzes data received from the information processing device 250, and so on.

[0247] The input unit 282 is an input unit such as a keyboard, mouse, button, switch, etc., and is an input unit that can be operated by a user using the user terminal 280, and receives information such as information for starting, stopping, and setting up processing. The output unit 283 is composed of a display unit, a speaker, etc., used to output images, text, etc., and is also used to output warnings.

[0248] The gaze area information acquisition unit 284 executes processing to acquire information required for analyzing the gaze area of ​​a user viewing three-dimensional image data, for example, digital twin image data, displayed on the user terminal 280 configured by an HMD, PC, smartphone, etc. For example, camera capture data for analyzing the user's line of sight, as well as images displayed on the display unit, etc., can be used. The gaze area analysis data acquired by the gaze area information acquisition unit 284 is transmitted to the information processing device 250 via the communication unit 285.

[0249] The communication unit 285 is used for communication with the information processing device 250. It receives three-dimensional image data, which is reproduction data of real space, i.e., digital twin image data, from the information processing device 250. Information acquired by the gaze area information acquisition unit 284, i.e., information required to analyze the gaze area of ​​the user who is viewing the digital twin image data, is transmitted to the information processing device 250.

[0250] The storage unit 286 is used as a storage area for programs and parameters for processing executed on the user terminal 280. It is also used as a storage area for data to be output to the display unit and a storage area for warning messages.

[0251] 6. Hardware Configuration Examples of Information Processing Devices, etc. Next, a hardware configuration example of the local terminal 230, the information processing device 250, the user terminal 280, etc., which have been described with reference to FIGS. 33 and 34, will be described with reference to FIG.

[0252] The hardware shown in Fig. 35 is an example of the hardware configuration of the local terminal 230, the information processing device 250, the user terminal 280, etc., which have been described with reference to Fig. 33 and Fig. 34. The hardware configuration shown in Fig. 35 will be described.

[0253] A CPU (Central Processing Unit) 501 functions as a data processing unit that executes various processes according to programs stored in a ROM (Read Only Memory) 502 or a storage unit 508. For example, the CPU executes processes according to the sequences described in the above-described embodiments. A RAM (Random Access Memory) 503 stores programs and data executed by the CPU 501. The CPU 501, ROM 502, and RAM 503 are interconnected by a bus 504.

[0254] The CPU 501 is connected to an input / output interface 505 via a bus 504, and the input / output interface 505 is connected to an input unit 506 consisting of various sensors, a camera, a switch, a keyboard, a mouse, a microphone, etc., and an output unit 507 consisting of a display, a speaker, etc.

[0255] A storage unit 508 connected to the input / output interface 505 is formed of, for example, a hard disk, and stores various data and programs executed by the CPU 501. A communication unit 509 functions as a transmitter / receiver for data communication via a network such as the Internet or a local area network, and communicates with external devices.

[0256] A drive 510 connected to the input / output interface 505 drives a removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory such as a memory card, and executes recording or reading of data.

[0257] [7. Summary of the Configuration of the Present Disclosure] The embodiments of the present disclosure have been described above in detail with reference to specific examples. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of examples and should not be interpreted as being limited. To determine the gist of the present disclosure, the claims should be taken into consideration.

[0258] The technology disclosed in this specification can be configured as follows: (1) An information processing method executed in an information processing device, in which a data processing unit generates a real-space user gaze area map that is analysis data of the gaze area of ​​a real-space user who directly observes real space, and an image-observing user gaze area map that is analysis data of the gaze area of ​​an image-observing user who observes a three-dimensional image of the real space, and detects at least one of a high-gaze area and a low-gaze area in the real space using at least one of the real-space user gaze area map and the image-observing user gaze area map.

[0259] (2) The information processing method described in (1) in (2) is characterized in that the data processing unit combines the real-space user gaze area map and the image-observing user gaze area map to generate a composite user gaze area map that combines the gaze area of ​​the real-space user and the gaze area of ​​the image-observing user, and uses the generated composite user gaze area map to detect at least one of areas in the real space that are highly gazed upon or areas that are low in gaze.

[0260] (3) The information processing method described in (2), wherein the data processing unit uses the synthetic user gaze area map to perform a calculation process of a gaze score S, which is a value reflecting the gaze degree of each area unit in the real space.

[0261] (4) The information processing method according to (3), wherein the data processing unit calculates the gaze score S as a value reflecting a gaze degree for each area unit for a specific period.

[0262] (5) The information processing method according to (3) or (4), wherein the data processing unit calculates the gaze score S for each advertisement displayable area in the real space.

[0263] (6) The information processing method according to (5), wherein the data processing unit calculates an advertising price P for each unit of area in the real space where an advertisement can be displayed, using an attention score S indicating the degree of attention for the unit of area in the real space where an advertisement can be displayed.

[0264] (7) The information processing method according to any one of (1) to (6), wherein the data processing unit further executes a process of generating and distributing a three-dimensional image of the real space.

[0265] (8) The information processing method according to any one of (1) to (7), wherein the data processing unit receives an image captured by a camera of the real space from a local terminal in the real space and generates a three-dimensional image of the real space.

[0266] (9) The information processing method described in (8), wherein the data processing unit executes a process of generating and distributing digital twin data including a three-dimensional image of the real space.

[0267] (10) An information processing method described in any one of (1) to (9), in which the data processing unit generates the image observation user gaze area map using information received from a user terminal to which the three-dimensional image of the real space is delivered.

[0268] (11) The information processing method described in any one of (1) to (10), wherein the data processing unit determines whether an area with low attention detected using the real-space user gaze area map corresponds to an area that should be paid attention to, and if it is an area that should be paid attention to, outputs a warning.

[0269] (12) The information processing method according to (11), wherein the real space is a factory, and the area to be watched is a work area of ​​a real space user who is a worker in the factory.

[0270] (13) The data processing unit determines whether an area with low attention detected using the image observation user attention area map corresponds to an area that should be paid attention to, and if it is an area that should be paid attention to, outputs a warning. This is an information processing method described in any of (1) to (12).

[0271] (14) The information processing method according to (13), wherein the real space is a factory, and the area to be watched is a work area of ​​a worker in the factory that is to be observed by an image observing user who is a supervisor of the factory.

[0272] (15) An information processing device having a data processing unit that performs analysis processing of gaze areas, wherein the data processing unit generates a real-space user gaze area map which is analysis data of the gaze area of ​​a real-space user who directly observes real space, and an image-observing user gaze area map which is analysis data of the gaze area of ​​an image-observing user who observes a three-dimensional image of the real space, and detects at least one of areas in the real space with high gaze levels or areas with low gaze levels using at least one of the gaze area maps, the real-space user gaze area map or the image-observing user gaze area map.

[0273] (16) A program for executing information processing in an information processing device, the program causing a data processing unit to execute the following processes: generating a real-space user gaze area map, which is analysis data of the gaze area of ​​a real-space user who directly observes real space; and an image-observing user gaze area map, which is analysis data of the gaze area of ​​an image-observing user who observes a three-dimensional image of the real space; and detecting at least one of areas of high gaze or low gaze in the real space using at least one of the gaze area maps, the real-space user gaze area map or the image-observing user gaze area map.

[0274] The series of processes described in this specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence can be installed and executed in the memory of a computer incorporated in dedicated hardware, or the program can be installed and executed on a general-purpose computer capable of executing various processes. For example, the program can be pre-recorded on a recording medium. In addition to installing the program on a computer from a recording medium, the program can also be received via a network such as a LAN (Local Area Network) or the Internet and installed on a recording medium such as an internal hard disk.

[0275] The various processes described in this specification may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capabilities of the devices executing the processes or as needed. Furthermore, in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices that are all located in the same housing.

[0276] As described above, according to one embodiment of the present disclosure, a configuration is realized in which a composite user gaze area map is generated by combining the gaze areas of a real-space user and an image-observing user, and the gaze level of each area in the real space is analyzed. Specifically, for example, a real-space user gaze area map is generated by analyzing the gaze areas of a real-space user who directly observes the real space, an image-observing user gaze area map is generated by analyzing the gaze areas of an image-observing user who observes a three-dimensional image of the real space, and a composite user gaze area map is generated by combining these two gaze area maps. At least one of these gaze area maps is used to detect areas in the real space with high or low gaze levels, and to perform effective advertisement display processing, warning output processing, etc. This configuration is realized in which a composite user gaze area map is generated by combining the gaze areas of a real-space user and an image-observing user, and the gaze level of each area in the real space is analyzed.

[0277] 10 Event venue 20 Advertisement display area 30 Real space user 40 Event venue terminal 41 Camera 45 Robot 50 Information processing device 60 Real space gaze area map 70 HMD 71 Smartphone 72 PC 80 Image observing user 90 Image observing user gaze area 95 Image observing user gaze area map 100 Synthesized user gaze area map 110 Advertisement displayable area 120 Factory 121 Camera 122 Robot 125 Factory terminal 135 Real space gaze area map 140 Factory monitoring center 141 Image observing user 145 Image observing user gaze area map 210 Camera 211 Control unit 212 Imaging unit 213 Communication unit 220 Robot 221 Control unit 222 Drive unit 223 Imaging unit 224 Output unit 2245 Communication unit 226 Memory unit 230 Local terminal 231 Control unit 232 Communication unit 233 Input unit 234 Output unit 235 Memory unit 250 Information processing device 251 Communication unit 252 Control unit (data processing unit) 253 Memory unit 261 Data generation unit (digital twin data etc. generation unit) 262 Gaze area analysis unit 263 Gaze score calculation unit 264 Advertising price calculation unit 265 Advertisement output control unit 266 Warning output control unit 280 User terminal 281 Control unit 282 Input unit 283 Output unit 284 Gaze area information acquisition unit 285 Communication unit 286 Memory unit 501 CPU 502 ROM 503 RAM 504 Bus 505 Input / output interface 506 Input unit 507 Output unit 508 Memory unit 509 Communication unit 510 Drive 511 Removable media

Claims

1. An information processing method executed in an information processing device, in which a data processing unit generates a real-space user gaze area map, which is analytical data of the gaze area of ​​a real-space user who directly observes real space, and an image observing user gaze area map, which is analytical data of the gaze area of ​​an image observing user who observes a three-dimensional image of the real space, and detects at least one of areas of high gaze or areas of low gaze in the real space using at least one of the gaze area maps, the real-space user gaze area map or the image observing user gaze area map.

2. The information processing method of claim 1, wherein the data processing unit combines the real-space user gaze area map with the image-observing user gaze area map to generate a composite user gaze area map that combines the gaze area of ​​the real-space user with the gaze area of ​​the image-observing user, and uses the generated composite user gaze area map to detect at least one of areas in the real space that are highly gazed upon or areas that are low gazed upon.

3. The information processing method according to claim 2, wherein the data processing unit uses the synthetic user gaze area map to perform a calculation process of a gaze score S, which is a value reflecting the gaze degree of each area unit in the real space.

4. The information processing method according to claim 3, wherein the data processing unit calculates the attention score S as a value reflecting the degree of attention per area unit for a specific period of time.

5. The information processing method according to claim 3, wherein the data processing unit calculates the gaze score S for each advertisement displayable area in the real space.

6. The information processing method according to claim 5, wherein the data processing unit calculates an advertising price P for each unit of area in the real space where an advertisement can be displayed using an attention score S indicating the degree of attention for the unit of area in the real space where an advertisement can be displayed.

7. The information processing method according to claim 1, wherein the data processing unit further executes a process of generating and distributing a three-dimensional image of the real space.

8. The information processing method according to claim 1, wherein the data processing unit inputs a camera image of the real space from a local terminal in the real space to generate a three-dimensional image of the real space.

9. The information processing method according to claim 8, wherein the data processing unit executes a process of generating and distributing digital twin data including a three-dimensional image of the real space.

10. The information processing method according to claim 1, wherein the data processing unit generates the image observation user gaze area map using information received from a user terminal to which the three-dimensional image of the real space is delivered.

11. The information processing method of claim 1, wherein the data processing unit determines whether an area with low attention level detected using the real-space user gaze area map corresponds to an area that should be attended to, and if it is an area that should be attended to, outputs a warning.

12. The information processing method according to claim 11, wherein the real space is a factory, and the area to be watched is a work area of ​​a real space user who is a worker in the factory.

13. The information processing method of claim 1, wherein the data processing unit determines whether an area with low attention level detected using the image observation user gaze area map corresponds to an area that requires attention, and if it is an area that requires attention, outputs a warning.

14. The information processing method according to claim 13, wherein the real space is a factory, and the area to be watched is a work area of ​​workers in the factory to be observed by an image observing user who is an observer of the factory.

15. An information processing device having a data processing unit that performs gaze area analysis processing, wherein the data processing unit generates a real-space user gaze area map which is analysis data of the gaze area of ​​a real-space user who directly observes real space, and an image observing user gaze area map which is analysis data of the gaze area of ​​an image observing user who observes a three-dimensional image of the real space, and detects at least one of areas of high gaze or low gaze in the real space using at least one of the gaze area maps, the real-space user gaze area map or the image observing user gaze area map.

16. A program for executing information processing in an information processing device, the program causing a data processing unit to execute a process of generating a real-space user gaze area map which is analysis data of the gaze area of ​​a real-space user directly observing real space, and an image observing user gaze area map which is analysis data of the gaze area of ​​an image observing user observing a three-dimensional image of the real space, and a process of detecting at least one of areas of high gaze or areas of low gaze in the real space using at least one of the gaze area maps, the real-space user gaze area map or the image observing user gaze area map.

Images