Information processing device, information processing method, and recording medium

Abstract

An information processing device according to the present disclosure is provided with a control unit (100) that determines, on the basis of first space information relating to a distribution source space in which a plurality of content items are associated with each other and second space information relating to a viewing space in which the plurality of content items are viewed, presentation positions of the plurality of content items in the viewing space.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a recording medium. [Background technology]

[0002] With the development of high-speed communication networks, real-time distribution of events such as live music concerts has become popular. Meanwhile, at actual events, the experience of movement before and after participating in the event is important for users to feel immersed. In particular, for collective events with multiple performers and locations (such as music festivals), the experience of actively moving between venues is extremely important for users to feel immersed.

[0003] Patent Document 1 discloses a technology for switching the viewpoint of an image on a head-mounted display. The technology described in Patent Document 1 makes it possible to switch between images of one venue and images of another venue at a music festival or the like. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-149122 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology described in Patent Document 1 does not take into consideration the movement of users between multiple venues, making it difficult to provide a sufficient sense of immersion in a remote event experience that requires a sense of immersion.

[0006] The present disclosure aims to provide an information processing device, an information processing method, and a recording medium that can give users a more immersive feeling during a remote event that is distributed. [Means for solving the problem]

[0007] The information processing device according to the present disclosure includes: first spatial information relating to a distribution source space to which a plurality of contents are associated; and a location where the plurality of contents are viewed; Includes multiple viewing areas and a control unit that determines presentation positions of the plurality of contents in the viewing space based on second spatial information about the viewing space. The plurality of viewing areas are movable between areas by a user viewing the plurality of contents, and the control unit determines the presentation position of each of the plurality of contents with respect to each of the plurality of viewing areas based on the first spatial information and the second spatial information. . [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram illustrating a configuration of an example of an information processing system applicable to an embodiment. [Figure 2] FIG. 1 is a schematic diagram illustrating an external appearance of an example of a terminal device as eyewear that can be applied to an embodiment. [Figure 3] FIG. 1 is a block diagram illustrating a hardware configuration of an example of a terminal device as eyewear that can be applied to an embodiment. [Figure 4] FIG. 2 is a functional block diagram illustrating an example of a function of a terminal device according to an embodiment. [Figure 5] 1 is a flowchart illustrating an example of an information processing method according to an embodiment. [Figure 6] 10 is a flowchart illustrating an example of an acquisition process for acquiring environmental information related to a user according to an embodiment. [Figure 7] 10A and 10B are schematic diagrams showing examples of image information and depth information acquired by an outward-facing camera. [Figure 8] FIG. 10 is a schematic diagram showing an example of a user interface screen that prompts the user to obtain image information and depth information of an appropriate area. [Figure 9] 10 is a schematic diagram showing an example of three-dimensional model data of an environment related to a user, generated by data integration by an environment information acquisition unit. FIG. [Figure 10A] 10A and 10B are schematic diagrams for explaining an example of a method for setting a plurality of viewing areas in a viewing space, which can be applied to an embodiment. [Figure 10B]10A and 10B are schematic diagrams for explaining an example of a method for setting a plurality of viewing areas in a viewing space, which can be applied to an embodiment. [Figure 10C] 10A and 10B are schematic diagrams for explaining an example of a method for setting a plurality of viewing areas in a viewing space, which can be applied to an embodiment. [Figure 10D] 10A and 10B are schematic diagrams for explaining an example of a method for setting a plurality of viewing areas in a viewing space, which can be applied to an embodiment. [Figure 11] 5A and 5B are schematic diagrams showing examples of three-dimensional model data and area information stored by a control unit according to an embodiment. [Figure 12] FIG. 10 is a schematic diagram illustrating an example of a map showing the setting of the positional relationships of multiple contents in a collective event. [Figure 13] FIG. 10 is a schematic diagram showing an example of an event timetable created on the distribution side. [Figure 14] 10A and 10B are diagrams for explaining the arrangement of areas corresponding to each stage relative to a grid, which can be applied to an embodiment. [Figure 15] FIG. 10 is a schematic diagram showing an example of arranging an area other than the venue of an event on a preset map, which is applicable to the embodiment. [Figure 16] 10 is a flowchart illustrating an example of a process for allocating content to a viewing space according to an embodiment. [Figure 17] FIG. 10 is a schematic diagram showing an example of information used to determine the placement of each piece of content, which is applicable to an embodiment. [Figure 18] 10A and 10B are schematic diagrams illustrating an example of a user interface for arranging content in a viewing area, which is applicable to an embodiment. [Figure 19] 3A and 3B are schematic diagrams illustrating an example of the arrangement of each content in each viewing area according to the embodiment. [Figure 20] 10 is a flowchart illustrating an example of a content viewing process according to the embodiment. [Figure 21] 10A and 10B are schematic diagrams illustrating an example of setting a viewing area according to the number of users in the viewing area according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are denoted by the same reference numerals, and redundant description will be omitted.

[0010] Hereinafter, embodiments of the present disclosure will be described in the following order. 1. Overview of the embodiments of the present disclosure 2. Configurations applicable to the embodiment 3. Processing according to the embodiment 3-1. Overview of processing according to the embodiment 3-2. Details of the processing according to the embodiment 3-2-1. Environmental information acquisition process 3-2-2. Allocating events to viewing environments 4. Viewing process according to the embodiment

[0011] 1. Overview of the Embodiments of the Present Disclosure First, an outline of an embodiment of the present disclosure will be described. The present disclosure relates to a technology for viewing an event held in a three-dimensional space in a viewing space remote from the location of the event.

[0012] More specifically, an event is held using multiple venues in a three-dimensional space. The three-dimensional space in which each venue is set may be a real space or a virtual space generated by executing a program on a computer. Content is associated with each venue. The type of event is not particularly limited as long as performers present content to an audience on a stage or the like. If the event is a music event, the content may be a musical performance such as singing, playing, or dancing to a song.

[0013] Here, we will assume that the event is a collective event that uses multiple venues and features multiple performers. If the event is related to music, such a collective event is called a music festival or the like. Hereinafter, a collective event may be referred to simply as an "event."

[0014] An information processing system according to an embodiment of the present disclosure sets a viewing space for viewing an event, such as a user's home, that is remote from the event venue. The information processing system according to the embodiment acquires first spatial information related to a three-dimensional space serving as a distribution source space to which multiple pieces of content are associated. The information processing system according to the embodiment also acquires second spatial information related to the viewing space in which the user views the multiple pieces of content for the event. The information processing system according to the embodiment determines, based on the acquired first and second spatial information, each presentation position in the viewing space at which each of the multiple pieces of content is presented.

[0015] In an embodiment of the present disclosure, this configuration allows users to virtually experience traveling between multiple venues within a viewing space, making it possible to provide users with a more immersive experience during a remote event via streaming.

[0016] [2. Configuration applicable to the embodiment] Next, a configuration applicable to the embodiment will be described. Fig. 1 is a schematic diagram showing the configuration of an example of an information processing system applicable to the embodiment. In Fig. 1, the information processing system 1 according to the embodiment includes a terminal device 10 and a server 20 that are communicably connected to each other via a network 2 such as the Internet. Note that while Fig. 1 shows the server 20 as a single device, this is merely an example, and the server 20 may be configured as a computer group consisting of multiple computers with distributed functions, or may be multiple computers connected by cloud computing technology.

[0017] The terminal device 10 constitutes an information processing device according to the embodiment and is used by a user 30 who watches content from an event. A wearable device such as a glasses-type device (eyewear device) or a smartphone can be used as the terminal device 10. The terminal device 10 is not limited to this, and may be a general information processing device such as a personal computer or a game device, to which an FPD (Flat Panel Display) is connected. The terminal device 10 can output sound using a sound output device such as a speaker, headphones, or earphones. The following description will be given assuming that the terminal device 10 is an eyewear device, unless otherwise specified.

[0018] An eyewear device applicable as the terminal device 10 may be a so-called see-through type head-mounted display (AR (Augmented Reality) glasses) that allows images of real space to pass through, or may be a goggle-type device (VR (Virtual Reality) goggles) that does not allow images of real space to pass through. When VR goggles are used as the terminal device 10, it is preferable to provide a means that allows the user 30 wearing the terminal device 10 to easily check the surrounding situation, which is real space, such as by displaying images of the surroundings captured by an outward-facing camera provided in the VR goggles.

[0019] In the example of FIG. 1, a streaming server 70, a content DB (database) 51, and a reference DB 72 are further connected to the network 2. The streaming server 70 processes video data of a real-time event. For example, the streaming server 70 distributes video data being shot at each venue of the event via the network 2. The content DB 71 stores, for example, the video data distributed by the streaming server 70. The reference DB 72 manages information related to the video data distributed by the streaming server 70 and the video data stored in the content DB 71. The reference DB 72 manages, for example, the event date, event information, etc. related to this video data.

[0020] In the above description, the terminal device 10 constitutes the information processing device according to the embodiment, but this is not limited to this example. For example, the information processing device according to the embodiment may be constituted by the entire information processing system 1 including the terminal device 10 and the server 20.

[0021] 2 is a schematic diagram illustrating the appearance of an example of a terminal device 10 as an eyewear device applicable to an embodiment. The main body of the terminal device 10 is generally a glasses-type or goggle-type device that is worn on the head of a user 30 and can realize functions such as superimposing digital information on the field of view of both or one of the user's 30 eyes, emphasizing or attenuating the image of a specific real object, and deleting the image of a specific real object to make it appear as if the real object does not exist. FIG. 2 illustrates the terminal device 10 worn on the head of a user 30.

[0022] In FIG. 2, terminal device 10 has a display unit 1201L for the left eye and a display unit 1201R for the right eye disposed in front of the left and right eyes of user 30, respectively. Display units 1201L and 1201R are transparent or semi-transparent, enabling superimposition of virtual objects on a real-space landscape, enhancement or attenuation of the image of a specific real object, and deletion of the image of a specific real object to make it appear as if the real object image does not exist. For example, left and right display units 1201L and 1201R may be independently driven to display parallax images, i.e., virtual objects, as three-dimensional information. Furthermore, an outward-facing camera 1101 facing the line of sight of user 30 is disposed approximately at the center of terminal device 10.

[0023] Fig. 3 is a block diagram showing an example of the hardware configuration of a terminal device 10 as an eyewear device applicable to an embodiment. In Fig. 3, the terminal device 10 includes a CPU 1500, a ROM (Read Only Memory) 1501, a RAM (Random Access Memory) 1502, a camera I / F (Interface) 1503, a sensor I / F 1504, a storage device 1505, a display control unit 1506, an audio I / F 1507, a communication I / F 1508, and a positioning unit 1509, which are communicably connected to each other via a bus 1520. In this way, the terminal device 10 has a configuration as a computer (information processing device) including a CPU, memory, and various I / Fs.

[0024] The storage device 1505 is a non-volatile storage medium such as a flash memory. The CPU 1500 operates in accordance with an information processing program stored in advance in the storage device 1505 or the ROM 1501, using the RAM 1502 as a work memory, and controls the overall operation of the terminal device 10.

[0025] Camera I / F 1503 is an interface for outward-facing camera 1101 and inward-facing camera 1102, and supplies image signals output from outward-facing camera 1101 and inward-facing camera 1102 to bus 1520. In addition, control signals for controlling outward-facing camera 1101 and inward-facing camera 1102, which are generated by CPU 1500 in accordance with an information processing program, are transmitted to outward-facing camera 1101 and inward-facing camera 1102 via camera I / F 1503.

[0026] The sensor I / F 1504 is an interface to the gyro sensor 1104, acceleration sensor 1105, orientation sensor 1106, and biosensor 1107, and each sensor signal output from the gyro sensor 1104, acceleration sensor 1105, orientation sensor 1106, and biosensor 1107 is supplied to the bus 1520 via the sensor I / F 1504.

[0027] The display control unit 1506 controls the display operations of the display units 1201L and 1201R in accordance with instructions from the CPU 1500. For example, the display control unit 1506 converts a display control signal generated by the CPU 1500 in accordance with an information processing program into a display signal that can be displayed by the display units 1201L and 1201R, and supplies the signal to the display units 1201L and 1201R.

[0028] The audio I / F 1507 is an interface with the microphone 1103 and the audio output unit 1202. For example, the audio I / F 1507 converts an analog sound signal based on the sound picked up by the microphone 1103 into a digital sound signal and supplies the digital sound signal to the bus 1520. The audio I / F 1507 also converts, based on a digital sound signal generated by the CPU 1500 in accordance with an information processing program and supplied via the bus 1520, into a signal in a format that can be played back by the audio output unit 1202 and supplies the signal to the audio output unit 1202.

[0029] The communication I / F 1508 controls communication between the terminal device 10 and the network 2 in accordance with instructions from the CPU 1500 .

[0030] The position measurement unit 1509 measures the current position of the terminal device 10. The position measurement unit 1509 measures the position using, for example, the Global Navigation Satellite System (GNSS) or connection information of Wi-Fi (registered trademark).

[0031] 4 is a functional block diagram illustrating an example of functions of the terminal device 10 according to the embodiment. The terminal device 10 includes a control unit 100, a sensor unit 101, a position measurement unit 102, and an output unit 103.

[0032] Sensor unit 101 includes various sensors for collecting information about the external environment of user 30 wearing terminal device 10 and the status of user 30. Sensor unit 101 includes, for example, outward-facing camera 1101 and inward-facing camera 1102, microphone 1103, gyro sensor 1104, acceleration sensor 1105, orientation sensor 1106, and biometric sensor 1107, which are described with reference to FIG.

[0033] The outward-facing camera 1101 and the inward-facing camera 1102 are typically electronic cameras that capture light with wavelengths in the visible light range. The wavelengths of light captured by the outward-facing camera 1101 and the inward-facing camera 1102 may include the ultraviolet to infrared ranges, or may be limited to a certain wavelength range within these wavelength ranges. Additionally, multiple outward-facing cameras 1101 may be used to measure depth information of the environment. Additionally, the outward-facing camera 1101 may be used in combination with a distance sensor such as a ToF (Time of Flight) sensor or an ultrasonic sensor.

[0034] Inward-facing camera 1102 may acquire gaze information of user 30, or may acquire biological information such as the number of blinks. Also, environmental illumination conditions may be estimated from the size of the pupil.

[0035] The microphone 1103 is intended to grasp the external acoustic conditions and the speech state of the user 30. The microphone 1103 may also be used for communication through sound waves with wavelengths in the inaudible range. The principle of the microphone 1103 is not limited, and it may be a dynamic microphone, a condenser microphone, or the like.

[0036] The gyro sensor 1104, acceleration sensor 1105, and orientation sensor 1106 are used to grasp the movement status and posture of the user 30. The gyro sensor 1104, acceleration sensor 1105, and orientation sensor 1106 may also be used to improve the accuracy of positioning by the position positioning unit 102, which will be described later.

[0037] The biosensor 1107 measures the heart rate, pulse rate, etc. of the user 30. By measuring these, the activity level, tension level, and concentration level of the user 30 can be grasped.

[0038] 2, and acquires location information indicating the current location of the user 30. The location measurement unit 102 acquires the location information using radio waves from GNSS, communication via Wi-Fi, or the like.

[0039] 4, the control unit 100 includes a user information acquisition unit 110, an environmental information acquisition unit 111, a map generation unit 112, an output control unit 113, a content information acquisition unit 114, a communication unit 115, and an operation input unit 116. The user information acquisition unit 110, the environmental information acquisition unit 111, the map generation unit 112, the output control unit 113, the content information acquisition unit 114, the communication unit 115, and the operation input unit 116 are configured by executing an information processing program according to the embodiment on the CPU 1500. However, some or all of the user information acquisition unit 110, the environmental information acquisition unit 111, the map generation unit 112, the output control unit 113, the content information acquisition unit 114, the communication unit 115, and the operation input unit 116 may be configured by hardware circuits that operate in cooperation with each other.

[0040] The content information acquisition unit 114 acquires, for example, the purchase status and type of content of content stored in the storage device 1505. The content information acquisition unit 114 acquires data from an external server (for example, the content DB 71) as necessary. For example, if the target event includes multiple types of content, the content information acquisition unit 114 can acquire information necessary for the user 30 to make a decision when viewing the content from multiple external servers.

[0041] The user information acquisition unit 110 acquires information from various sensors included in the sensor unit 101 and location information measured by the location measurement unit 102. The user information acquisition unit 110 aggregates and manages the acquired information.

[0042] Environmental information acquisition unit 111 acquires environmental information and map information that indicate the surrounding environment of user 30 based on information such as an image captured by outward-facing camera 1101. Furthermore, environmental information acquisition unit 111 may refer to known information from a database, such as separately acquired floor plan information, as environmental information.

[0043] The map generation unit 112 generates a layout map for arranging content in the viewing space based on various information acquired by the environment information acquisition unit 111 and data acquired from the content information acquisition unit 114. When generating the layout map, the map generation unit 112 determines compatibility with the environment for each piece of content. The map generation unit 112 may determine compatibility by, for example, calling up reference environment information stored in the storage device 1505 as an ideal viewing environment for the content, and taking into consideration the preferences of the user 30.

[0044] The output control unit 113 presents information to the user 30 via the output unit 103. The output unit 103 includes, for example, display units 1201L and 1201R and an audio output unit 1202 in Fig. 3. The information presented by the output control unit 113 may include video and audio related to the content, information related to the content placement, and the like.

[0045] Operation input unit 116 detects an input action of user 30. Operation input unit 116 can detect an input action of user 30 using each sensor included in sensor unit 101 described above. For example, operation input unit 116 can detect an input action by recognizing the hand of user 30 from an image captured by outward-facing camera 1101 and using hand tracking technology to track the movement of the recognized hand. Alternatively, a controller for user 30 to perform input operations may be connected to terminal device 10, and operation input unit 116 may detect user operations on this controller.

[0046] The communication unit 115 communicates with the outside of the terminal device 10 via wired or wireless communication. The communication unit 115 communicates with, for example, a terminal device used by another user, a device worn by the user 30 himself, an external information processing device, etc. via Wi-Fi, Bluetooth (registered trademark), infrared communication, etc., and acquires information necessary for various determinations, etc.

[0047] In the terminal device 10, the CPU 1500 executes the information processing program according to the embodiment, and thereby configures the above-mentioned user information acquisition unit 110, environmental information acquisition unit 111, map generation unit 112, output control unit 113, content information acquisition unit 114, communication unit 115 and operation input unit 116, each as, for example, a module, on the main memory area of ​​the RAM 1502.

[0048] The information processing program can be acquired from an external source (for example, the server 20) via a network 2 such as a LAN or the Internet by communication via the communication I / F 1508, and can be installed on the terminal device 10. In this case, the information processing program is stored and saved in a storage medium possessed by the server 20. However, the information processing program may also be provided by being stored in a removable storage medium such as a CD (Compact Disk), a DVD (Digital Versatile Disk), or a USB (Universal Serial Bus) memory.

[0049] [3. Processing according to the embodiment] Next, the processing according to the embodiment will be described.

[0050] (3-1. Overview of Processing According to the Embodiment) The process according to the embodiment will be briefly described below. Information required for the process according to the embodiment can be roughly divided into the following two types. (A) Information about the content viewed by user 30 (B) Information about the viewing environment of the user 30

[0051] The information about the content viewed by the user 30 in (A) includes, for example, if the event is a live music concert, the performers, the performance format, etc. The information about the content may include location information indicating the location where the content is presented in the distribution source space, and a timetable indicating the time period during which the content is presented. Furthermore, the environment in which the user 30 views in (B) includes, for example, the device used for viewing, surrounding space information (such as furniture type and arrangement) regarding the space surrounding the location where the user 30 views, etc. In the embodiment, by associating the information about the content with the information about the viewing environment, when the content is arranged in a real space (such as the home of the user 30), an arrangement that has a high affinity between the content and the real space is realized.

[0052] In order to realize a content arrangement that has a high affinity between the content and the real space, it is desirable that the content to be viewed and the viewing environment match. Furthermore, when the user 30 explicitly moves in the real space and the viewing environment changes, it is desirable that the content transition occurs in response to the movement. For example, interruptions in viewing during a content transition will result in a degradation of the viewing experience.

[0053] Therefore, in the embodiment, for example, fade-in and fade-out processing (cross-fade processing) may be inserted while taking into consideration the loading of the video (content) after the transition when the content transitions as the user 30 moves. Also, when it is difficult to handle video buffering due to factors such as communication speed, a video with reduced image quality or only audio may be output in advance.

[0054] The following provides a more detailed description of the automatic placement of content in the viewing space and the transition that accompanies movement of the user 30 according to the embodiment. The processing described below is mainly executed by the environment information acquisition unit 111, the map generation unit 112, and the content information acquisition unit 114 in the functional block diagram of FIG.

[0055] FIG. 5 is a flowchart illustrating an example of an information processing method according to the embodiment.

[0056] Prior to the processing of the flowchart in FIG. 5, the user 30 wears the terminal device 10 (eyewear device) and starts the information processing program according to the embodiment. It is also assumed that the user 30 has previously acquired information about the event that the user wants to watch, and will watch the event at home. That is, in this case, the home of the user 30 constitutes the viewing space for watching the event. It is also assumed that the event is a collective event with multiple venues and multiple performers, as described above.

[0057] In step S10, the control unit 100 of the terminal device 10 acquires environmental information for the user 30 using the environmental information acquisition unit 111. The environmental information acquisition unit 111 captures an image of the surroundings of the user 30 using, for example, an outward-facing camera 1101 provided in the terminal device 10, and acquires captured image data. The control unit 100 models a three-dimensional shape of the surroundings based on, for example, the captured image data, and generates a map of the viewing space.

[0058] In the next step S11, the control unit 100 acquires information about the event that the user 30 is watching, using the content information acquisition unit 114. For example, the content information acquisition unit 114 communicates with the reference DB 72, for example, via the network 2, and acquires information about the event specified by the user 30 from the reference DB 72. The information about the event acquired by the control unit 100 may include, for example, an event map based on map information indicating the locations of multiple venues prepared for the event, and a timetable indicating the progress of each venue in the event.

[0059] In the next step S12, the control unit 100 acquires the user 30's planned activities for the event, for example, by the content information acquisition unit 114. For example, the content information acquisition unit 114 causes the output control unit 113 to display a timetable of the event on the display unit and prompts the user 30 to input their plans. The content information acquisition unit 114 can acquire the user 30's planned activities in response to an input operation by the user 30.

[0060] In the next step S13, the control unit 100 causes the map generation unit 112 to allocate the event to the environment of the user 30. That is, the map generation unit 112 allocates, for each piece of content, a presentation position for each piece of content to be presented at each venue of the event in the viewing space of the user 30. At this time, the map generation unit 112 can set multiple viewing areas in the viewing space of the user 30 and allocate each presentation position for each piece of content to each of the multiple viewing areas. The user 30 is allowed to move between the multiple viewing areas set in the viewing space of the user 30. The map generation unit 112 presents information indicating each presentation position of each piece of content allocated to the viewing space to the user 30 via the output unit 103 (e.g., display units 1202L and 1202R).

[0061] In the next step S14, if the user 30 modifies the presentation positions of the content assigned to the viewing space in step S13, the control unit 100 causes the map generation unit 112 to reflect the modifications in the assignment of each display position of the content to the viewing space.

[0062] By applying the information processing system 1 according to the embodiment, each presentation position of content presented at multiple venues for a collective event is assigned to a viewing space set up, for example, in the home of a user 30. Therefore, when viewing a collective event remotely, for example, the user 30 can experience movement between multiple venues in real space in the viewing space set up, for example, in the home, and can obtain a deeper sense of immersion.

[0063] (3-2. Details of Processing According to the Embodiment) Next, the processing according to the embodiment will be described in more detail.

[0064] (3-2-1. Environmental information acquisition processing) First, a more detailed description will be given of the acquisition process for acquiring environmental information about the user 30 by the information processing system 1 according to the embodiment, which was explained in step S10 of Fig. 5. Fig. 6 is a flowchart illustrating an example of the acquisition process for acquiring environmental information about the user 30 according to the embodiment.

[0065] In step S100, control unit 100 acquires environmental information related to user 30 using environmental information acquisition unit 111. More specifically, environmental information acquisition unit 111 acquires environmental image information related to user 30, depth information, and position information of terminal device 10 as environmental information based on the outputs of sensor unit 101 and position positioning unit 102. For example, environmental information acquisition unit 111 acquires image information as RGB information based on color information of R (red), G (green), and B (blue) using outward-facing camera 1101 included in sensor unit 101. Furthermore, environmental information acquisition unit 111 acquires depth information using outward-facing camera 1101.

[0066] Fig. 7 is a schematic diagram showing an example of image information and depth information acquired by outward-facing camera 1101. In Fig. 7, section (a) shows an example of image information, and section (b) shows an example of depth information. In section (b), depth information is acquired at the position of each dot.

[0067] When acquiring image information and depth information, the environment information acquisition unit 111 may use a user interface or the like to prompt the user 30 to acquire information on an appropriate area so as to cover the area (viewing space) that the user 30 is expected to use in the current viewing experience. The user interface includes at least one of a screen that presents information to the user and an input unit through which the user inputs information. FIG. 8 is a schematic diagram showing an example of a user interface screen that prompts the user 30 to acquire image information and depth information on an appropriate area. In FIG. 8, a user interface screen 300 is presented to the user 30 by the output unit 103 (e.g., display units 1202L and 1202R), including a message 301 that prompts the user 30 to acquire image information and depth information on an appropriate area.

[0068] 6, in the next step S101, the control unit 100 integrates the environmental information acquired in step S100 using the environmental information acquisition unit 111, and performs three-dimensional modeling of the environment related to the user 30. The three-dimensional modeling method uses, for example, matching of feature amounts extracted based on image information and depth information.

[0069] If a predetermined accuracy cannot be obtained by 3D modeling, the environmental information acquisition unit 111 may generate a floor plan model using 2D information. Also, if the environmental information acquisition unit 111 can acquire existing data (2D floor plan, 3D scan data), the environmental information acquisition unit 111 may refer to the existing data in subsequent processes without performing the modeling process.

[0070] Fig. 9 is a schematic diagram showing an example of 3D model data of the environment related to user 30, generated by data integration by environment information acquisition unit 111 in step S101. In Fig. 9, a 3D model 40 based on the 3D model data corresponds to the viewing space in which user 30 views an event. In the example of Fig. 9, 3D model 40 is a 3D model of the room in which user 30 lives.

[0071] When the 3D model 40 is generated, information such as the area in the 3D model 40 in which the user 30 will view content is insufficient. Therefore, in the next step S102, the control unit 100 causes the map generation unit 112 to assign meaning to the 3D model 40 and generate a map. At this time, the map generation unit 112 may retain data on the types of furniture and other items that have been placed in advance, using object recognition or the like. Furthermore, the map generation unit 112 may refer to the user 30's past purchase history or the like, and use this as reference data for identifying the type.

[0072] 9, a living room 400 is recognized in the three-dimensional model 40, and a bed 401 and furniture 403 are recognized relative to the living room 400, along with an area 402 with a different floor condition from that of the living room 400. The three-dimensional model 40 also recognizes a kitchen unit 404 and a bathroom 405. To the left of the kitchen unit 404, an area 407 with a different floor condition from that of the living room 400 is recognized, and furniture 406 is recognized adjacent to area 407.

[0073] In subsequent processing, when associating content with a viewing space, it is important to know in which area of ​​the viewing space, which is a real space, the user 30 is likely to view the content. Therefore, for example, an area in which the user 30 is likely to view the content may be explicitly set, and that point may be used as a reference point for placement.

[0074] For example, map generation unit 112 may use a user interface screen to prompt user 30 to set a viewing area where the user is likely to view content from among the parts recognized in three-dimensional model 40. Map generation unit 112 stores the reference coordinates of each set viewing area as the reference point of each viewing area.

[0075] At this time, map generation unit 112 sets a plurality of viewing areas for the viewing space based on at least one of division information, role information, structural information, floor surface information, shape information, and facility information, and incorporates the set plurality of viewing areas into 3D model 40. Figures 10A to 10D are schematic diagrams for explaining an example of a method for setting a plurality of viewing areas for a viewing space, which is applicable to the embodiment.

[0076] 10A is a diagram showing an example of setting multiple viewing areas for a viewing space using division information based on structural information that indicates the structure of the viewing space. In FIG. 10A, viewing space 310a is physically divided by structure 320, such as a door or threshold. In this case, map generator 112 can set separate viewing areas, area #1 and area #2, in the separated space, as shown in space 310b.

[0077] 10B is a diagram showing an example of setting multiple viewing areas for a viewing space using division information based on floor surface information of the viewing space. In FIG. 10B, viewing space 311a has different floor surface conditions between area 321 and other parts, and the floor surface information differs between these areas. For example, if the parts other than area 321 have hardwood flooring and area 321 is carpeted or the floor of area 321 is linoleum, the floor surface information differs between area 321 and other parts. In this case, map generation unit 112 can set separate viewing areas, area #1 and area #2, for area 321 and other parts, respectively, as shown in space 311b.

[0078] 10C is a diagram showing an example of setting multiple viewing areas for a viewing space using division information based on shape information that indicates the shape of the viewing space. As shown in space 312a in FIG. 10C, even in an environment where the floor surface condition is the same and there is no separation by structures, there is a possibility that space 312a may be divided by walls 322a and 322b. In this case, map generator 112 can set separate viewing areas, area #1 and area #2, by extending walls 322a and 322b, as shown in space 312b.

[0079] FIG. 10D illustrates an example of setting multiple viewing areas for a viewing space based on segmentation information based on equipment information for equipment installed in the viewing space. In FIG. 10D, a space 313a may contain equipment 323, such as a kitchen unit, that clearly indicates different activities occurring around it. Equipment 323 includes fixtures and electrical appliances that are fixedly or movably installed in the space. In this case, map generation unit 112 may set separate viewing areas, area #1 and area #2, around equipment 323, as separate spaces, as shown in space 313b. For example, map generation unit 112 may determine that an area containing a stove or kitchen unit as equipment 323 is a kitchen space, and that an area containing a sofa is a living room. Furthermore, margin areas may be set in advance according to the furniture, and the area including the margin areas may be used as the viewing area.

[0080] The reason for setting multiple viewing areas within a viewing space as described with reference to Figures 10A to 10D is that it is anticipated that when arranging content in a viewing space, it may be necessary to avoid arranging content across areas that separate these viewing spaces. For example, even in the same viewing space, if users 30 use the space for different purposes, this may be taken into consideration. Furthermore, even in a one-room apartment, it is anticipated that the area around the kitchen will be separated as a separate space. This information is stored as area information.

[0081] 6, in step S103, the control unit 100 stores the area information, including the 3D model data and reference point data acquired in steps S101 and S102, as environmental data in, for example, the storage device 1505, thereby completing the environment understanding. In addition to the 3D model data, the area information may include, for example, the reference point coordinates of each area, area, information about furniture to be placed, whether or not seating is permitted, the presence or absence of nearby dangerous objects (such as the presence or absence of heated objects), and soundproofing based on floor material, etc. The control unit 100 stores each item of this area information as, for example, a list.

[0082] 11 is a schematic diagram showing an example of three-dimensional model data and area information stored by the control unit 100 according to an embodiment. Section (a) of FIG. 11 shows an example in which reference positions A to D are set for each area of ​​the three-dimensional model 40. The reference positions A to D may be, for example, the coordinates of any vertex of each area or the coordinate positions obtained by averaging the vertex coordinates.

[0083] Section (b) of FIG. 11 shows an example of area information (second spatial information). In the area information, the items "reference point coordinates," "tag," and "area information" are associated with each of the reference positions A to D. The item "reference point coordinates" indicates the coordinates of the reference positions A to D. The item "tag" is associated with the viewing area information. In the example of FIG. 11, in the area information, the item "tag" indicates the meaning of each area, and the area based on reference position A is assigned the meaning of "living room," the area based on reference position B is assigned the meaning of "bed," the area based on reference position C is assigned the meaning of "kitchen," and the area based on reference position D is assigned the meaning of "bathroom." These meanings enable the areas based on reference positions A and B to be determined as areas for relaxation, the area based on reference position C is assigned the meaning of cooking, and the area based on reference position D is assigned the meaning of showering, bathing, etc. In other words, the information shown in the item "tag" can be said to be role information indicating the role of the corresponding area.

[0084] In the area information, the item "area information" indicates information to be set as a viewing area among the areas indicated by the reference positions A to D. In the example of FIG. 11, the two areas indicated by the reference positions A and B are spatially continuous and have a common role, so they are integrated and set as a viewing area indicated by area information [1]. Furthermore, the area indicated by the reference position C is set as a viewing area indicated by area information [2], and the area indicated by the reference position D is set as a viewing area indicated by area information [3]. Note that the user 30 is allowed to move between the viewing areas indicated by the area information [1] to [3].

[0085] (3-2-2. Allocating Events to Viewing Environments) Next, the process of acquiring event information and allocating it to an audio-visual environment by the information processing system 1 according to the embodiment, which has been described in steps S11 to S13 in FIG. 5, will be described.

[0086] The content to be viewed by user 30 at an event includes content for which the time (time zone) and location of presentation are predetermined, and content for which these are not predetermined. The following description focuses on content for which the time zone and location of presentation are predetermined at an event.

[0087] Furthermore, the content that the embodiments particularly target is not content presented by a single performer, but content presented at a so-called collective event where multiple performers appear in parallel at multiple venues. The embodiments are not limited to this, and also contemplate an exhibition-style event where multiple contents are distributed in parallel, even if there is only one performer. Therefore, the following description will be given assuming that multiple contents are distributed.

[0088] (About weighting multiple contents) Weighting can be applied to multiple contents. For example, weighting on the distribution side can be applied to each venue in a collective event. That is, the distributor can weight the importance of the main venue in a collective event more highly. The distributor can also apply weighting in a similar manner to an exhibition format.

[0089] On the other hand, weighting on the user 30 side may be based on the preferences of the user 30. For example, the user 30 may assign weights according to the user's own viewing priority, such as content presented by a performer of which the user 30 is already a fan. Furthermore, the user 30 may assign weights to the genres of content presented by the performers. For example, the control unit 100 presents a user interface showing each item to which the user 30 assigns weights. The control unit 100 saves the weighting information input to this user interface.

[0090] (Regarding the spatial layout of each venue in the actual space) In the case of collective events or exhibitions, the distribution side (organizer) often sets the positional relationships between the expected contents. The distribution side prepares an event map showing the positional relationships between the contents, and the positional relationships between each piece of content are calculated based on this. For the sake of explanation, we will assume that the event organizer and the distributor who distributes the content presented at the event are the same person.

[0091] FIG. 12 is a schematic diagram showing an example of a map (event map) showing the setting of the positional relationships of a plurality of contents in a collective event.

[0092] 12, the event map 50 (first spatial information) shows stage 500, which is the main stage where content is presented, and stages 501 and 502, which are sub-stages where content is presented. Note that in FIG. 12, stages 501 and 502 are also shown as sub-stage #1 and sub-stage #2, respectively. Stage 500 is set up inside stadium 510, and stages 501 and 502 are located on beach 511 and hill 512, respectively. In FIG. 12, if north (N) is directed upward, stages 501 and 502 are located in the northwest and southwest directions relative to stage 500, respectively.

[0093] In many cases, the user 30 has prior knowledge of such an event map 50. Therefore, when arranging each piece of content in the audio-visual space, it is necessary to also refer to the arrangement of each stage 500-502 on the event map 50.

[0094] Furthermore, in addition to the layout in the real space, a spatial layout for distribution may be set. For example, the layout area may be changed depending on the genre or label of the content, or the amount of interactive participation.

[0095] (Regarding preset maps provided by the distributor) Next, a preset map presented by the distribution side according to the embodiment will be described. The distribution side, for example, uses the server 20 to generate an initial layout for arranging the display positions of each content in the viewing space of the user 30, based on the relative positions of each venue in the event and the time slots (timetables) in which content is presented by each performer at each venue. The distribution side, for example, uses the server 20 to present the generated initial layout to the user 30 as a preset map.

[0096] Fig. 13 is a schematic diagram showing an example of an event timetable created on the distribution side and acquired by, for example, server 20. In Fig. 13, timetable 520 shows the performance times of performers A to F on stages 500, 501, and 502 (referred to as main stage, sub-stage #1, and sub-stage #2, respectively) on event map 50 shown in Fig. 12. In the example of Fig. 13, timetable 520 shows that performer A will perform from 10:00 to 11:00 on stage 500, and performer B will perform from 11:30 to 12:30. Similarly, the performance times of performers C to F are shown on stages 501 and 502.

[0097] The distributor defines in advance the positional relationships between the stages 500-502 for spatial arrangement, as shown in Fig. 12. For example, the distributor refers to the positional relationships between these stages when initially arranging the content to be presented on the stages 500-502 in the viewing space of the user 30. The distributor prepares a two-dimensional grid using the server 20, for example, and creates a preset map by arranging areas corresponding to the stages 500-502 on this grid.

[0098] Fig. 14 is a diagram for explaining the arrangement of areas corresponding to the stages 500 to 502 relative to a grid, which is applicable to the embodiment. In Fig. 14, section (a) shows the event map 50, and section (b) shows an example of a preset map 530. In the diagram, the preset map 530 is configured from a grid on a two-dimensional plane. In the diagram, the upward direction of the preset map 530 corresponds to the north direction.

[0099] 14 indicates an area to be placed on the preset map 530. In this example, areas 500lay, 501lay, and 502lay are automatically placed on the preset map 530 according to the grid, in accordance with the relative positions and sizes of the stages 500, 501, and 502 on the event map 50.

[0100] That is, for example, the server 20 places an area 500lay corresponding to a stage 500 placed at the eastern end (right end) of an area 540 of the event map 50 at the right end, i.e., the eastern end, of the preset map 530. For example, the server 20 places an area 501lay corresponding to a stage 501 placed northwest of the stage 500 in the event map 50 at the upper left of the preset map 530, i.e., in a position northwest of the area 500lay. Similarly, for example, the server 20 places an area 502lay corresponding to a stage 502 placed southwest of the stage 500 in the event map 50 at the lower left of the preset map 530, i.e., in a position southwest of the area 500lay.

[0101] In the illustrated example, each of the areas 500lay to 502lay is arranged on the preset map 530 in a size that corresponds to the size ratio of the corresponding stage 500 to 502.

[0102] The distributor can manually set margins between stages for each of the areas 500lay to 502lay that are automatically arranged on the preset map 530 as in section (b) of FIG.

[0103] 13, for example, the stage 501 (sub-stage #1) is not in use during the time slot from [10:00] to [10:30]. In this way, when there is a time slot in which each of the stages 500 to 502 is available, the server 20 may dynamically change the arrangement of each of the areas 500lay to 502lay on the preset map 530, for example, by expanding the area 500lay on the preset map 530 that corresponds to the stage 500, which is the main stage.

[0104] Furthermore, for example, the server 20 may perform automatic placement on the preset map 530 according to the attributes of the content presented at each venue, without referring to the event map 50. For example, the server 20 may use information such as the genre or the number of fans of the performer as content attributes, and automatically place adjacent areas on the preset map 530 for content with overlapping genres or fans. Furthermore, for example, the server 20 may perform automatic placement of content of different genres, taking into consideration factors such as ensuring a margin between areas.

[0105] Furthermore, for example, the server 20 may set a transition area that provides an effect to content when content transitions to other content at the boundary between adjacent or nearby areas (for example, area 500lay and area 501lay) in the preset map 530. Also, for example, the server 20 may arrange an area that presents other content that is not presented on the actual stages 500 to 502 in an area on the preset map 530 where areas 500lay to 502lay in which content is presented are not arranged.

[0106] FIG. 15 is a schematic diagram illustrating another example of a preset map 530 applicable to an embodiment. In FIG. 15, areas 540lay, 541lay, and 542lay arranged on the preset map 530 correspond to the venues where content #1, #2, and #3 are respectively presented during the event. Area 543 is an area including the boundary between adjacent areas 540lay and 541lay, and is a transition area for presenting a predetermined effect when content transitions to another content. The predetermined effect in the transition area includes at least one of a visual effect and an auditory effect. A specific example of such an effect is a cross-fade process between the content before the transition and the content after the transition. Alternatively, a moving image showing movement through the areas in real space corresponding to areas 540lay and 541lay may be created in advance and presented in area 543 in the viewing space viewed by user 30.

[0107] 15, area 544 is an area where content different from contents #1, #2, and #3 presented at each venue of the event is placed, and is an area where merchandise such as items are sold. In the viewing space where user 30 watches, area 544 can be configured to allow items to be purchased by electronic payment.

[0108] The server 20 stores the generated preset map 530 in, for example, the reference DB 72. However, the server 20 may store the generated preset map 530 in a storage medium provided in the server 20 itself.

[0109] In the above description, the preset map 530 is generated by the server 20, but this is not limited to this example. For example, the preset map 530 can also be generated in the terminal device 10. In this case, the terminal device 10 acquires the event map 50 and the timetable 520 from the server 20, and generates the preset map 530 based on the acquired event map 50 and the timetable 520. The terminal device 10 stores the generated preset map 530 in, for example, the storage device 1505.

[0110] (Allocating content to viewing space) Next, the association between the preset map 530 and the viewing space in which the user 30 views the content will be described.

[0111] 16 is a flowchart illustrating an example of a process for allocating content to a viewing space according to an embodiment. Each process in the flowchart of FIG.

[0112] In step S200, the control unit 200 reads the environmental data acquired and saved by the processing according to the flowchart of FIG.

[0113] In the next step S201, the control unit 100 acquires video data from the content DB 71 using the content information acquisition unit 114, and also acquires information related to the video data from the reference DB 72 through communication using the communication unit 115. The information related to the video data acquired from the reference DB 72 includes a preset map 530 and a timetable 520. Furthermore, when an event has started, for example, the control unit 100 acquires video data of the event to be distributed from the streaming server 70 through communication using the communication unit 115.

[0114] In the next step S202, the control unit 100 separates the video data, an index for the video data, and additional information added to the video data from the video data acquired in step S201 from the streaming server 70 and the content DB 71. The index and additional information may include, for example, information indicating the venue (stage) where the corresponding video data was shot.

[0115] In the next step S203, the control unit 100 causes the map generation unit 112 to perform a matching evaluation between each venue (stage) where content will be presented at the event and the environment related to the user 30, i.e., the viewing space of the user 30, based on the environmental data. In the next step S204, the control unit 100 causes the map generation unit 112 to allocate each piece of content presented at the event to the environment related to the user 30 (viewing space) based on the evaluation result of the matching evaluation in step S203.

[0116] (Allocating content to each viewing area) The processing of steps S203 and S204 described above will be explained in more detail. The positional relationship between the contents and the positional relationship between each viewing area in the viewing space can be used to evaluate the matching between each venue where the content is presented and the viewing space of user 30. Furthermore, the matching evaluation takes into consideration the degree of participation of the user in the content, the size of the venue where the content is presented, the form of video distribution, and the like.

[0117] The degree of user participation will now be explained. The degree of user participation varies depending on the content. Even in remote events that are distributed, it is expected that user calls and other actions will be reflected in the content. For example, content in which users have a high degree of participation, such as when users routinely join in, call out, or sing along when content is presented in real space, is defined as content with a high degree of participation. With content with a high degree of participation, it is expected that there will be a desire for users to perform these actions in remote events, and that these actions will be incorporated into the content as part of the performance.

[0118] It is preferable to select a space where participants can enjoy content with a high participation level as the viewing area. For example, if the content requires users to speak loudly, the user may hesitate to do so in a room with a normal soundproof environment, considering the impact on the surrounding environment. Therefore, consideration is given to allocating a viewing area with good soundproofing as the presentation location where content with a high participation level is presented.

[0119] When allocating content presentation locations to viewing areas, it is preferable to take into consideration the size of the venue where the content will be presented. For example, it is conceivable to allocate content presented in the largest of multiple venues to the largest of multiple viewing areas in the user viewing space.

[0120] Furthermore, when allocating the presentation position of content to a viewing area, it is preferable to take into consideration the distribution format of the content video. That is, the user's viewing experience will differ depending on whether the display device for displaying the content in the viewing area is a fixed angle of view (including switching) based on an FPD, or a 360-degree video including a free viewpoint like eyewear. Therefore, it is conceivable to determine the content to be allocated to the viewing area depending on whether the distribution format of the content in the viewing area is, for example, a fixed angle of view or a 360-degree video.

[0121] (Specific example of content allocation to viewing space) Next, the allocation of content to the viewing space will be explained in more detail. For example, suppose there is a collective event such as a music festival featuring performers W, X, Y, and Z. The viewing space is a room shown as a 3D model 40 in Fig. 11, and the content presented at this collective event is spatially arranged. Furthermore, each content is a 360° video in which the viewpoint and sound image change depending on the position and orientation of user 30.

[0122] In this case, four reference positions A to D are set for the room that will be used as the viewing space, as shown in Figure 11. When spatial continuity is taken into consideration, the viewing areas in which content can be placed are set as three viewing areas indicated by area information [1], [2], and [3], as shown in section (b) of Figure 11. Therefore, the upper limit for the number of pieces of content that can be placed simultaneously in the viewing space is set to [3]. However, since there are four pieces of content that need to be placed for the target event, the placement priority is determined.

[0123] Fig. 17 is a schematic diagram showing an example of information used to determine the placement of each piece of content, which can be applied to an embodiment. In the example of Fig. 17, this information includes the items "performer," "genre," "user preferences," "user participation," and "venue size." The item "placement area" on the right side of Fig. 17 indicates the viewing area in which each piece of content will be placed, determined based on the above items. Fig. 17 also shows information regarding content presented in parallel during a specific time period at an event.

[0124] The item "Performer" indicates the performers who will be presenting content in parallel during the time period, and in the example of Fig. 17, performers W, X, Y, and Z are shown. The information in the item "Genre" is genre information indicating the type (genre) of content presented by the performer indicated in the item "Performer". In the example of Fig. 17, the genre information indicating the genre of content presented by each performer W, X, Y, and Z is [Rock], [Rock], [Idol], and [Jazz], respectively.

[0125] In FIG. 17, the information in the "User Preference" item is preference information indicating the user's preference for the performer indicated in the "Performer" item or the content presented by that performer. This preference information in the "User Preference" item is input by the user 30 viewing each content and indicates the user's 30 preference, i.e., the degree of liking, for the corresponding performer. In the example of FIG. 17, the information in the "User Preference" item indicates the preference levels as "High," "Low," "Medium," and "Medium," respectively, for performers W, X, Y, and Z. A value of "High" in the "User Preference" item indicates that the user 30 has a high preference for the corresponding performer or the content presented by that performer; a value of "Low" indicates a low preference; and a value of "Medium" indicates a medium preference.

[0126] 17, the information in the item "User participation level" is participation level information indicating the level of participation of all users who watch the content presented by the performer indicated in the item "Performer" in the content. In this case, the user participation level may be obtained, for example, from the reference DB 72. However, the item "User participation level" may also be the participation level input by the user 30 who watches the content in his or her own viewing space.

[0127] The information in the item "venue size" is size information that indicates the size of the venue in real space where the content will be presented by the performer indicated in the item "performer."

[0128] The map generating unit 112 weights the content to be placed in the audio-visual space based on the value of each item in Fig. 17. This weighting will be described in detail below.

[0129] The map generating unit 112 first prioritizes the content to be placed in the viewing space according to the value of the item "user preference" in Fig. 17. In the example of Fig. 17, the value of the item "user preference" for performer X is set to [low], so the map generating unit 112 excludes performer X from the content to be placed.

[0130] Next, performers W, Y, and Z are positioned. As can be seen from Figure 11, the viewing area indicated by area information [1] is the largest in this space. Therefore, the map generation unit 112 positions performer W, whose "venue size" item is [Large], as indicated in the "Position area" item.

[0131] Next, for performers Y and Z, the item "user preference" is "medium," meaning that they are equally preferred by user 30. However, the item "user participation" is "high" for performer Y and "low" for performer Z, meaning that performer Y is more airtight. In this case, the viewing area indicated by area information [2] is the "kitchen," and the viewing area indicated by area information [3] is the "bathroom." It is therefore assumed that the viewing area indicated by area information [3] is more airtight than the viewing area indicated by area information [2]. Therefore, it is assumed that the viewing area indicated by area information [3] is more suitable for users 30 to cheer on the presented content, for example. Therefore, the map generation unit 112 places performer Y in the viewing area indicated by area information [3] and place performer Z in the viewing area indicated by area information [2], as shown in the item "placement area."

[0132] By experiencing the event while moving through the viewing space in which content is arranged in each viewing area, the user 30 can have an event experience that is tailored to the surrounding environment of the user 30.

[0133] FIG. 18 is a schematic diagram illustrating an example of a user interface for arranging content in a viewing area, which is applicable to the embodiment.

[0134] As described above, the user interface includes at least one of a screen that presents information to the user and an input unit through which the user inputs information. In the examples of sections (a) to (d) of FIG. 18, the user interface includes the screen and the input unit. When an eyewear device is used as the terminal device 10, the input unit may include an outward-facing camera 1101 that captures an image of the hand of the user 30 for hand tracking, for example.

[0135] It is assumed that spatial information of the environment (viewing space) in which user 30 will view each content of the event has already been acquired through the process described with reference to Fig. 6. User 30, for example, purchases a ticket for the content that can be viewed, and performs tasks such as signing in.

[0136] In terminal device 10, control unit 100 causes output control unit 113 to display user interface screen 600 shown in section (a) of Fig. 18 on a display unit (for example, at least one of display units 1201L and 1201R). User interface screen 600 displays a three-dimensional model 610 of the viewing space of user 30 and a message 611 that prompts user 30 to confirm the viewing area and add or delete a viewing area.

[0137] User 30 can check the environmental information of his / her own viewing space based on user interface screen 600. Control unit 100 presents the maximum number of areas that can be used as viewing areas based on the environmental information (spatial information) of the viewing space, and prompts user 30 to confirm whether each viewing area is available for use, for example, by message 611. In the example shown in the figure, three-dimensional model 610 shows three viewing areas based on reference positions A to C. This is because the situation of the viewing space may differ between when the spatial information is acquired and when it is used, and it is necessary to determine whether each viewing area is available for use depending on the situation.

[0138] Furthermore, the control unit 100 may enable adjustment of the viewing area at this stage. For example, if the presented viewing area has a small layout or includes a dangerous area, the user 30 can make adjustments such as enlarging or reducing the viewing area or setting an excluded area for the viewing area based on this user interface screen 600.

[0139] Next, the control unit 100 presents a user interface screen 601 shown in section (b) of FIG. 18 on which the user 30 inputs information indicating preferences for each piece of content. In the example shown in the figure, the user interface screen 601 displays an event timetable 521. In this example, for the time slot from 10:00 to 11:30, performer W is located on the main stage (main), and performers X, Y, and Z are located on sub-stages #1 to #3 (sub-stages #1 to #3), respectively. Furthermore, for the time slot from 11:30 to 12:30, performer A is located on the main stage, and performers C and D are located on sub-stages #2 and #3, respectively. Furthermore, performer B is located for the time slot from 12:00 to 12:30.

[0140] 18, the user 30 inputs preference information indicating his / her own preferences for the timetable 521 to this user interface screen 601. In the example shown in the figure, as an operation for inputting preference information, the user 30 performs an operation on the terminal device 10 to mark 620 a performer in the timetable 521 who is highly to his / her preference, i.e., a performer whose content he / she plans to watch.

[0141] In the example of section (c) in Fig. 18, as shown in Fig. 17, marks 620 are added to performers W, Y, and Z in the time slot from [10:00] to [11:30], indicating that the content presented by performers W, Y, and Z is scheduled to be viewed during that time slot. Also, in the example of the figure, marks 620 are added to performer C in the time slot from [11:30] to [12:30], and to performer B in the time slot from [12:30] to [12:30], indicating that the content presented by performers C and D is scheduled to be viewed during that time slot.

[0142] When the settings in section (c) of Fig. 18 are confirmed, the control unit 100 causes the display unit to display a user interface screen 603 for confirming each set viewing area, as shown in section (d) of Fig. 18. The example shown in the figure shows how performers W, Y, and Z are automatically positioned at reference positions A to C indicating each viewing area in a three-dimensional model 610 representing the viewing space of the user 30. Also, in the example shown in the figure, a message 612 is displayed on the user interface screen 603 to prompt the user 30 to confirm whether or not the automatically positioned content is acceptable.

[0143] In the example of section (c) of FIG. 18, a mark 620 indicating a high preference is directly attached to the timetable 521, but this is not limited to this example. For example, the user 30 may input a preferred genre and set content to be viewed based on recommendations from the system. In this case, for example, the control unit 100 performs the weighting process described above for the genre, and the results of automatic arrangement according to the weighting results are reflected in the user interface screen 603 shown in section (d) of FIG. 18. The user 30 confirms the arrangement based on this user interface screen 603.

[0144] Furthermore, for example, the arrangement of content in each viewing area is not limited to automatic arrangement by the control unit 100, but may be performed manually by the user 30. For example, the user 30 may manually change the arrangement of content in response to an operation on the screen by the user 30.

[0145] Furthermore, for example, the control unit 100 may automatically arrange the content in each viewing area by directly using the above-described preset map 530. Furthermore, the arrangement based on the preset map 530 may be edited, such as rotated or flipped, in response to an operation by the user 30, so that the user 30 can explicitly change the arrangement.

[0146] Fig. 19 is a schematic diagram showing an example of the arrangement of each content in each viewing area according to an embodiment. Section (a) of Fig. 19 is the same as section (a) of Fig. 14 described above, and shows an example of a preset map 530 in which areas 500lay to 502lay corresponding to stages 500 to 502 are arranged in accordance with the positional relationship of the stages 500 to 502.

[0147] Section (b) of Fig. 19 is an example in which the layout of each of the areas 500lay to 502lay in the preset map 530 is applied to the 3D model 40 showing the viewing space with the same positional relationship. In this example, each of the areas 500lay to 502lay is displayed as a viewing area, superimposed on the image of the 3D model 40.

[0148] In the example of section (b) in FIG. 19, an area 500lay corresponding to the stage 500, which is the main stage, is arranged according to the reference position A.

[0149] With respect to the area 500lay, an area 501lay corresponding to the stage 501 arranged northwest of the stage 500 as substage #1 is arranged in a position at the upper left of the area 500lay in accordance with the relative positional relationship between the stage 500 and the stage 501. Furthermore, with respect to the area 500lay, an area 502lay corresponding to the stage 502 arranged southwest of the stage 500 as substage #2 is arranged in a position at the lower left of the area 500lay in accordance with the relative positional relationship between the stage 500 and the stage 502.

[0150] Section (c) of Fig. 19 shows an example in which the arrangement of the areas 500lay to 502lay according to section (b) of Fig. 19 is changed in response to an operation by the user 30. The control unit 100, for example, causes an image as shown in section (b) of Fig. 19 to be displayed on the screen of the display unit. The user 30 can specify an area to be moved among the areas 500lay to 502lay on the screen and move it.

[0151] In the example of section (c) in Fig. 19, the position of area 502lay is moved to the right to become area 502lay' as the viewing area, and area 500lay is shrunk vertically on the drawing so as not to overlap with area 502lay to become area 500lay' as the viewing area. Such changes in shape accompanying the movement of the areas are controlled by control unit 100.

[0152] [4. Viewing process according to the embodiment] Next, a content viewing process according to the embodiment will be described with reference to Fig. 20, which is a flowchart illustrating an example of the content viewing process according to the embodiment.

[0153] 20, in step S300, the terminal device 10 is started up in response to, for example, an operation by the user 30, and an information processing program according to the embodiment is executed, thereby configuring the control unit 100. In the terminal device 10, the control unit 100 requests event information from, for example, the server 20. In response to this request, the server 20 transmits the event information to the terminal device 10. In the terminal device 10, the control unit 100 causes the display unit to display a screen for specifying the event transmitted from the server 20.

[0154] In the next step S301, the control unit 100 designates a target event in response to an operation by the user 30. In response to this event designation operation, the control unit 100 requests event information, including performer information, an event map 50, a timetable 520, etc., from the server 20. The control unit 100 acquires the event information transmitted from the server 20 in response to this request.

[0155] In the next step S302, the control unit 100 determines whether or not the viewing environment of the user 30 is associated with each piece of content presented in the event. If the control unit 100 determines that such association is not established (step S302, "No"), the control unit 100 proceeds to step S303. In step S303, the control unit 100 executes the process according to the flowchart of FIG. 6 described above, and allocates each piece of content in the event to each viewing area in the viewing space of the user 30. Then, the process proceeds to step S304.

[0156] On the other hand, if the control unit 100 determines in step S302 that correspondence has been established with each content presented within the event (step S302, "Yes"), it skips the processing of step S303 and proceeds to step S304.

[0157] In step S304, the control unit 100 determines whether multiple pieces of content are associated with the viewing space of the user 30. If the control unit 100 determines that only a single piece of content is associated with the viewing space of the user 30 (step S304, "No"), the control unit 100 proceeds to step S310. In step S310, the control unit 100 presents the content in the viewing space and ends the series of processes.

[0158] On the other hand, if the control unit 100 determines in step S304 that multiple contents are associated with the viewing space of the user 30 (step S304, "Yes"), the control unit 100 proceeds to step S305. In this case, the display positions of the multiple contents are respectively arranged in multiple viewing areas within the viewing space.

[0159] In the next step S305, the control unit 100 selects content from the plurality of contents according to the position of the user 30. That is, the control unit 100 acquires the current position of the user 30 using the position measurement unit 102, and detects in which viewing area within the viewing space the user 30 is located based on the acquired current position. The control unit 100 selects from the plurality of contents according to the viewing area corresponding to the position of the user 30.

[0160] In the next step S306, the control unit 100 outputs the content selected in step S305. For example, the control unit 100 extracts the content selected in step S305 from the content presented on each of the stages 500 to 502, which is transmitted from the streaming server 70. The control unit 100 outputs the extracted content in the viewing area where the user 30 is currently located.

[0161] In the next step S307, the control unit 100 determines whether the user 30 is about to move to another viewing area. The control unit 100 can detect whether the user 30 is moving and the direction of movement, for example, based on the positioning output from the position positioning unit 102 and the outputs of the gyro sensor 1104, the acceleration sensor 1105, and the direction sensor 1106. The control unit 100 can infer whether the user 30 is about to move to another viewing area based on the position and direction of movement of the user 30.

[0162] If the control unit 100 determines that the user 30 is about to move to another viewing area (step S307, "Yes"), the control unit 100 proceeds to step S308. In step S308, the control unit 100 displays, on the display unit, a transition effect that occurs when moving across viewing areas. Specific examples of the transition effect will be described later. After displaying the transition effect in step S308, the control unit 100 returns the process to step S305.

[0163] On the other hand, if the control unit 100 determines in step S307 that the user 30 is not moving to another viewing area (step S307, "No"), it proceeds to step S320. In step S320, the control unit 100 determines whether the user 30 has finished viewing all the content of the event. If the control unit 100 determines that viewing has ended, for example, in response to a predetermined operation on the terminal device 10 (step S320, "Yes"), it terminates the series of processes according to the flowchart in FIG. 20.

[0164] On the other hand, if the control unit 100 determines that the user 30 has not finished viewing the content (step S320, "No"), the control unit 100 returns the process to step S306.

[0165] (About transition effect display) Next, the display of the transition effect in step S308 described above will be described. As an example, consider a case where user 30 moves from viewing area A, where he or she is viewing content A, to viewing area B, where he or she is viewing content B. In this case, the moment user 30 moves from viewing area A to viewing area B, the display on the display unit switches from content A to content B. Such an abrupt change in the displayed content may ruin the sense of immersion. For this reason, it is preferable to provide a transition area, for example, shown as area 543 in FIG. 15, at the boundary between viewing area A and viewing area B, and to apply predetermined video processing in this transition area as a transition effect, which indicates the transition from content A to content B.

[0166] As an example of a transition effect, the control unit 100 may pre-read the content to be output in the destination viewing area B, fade out content A in the transition area, and fade in content B, thereby processing content A and content B. In this case, the control unit 100 may set the amount of fade-in and fade-out depending on the direction and speed at which the user 30 is moving.

[0167] As another example of a transition effect, the control unit 100 may output, in the transition area, content for movement that has been prepared in advance. The content for movement may be, for example, a video reconstructed from a three-dimensional model, which is output in the transition area. For example, the control unit 100 may reconstruct, as a three-dimensional model, the scenery along the route between venues in real space. Since the amount of movement from viewing area A to viewing area B is smaller than the amount of movement in the actual venue, in these cases, the playback speed of the video output in the transition area may be set to high-speed playback.

[0168] Here, when user 30 moves within the viewing area, control unit 100 may adjust the amount of movement in the transition effect depending on the size of the area in which the user is placed. For example, even if the actual amount of movement of the user is the same, the relative amount of movement within the viewing area differs depending on whether the set viewing area is large or small. Therefore, it is preferable that control unit 100 adjusts the amount of movement within the content, taking into account the size of the set viewing area and the size of the venue in the real space where the content assigned to the viewing area is presented.

[0169] Furthermore, if it is difficult to spatially separate viewing area A and viewing area B, for example, control unit 100 may encourage user 30 to change the lighting environment, thereby promoting a bodily sensation of separation between viewing area A and viewing area B. For example, it may be possible to differentiate the brightness of the lighting in viewing area A and viewing area B.

[0170] When multiple people view content presented at an event in the same viewing area, if multiple users enter the same viewing area, the size of the viewing area may be expanded from the initially set size. That is, the map generation unit 112 can set the viewing area according to the number of users included in the viewing area. Furthermore, the map generation unit 112 can dynamically set the viewing area according to changes in the number of users included in the viewing area.

[0171] 21 is a schematic diagram showing an example of setting a viewing area according to the number of users in the viewing area according to the embodiment. Initially, as shown in section (a) of FIG. 21, it is assumed that a user 31 wearing his / her own terminal device 10 is initially present in viewing area 550a (area #1), and a user 30 similarly wearing his / her own terminal device 10 is present in viewing area 551a (area #2). The control unit 100 sets the size and the like of viewing area 551a, for example, assuming that the number of users is one.

[0172] Consider a case where user 31 moves from viewing area 550a to viewing area 551a where user 30 is present. In this case, two users, users 30 and 31, are present in viewing area 551a. If viewing area 551a is set assuming one user 30, there is a possibility that users 30 and 31 may come into contact or collide with each other when users 30 and 31 move. Therefore, as shown in section (b) of FIG. 21 , control unit 100 expands viewing area 551a to become viewing area 551b corresponding to two users, users 30 and 31. Viewing area 550a where user 31 is no longer present and the number of users is [0] is set to reduced viewing area 550b.

[0173] Furthermore, when multiple users view content at a collective event in their own environments (viewing spaces), the amount of movement adjustment within each environment and the time required for physical movement will differ depending on the environment. Therefore, to ensure that each user experiences moving between content items together at the same event, the terminal device 10 worn by each user may adjust the movement speed within the environment between users and perform processing to ensure consistency in movement time. In this case, the adjustment of movement time between users may be performed by the server 20 or by direct communication between each terminal device 10.

[0174] In the above description, the information processing system 1 according to the embodiment has been described as being targeted at each piece of content presented at an event held in a real space, but this is not limited to this example. That is, the information processing system 1 according to the embodiment can also be applied to an event held in a virtual space. Furthermore, the information processing system 1 can also be applied to an event held in a two-dimensional space that presents content based on two-dimensional information such as video and paintings.

[0175] As described above, the information processing system 1 according to the embodiment allows users to virtually experience moving between multiple venues by moving between viewing areas set up within the viewing space. This allows users to feel more immersed in a remote event streamed.

[0176] Furthermore, according to the information processing system 1 according to the embodiment, the display position of each content in the user's viewing space is set in accordance with the relative positions of the venues in the real space. This allows the user to virtually experience moving between the venues in the real space in the viewing space, making it possible to feel more immersed in the remote event via streaming.

[0177] Furthermore, in the information processing system 1 according to the embodiment, when a user moves across viewing areas, a transition effect can be applied to the transition of content. Therefore, even when a user moves across viewing areas to view different content, the sense of discomfort felt by the user due to an abrupt change in content can be reduced. This allows the user to feel more immersed in a remote event streamed.

[0178] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0179] The present technology can also be configured as follows. (1) a control unit that determines presentation positions of the plurality of contents in the viewing space based on first spatial information about a distribution source space to which the plurality of contents are associated and second spatial information about a viewing space in which the plurality of contents are viewed; Equipped with Information processing device. (2) the viewing space includes a plurality of viewing areas; The control unit determining the presentation positions for each of the plurality of content items with respect to each of the plurality of viewing areas based on the first spatial information and the second spatial information; The information processing device according to (1) above. (3) The plurality of viewing areas are movable between areas by a user viewing the plurality of contents. The information processing device according to (2) above. (4) the first spatial information includes first position information indicating positions of a plurality of presentation areas in which the plurality of contents are respectively presented; the second spatial information includes second position information indicating positions of the plurality of viewing areas; The control unit determining, based on the first position information and the second position information, the presentation positions of the plurality of contents to the plurality of viewing areas that correspond in positional relationship to the plurality of presentation areas, respectively; The information processing device according to (2) or (3). (5) the first spatial information includes preference information indicating a preference of a user who views each of the plurality of contents for the plurality of contents, and The control unit selecting, from the plurality of contents, contents whose presentation positions are the plurality of viewing areas based on the preference information included in the first spatial information; The information processing device according to any one of (2) to (4). (6) the first spatial information includes information indicating the type of each of the plurality of contents; The control unit controlling, based on the information indicating the type included in the first spatial information, which of the plurality of viewing areas to determine the presentation position of each of the plurality of contents; The information processing device according to any one of (2) to (5). (7) The control unit setting the plurality of viewing areas for the viewing space based on at least one of division information, role information, structural information, floor surface information, shape information, and facility information; The information processing device according to any one of (2) to (6). (8) The control unit presenting a predetermined effect in an area including a boundary between two adjacent viewing areas among the plurality of viewing areas; The information processing device according to any one of (2) to (7). (9) The predetermined effect includes at least one of a visual effect and an auditory effect. The information processing device according to (8). (10) the predetermined effect is a cross-fade of content presented in each of the two viewing areas; The information processing device according to (8) or (9). (11) the predetermined effect is the presentation of a prepared content for movement; The information processing device according to (8) or (9). (12) The content for movement is The content is based on the scenery of a movement route when moving between a position where a content of the plurality of contents, of which one of the two viewing areas is the presentation position, is presented in the distribution source space, and a position where a content of which the other of the two viewing areas is the exhibition position is presented in the distribution source space. The information processing device according to (11) above. (13) The content for movement is The content is a reconstructed 3D model of the scenery. The information processing device according to (12) above. (14) The control unit The moving content is presented at a speed according to the distance between a position where one of the two viewing areas among the plurality of contents is presented in the distribution source space, and a position where the other of the two viewing areas among the plurality of contents is presented in the distribution source space, The information processing device according to any one of (11) to (13). (15) The control unit changing a correspondence relationship between each of the plurality of contents and each of the plurality of viewing areas according to a time period in which the plurality of contents are presented in the distribution source space; The information processing device according to any one of (2) to (14). (16) The control unit presenting a user interface for editing a correspondence between each of the plurality of contents and each of the plurality of viewing areas; The information processing device according to any one of (2) to (15). (17) The control unit setting at least one of the plurality of viewing areas based on the number of users in the viewing area; The information processing device according to any one of (2) to (16). (18) The control unit setting the viewing area according to a change in the number of users; The information processing device according to (17) above. (19) Executed by a processor, a control step of determining presentation positions of the plurality of contents in the viewing space based on first spatial information on a distribution source space to which the plurality of contents are associated and second spatial information on a viewing space in which the plurality of contents are viewed; having Information processing methods. (20) On the computer, a control step of determining presentation positions of the plurality of contents in the viewing space based on first spatial information on a distribution source space to which the plurality of contents are associated and second spatial information on a viewing space in which the plurality of contents are viewed; A computer-readable recording medium on which an information processing program for executing the above is recorded. [Explanation of symbols]

[0180] 1. Information Processing Systems 10 Terminal Equipment 20 servers 30,31 users 40,610 3D models 50 Event Map 70 Streaming Server 71 Content DB 72 Reference DB 100 control section 101 Sensor unit 102,1509 Positioning unit 103 Output section 110 User information acquisition unit 111 Environmental Information Acquisition Department 112 Map Generation Unit 113 Output control section 114 Content information acquisition unit 115 Communications Department 116 Operation input section 300,600,601,603 User interface screen 500, 501, 502 stages 500lay,500lay',501lay,502lay,502lay',540lay,541lay,542lay,543,544 area 520,521 Timetable 530 preset maps 620 marks 1101 Outward-facing camera 1104 Gyro sensor 1105 Acceleration Sensor 1106 Orientation Sensor 1201L,1201R Display section 1505 Storage device

Claims

1. a control unit that determines presentation positions of the plurality of contents in the viewing space based on first spatial information about a distribution source space to which the plurality of contents are associated and second spatial information about a viewing space including a plurality of viewing areas in which the plurality of contents are viewed; Equipped with the plurality of viewing areas are movable between areas by a user viewing the plurality of contents; The control unit determining the presentation positions of the plurality of contents with respect to the plurality of viewing areas based on the first spatial information and the second spatial information; Information processing device.

2. the first spatial information includes first position information indicating positions of a plurality of presentation areas in which the plurality of contents are respectively presented; the second spatial information includes second position information indicating positions of the plurality of viewing areas; The control unit determining, based on the first position information and the second position information, the presentation positions of the plurality of contents to the plurality of viewing areas that correspond in positional relationship to the plurality of presentation areas, respectively; The information processing device according to claim 1 .

3. the first spatial information includes preference information indicating a preference of a user who views each of the plurality of contents for the plurality of contents, and The control unit selecting, from the plurality of contents, contents whose presentation positions are the plurality of viewing areas based on the preference information included in the first spatial information; The information processing device according to claim 1 .

4. the first spatial information includes information indicating a type of each of the plurality of contents; The control unit controlling, based on the information indicating the type included in the first spatial information, in which of the plurality of viewing areas a presentation position of each of the plurality of contents is to be determined; The information processing device according to claim 1 .

5. The control unit setting the plurality of viewing areas for the viewing space based on at least one of division information, role information, structural information, floor surface information, shape information, and facility information; The information processing device according to claim 1 .

6. The control unit presenting a predetermined effect in an area including a boundary between two adjacent viewing areas among the plurality of viewing areas; The information processing device according to claim 1 .

7. The predetermined effect includes at least one of a visual effect and an auditory effect. The information processing device according to claim 6 .

8. the predetermined effect is a cross-fade of content presented in each of the two viewing areas; The information processing device according to claim 6 .

9. the predetermined effect is the presentation of a prepared content for movement; The information processing device according to claim 6 .

10. The content for movement is The content is based on a view of a travel route when moving between a position where a content of the plurality of contents, of which one of the two viewing areas is the presentation position, is presented in the distribution source space, and a position where a content of which the other of the two viewing areas is the presentation position is presented in the distribution source space. The information processing device according to claim 9 .

11. The content for movement is The content is a reconstructed three-dimensional model of the scenery. The information processing device according to claim 10.

12. The control unit The content for movement is presented at a speed according to a distance between a position where a content of the plurality of contents, the presentation position of which is one of the two viewing areas, is presented in the distribution source space, and a position where a content of the plurality of contents, the presentation position of which is the other of the two viewing areas, is presented in the distribution source space. The information processing device according to claim 9 .

13. The control unit changing a correspondence relationship between each of the plurality of contents and each of the plurality of viewing areas according to a time period in which the plurality of contents are presented in the distribution source space; The information processing device according to claim 1 .

14. The control unit presenting a user interface for editing a correspondence between each of the plurality of contents and each of the plurality of viewing areas; The information processing device according to claim 1 .

15. The control unit setting at least one of the plurality of viewing areas based on the number of users in the viewing area; The information processing device according to claim 1 .

16. The control unit setting the viewing area according to a change in the number of users; The information processing device according to claim 15.

17. Executed by a processor, a control step of determining presentation positions of the plurality of contents in the viewing space based on first spatial information about a distribution source space to which the plurality of contents are associated and second spatial information about a viewing space including a plurality of viewing areas in which the plurality of contents are viewed; and the plurality of viewing areas are movable between areas by a user viewing the plurality of contents; The control step determining the presentation positions of the plurality of contents with respect to the plurality of viewing areas based on the first spatial information and the second spatial information; Information processing methods.

18. On the computer, a control step of determining presentation positions of the plurality of contents in the viewing space based on first spatial information about a distribution source space to which the plurality of contents are associated and second spatial information about a viewing space including a plurality of viewing areas in which the contents are viewed; Execute the plurality of viewing areas are movable between areas by a user viewing the plurality of contents; The control step determining the presentation positions of the plurality of contents with respect to the plurality of viewing areas based on the first spatial information and the second spatial information; A computer-readable recording medium on which an information processing program for the purpose is recorded.

Images