Information processing device, information processing method, and recording medium

Abstract

The objective is to provide an information processing device, an information processing method, and a recording medium that can give users a greater sense of immersion in remote events delivered via streaming. [Solution] The information processing device according to the present disclosure includes a control unit that determines the presentation position of the multiple contents in the viewing space based on first spatial information relating to a distribution source space to which multiple contents are associated, and second spatial information relating to a viewing space including multiple viewing areas in which the multiple contents are viewed. The control unit determines, based on the first spatial information and the second spatial information, which area of ​​the multiple viewing areas each of the multiple contents is placed in.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a recording medium.

Background Art

[0002] With the development of high-speed communication networks, real-time distribution of events such as music concerts has become popular. On the other hand, in actual events, the mobile experience before and after participating in the event is important for the user to obtain a sense of immersion. In particular, in a collective event (for example, a music festival) where there are multiple performers and locations, the experience of actively moving between venues is very important for the user to obtain a sense of immersion.

[0003] Patent Document 1 discloses a technique for switching the viewpoint of video in a head-mounted display. According to the technique described in Patent Document 1, in a music festival or the like, it is possible to switch and display the video of one venue and the video of another venue.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technique described in Patent Document 1, the movement of the user between multiple venues is not considered, and it is difficult to give a sufficient sense of immersion in a remote event experience that requires a sense of immersion.

[0006] An object of the present disclosure is to provide an information processing apparatus, an information processing method, and a recording medium that can give a user a greater sense of immersion in a remote event by distribution.

Means for Solving the Problems

[0007] The information processing device according to this disclosure includes a control unit that determines the presentation position of the multiple contents in the viewing space based on first spatial information relating to a distribution source space to which multiple contents are associated, and second spatial information relating to a viewing space including multiple viewing areas in which the multiple contents are viewed. The control unit determines, based on the first spatial information and the second spatial information, which area of ​​the multiple viewing areas each of the multiple contents is placed in. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic diagram showing the configuration of an example of an information processing system applicable to the embodiment. [Figure 2] This is a schematic diagram illustrating the appearance of an example of a terminal device as eyewear applicable to the embodiment. [Figure 3] This block diagram shows the hardware configuration of an example of a terminal device as eyewear, applicable to the embodiment. [Figure 4] This is an example of a functional block diagram illustrating the functions of a terminal device according to an embodiment. [Figure 5] This is an example flowchart illustrating the information processing method according to the embodiment. [Figure 6] This is an example flowchart showing an acquisition process for obtaining user-related environmental information according to the embodiment. [Figure 7] This is a schematic diagram illustrating an example of image information and depth information acquired by an outward-facing camera. [Figure 8] This is a schematic diagram illustrating an example of a user interface screen that prompts the user to acquire image and depth information for the appropriate region. [Figure 9] This is a schematic diagram showing an example of 3D model data of the user's environment, generated by data integration by the environmental information acquisition unit. [Figure 10A]This is a schematic diagram illustrating an example of a method for setting up multiple viewing areas in a viewing space, applicable to the embodiment. [Figure 10B] This is a schematic diagram illustrating an example of a method for setting up multiple viewing areas in a viewing space, applicable to the embodiment. [Figure 10C] This is a schematic diagram illustrating an example of a method for setting up multiple viewing areas in a viewing space, applicable to the embodiment. [Figure 10D] This is a schematic diagram illustrating an example of a method for setting up multiple viewing areas in a viewing space, applicable to the embodiment. [Figure 11] This is a schematic diagram showing an example of 3D model data and area information stored by the control unit according to the embodiment. [Figure 12] This is a schematic diagram illustrating an example of a map showing the spatial relationships between multiple pieces of content in a collective event. [Figure 13] This is a schematic diagram showing an example of an event timetable created by the broadcasting side. [Figure 14] This diagram illustrates the arrangement of regions corresponding to each stage in a grid, applicable to the embodiment. [Figure 15] This is a schematic diagram illustrating an example of placing areas other than the event venue on a preset map, applicable to the embodiment. [Figure 16] This is an example flowchart illustrating the content allocation process for a viewing space according to an embodiment. [Figure 17] This is a schematic diagram illustrating an example of information used to determine the placement of each piece of content, applicable to the embodiment. [Figure 18] This is a schematic diagram illustrating an example of a user interface for placing content in a viewing area, applicable to the embodiment. [Figure 19] This is a schematic diagram showing an example of the arrangement of each content item in each viewing area according to the embodiment. [Figure 20] This is an example flowchart illustrating the content viewing process according to the embodiment. [Figure 21] It is a schematic diagram showing an example of setting a viewing area according to the number of users in the viewing area according to an embodiment.

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present disclosure will be described in detail based on the drawings. In the following embodiments, the same parts are denoted by the same reference numerals, and redundant explanations are omitted.

[0010] Hereinafter, embodiments of the present disclosure will be described in the following order. 1. Outline of Embodiments of the Present Disclosure 2. Applicable Configurations to Embodiments 3. Processes According to Embodiments 3-1. Outline of Processes According to Embodiments 3-2. Details of Processes According to Embodiments 3-2-1. Environmental Information Acquisition Process 3-2-2. Assignment of Events to Viewing Environments 4. Viewing Processes According to Embodiments

[0011] [1. Outline of Embodiments of the Present Disclosure] First, embodiments of the present disclosure will be schematically described. The present disclosure relates to a technique for viewing an event held in a three-dimensional space in a viewing space remote from the venue of the event.

[0012] More specifically, an event is held using a plurality of venues in a three-dimensional space. The three-dimensional space where each venue is installed may be a real space or a virtual space generated by executing a program on a computer. Content is associated with each venue. The type of the event is not particularly limited as long as it is an event where a performer presents content to an audience on a stage or the like. If the event is a music event, the content is, for example, a performance related to music such as singing, playing, and dancing of a music piece.

[0013] Here, we assume that the event in question is a collective event using multiple venues and featuring multiple performers. If the event is music-related, such a collective event would be called a music festival, for example. Hereafter, collective events may simply be referred to as "events."

[0014] The information processing system according to the embodiment of this disclosure sets a viewing space for viewing the event, such as the user's home, which is far from the event venue. The information processing system according to the embodiment acquires first spatial information relating to a three-dimensional space as a distribution source space to which multiple contents are associated. The information processing system according to the embodiment also acquires second spatial information relating to the viewing space in which the user views multiple contents of the event. Based on the acquired first and second spatial information, the information processing system according to the embodiment determines each presentation position in the viewing space where each of the multiple contents is presented.

[0015] In the embodiments of this disclosure, such a configuration allows users to virtually experience movement between multiple venues within the viewing space, thereby providing users with a greater sense of immersion in remote events delivered via streaming.

[0016] [2. Configurations applicable to the embodiment] Next, a configuration applicable to the embodiment will be described. Figure 1 is a schematic diagram showing the configuration of an example of an information processing system applicable to the embodiment. In Figure 1, the information processing system 1 according to the embodiment includes terminal devices 10 and a server 20 that are connected to each other via a network 2 such as the Internet. In Figure 1, the server 20 is shown as a single device, but this is just an example, and the server 20 may be composed of a group of computers with distributed functions, or it may be a group of computers connected by cloud computing technology.

[0017] The terminal device 10 constitutes an information processing device according to the embodiment and is used by a user 30 to view content from an event. It can be a wearable device such as glasses (eyewear device) or a smartphone. However, it is not limited to these, and the terminal device 10 may be a general information processing device such as a personal computer or a game console, and may be configured by connecting an FPD (Flat Panel Display). Furthermore, the terminal device 10 can output sound using an audio output device such as a speaker, headphones, or earphones. In the following description, unless otherwise specified, the terminal device 10 will be described as an eyewear device.

[0018] The eyewear device applicable as the terminal device 10 may be a so-called see-through type head-mounted display (AR (Augmented Reality) glasses) that allows images of real space to pass through, or a goggle type that does not allow images of real space to pass through (VR (Virtual Reality) goggles). When using VR goggles as the terminal device 10, it is preferable to provide a means that allows the user 30 wearing the terminal device 10 to easily confirm the surrounding situation in real space, for example, by displaying images of the surroundings captured by an outward-facing camera installed in the VR goggles.

[0019] In the example shown in Figure 1, a streaming server 70, a content DB (database) 51, and a reference DB 72 are further connected to network 2. The streaming server 70 processes real-time event video data. For example, the streaming server 70 processes video data being filmed at each venue of the event and distributes it via network 2. The content DB 71 stores, for example, the video data distributed by the streaming server 70. The reference DB 72 manages information related to the video data distributed by the streaming server 70 and the video data stored in the content DB 71. For example, the reference DB 72 manages information related to this video data, such as the date of the event and event information.

[0020] Although the above description explains that the terminal device 10 constitutes the information processing apparatus according to the embodiment, this is not limited to this example. For example, the entire information processing system 1, including the terminal device 10 and the server 20, may constitute the information processing apparatus according to the embodiment.

[0021] Figure 2 is a schematic diagram illustrating the appearance of an example of a terminal device 10 as an eyewear device applicable to the embodiment. The main body of the terminal device 10 is generally a glasses-type or goggle-type device, worn on the head of a user 30, and can perform functions such as superimposing digital information onto the field of view of both or one eye of the user 30, enhancing or attenuating the image of a specific real object, or removing the image of a specific real object to make it appear as if that real object does not exist. Figure 2 shows the terminal device 10 worn on the head of a user 30.

[0022] In Figure 2, the terminal device 10 has a display unit 1201L for the left eye and a display unit 1201R for the right eye positioned in front of the user 30's left and right eyes, respectively. The display units 1201L and 1201R are transparent or semi-transparent, enabling the superimposition of virtual objects onto a real-world landscape, enhancement or attenuation of images of specific real objects, and the removal of images of specific real objects to make them appear as if they do not exist. The left and right display units 1201L and 1201R may, for example, be driven independently to display parallax images, i.e., virtual objects, as three-dimensional information. Also, an outward-facing camera 1101 is positioned approximately in the center of the terminal device 10, facing the direction of the user 30's line of sight.

[0023] Figure 3 is a block diagram showing the hardware configuration of an example of a terminal device 10 as an eyewear device applicable to the embodiment. In Figure 3, the terminal device 10 includes a CPU 1500, a ROM (Read Only Memory) 1501, a RAM (Random Access Memory) 1502, a camera I / F (interface) 1503, a sensor I / F 1504, a storage device 1505, a display control unit 1506, an audio I / F 1507, a communication I / F 1508, and a positioning unit 1509, all of which are connected to each other via a bus 1520 so as to be able to communicate with each other. Thus, the terminal device 10 has the configuration of a computer (information processing device), including a CPU, memory, and various I / Fs.

[0024] The storage device 1505 is a non-volatile storage medium, such as flash memory. The CPU 1500 operates using the RAM 1502 as work memory, according to information processing programs pre-stored in the storage device 1505 and ROM 1501, and controls the overall operation of the terminal device 10.

[0025] The camera I / F 1503 is an interface to the outward-facing camera 1101 and the inward-facing camera 1102, and supplies the image signals output from the outward-facing camera 1101 and the inward-facing camera 1102 to the bus 1520. In addition, control signals for controlling the outward-facing camera 1101 and the inward-facing camera 1102, generated by the CPU 1500 according to the information processing program, are transmitted to the outward-facing camera 1101 and the inward-facing camera 1102 via the camera I / F 1503.

[0026] The sensor I / F 1504 is an interface to the gyro sensor 1104, the accelerometer 1105, the compass sensor 1106, and the biosensor 1107, and the sensor signals output from each of these sensors are supplied to the bus 1520 via the sensor I / F 1504.

[0027] The display control unit 1506 controls the display operations of the display units 1201L and 1201R according to instructions from the CPU 1500. For example, the display control unit 1506 converts the display control signals generated by the CPU 1500 according to the information processing program into display signals that can be displayed by the display units 1201L and 1201R, and supplies them to the display units 1201L and 1201R.

[0028] The audio interface 1507 is an interface to the microphone 1103 and the sound output unit 1202. For example, the audio interface 1507 converts an analog sound signal based on the sound picked up by the microphone 1103 into a digital sound signal and supplies it to the bus 1520. The audio interface 1507 also converts a digital sound signal, which is supplied via the bus 1520 and generated by the CPU 1500 according to an information processing program, into a signal that can be reproduced by the sound output unit 1202, and supplies it to the sound output unit 1202.

[0029] The communication interface 1508 controls communication between the terminal device 10 and network 2, according to instructions from the CPU 1500.

[0030] The positioning unit 1509 determines the current location of the terminal device 10. The positioning unit 1509 performs positioning using, for example, GNSS (Global Navigation Satellite System) or Wi-Fi (registered trademark) connection information.

[0031] Figure 4 is an example functional block diagram illustrating the functions of a terminal device 10 according to this embodiment. The terminal device 10 includes a control unit 100, a sensor unit 101, a positioning unit 102, and an output unit 103.

[0032] The sensor unit 101 includes various sensors for collecting information about the external environment and the user 30's condition when the terminal device 10 is attached. The sensor unit 101 includes, for example, an outward-facing camera 1101 and an inward-facing camera 1102, as described using Figure 3, a microphone 1103, a gyro sensor 1104, an acceleration sensor 1105, a compass sensor 1106, and a biosensor 1107.

[0033] The outward-facing camera 1101 and the inward-facing camera 1102 are typically electronic cameras that capture light in the visible light range. The wavelengths of light captured by the outward-facing camera 1101 and the inward-facing camera 1102 may include the ultraviolet to infrared range, or they may be limited to a specific wavelength range within these ranges. Furthermore, multiple outward-facing cameras 1101 may be used to measure the depth information of the environment. In addition, the outward-facing camera 1101 may be used in combination with distance sensors such as ToF (Time of Flight) sensors or ultrasonic sensors.

[0034] The inward-facing camera 1102 may acquire the user's gaze information, as well as biometric information such as the number of blinks. It may also estimate ambient light conditions from pupil size.

[0035] The microphone 1103 is intended to understand the external acoustic conditions and the speech status of the user 30. The microphone 1103 may also be used for communication via sound waves in the inaudible range. The microphone 1103 is not limited in its principle of operation; it can be a dynamic microphone or a condenser microphone, for example.

[0036] The gyro sensor 1104, acceleration sensor 1105, and orientation sensor 1106 are used to understand the user 30's movement and posture. These gyro sensors 1104, acceleration sensor 1105, and orientation sensor 1106 may also be used in conjunction to improve the accuracy of positioning by the positioning unit 102, which will be described later.

[0037] The biosensor 1107 measures the user 30's heart rate, blood flow, and other parameters. By measuring these, it is possible to understand the user 30's activity level, stress level, and concentration level.

[0038] The positioning unit 102 corresponds to the positioning unit 1509 in Figure 2 and acquires location information indicating the current location of the user 30. The positioning unit 102 acquires location information using radio waves from GNSS or communication via Wi-Fi.

[0039] In Figure 4, the control unit 100 includes a user information acquisition unit 110, an environment information acquisition unit 111, a map generation unit 112, an output control unit 113, a content information acquisition unit 114, a communication unit 115, and an operation input unit 116. These user information acquisition unit 110, environment information acquisition unit 111, map generation unit 112, output control unit 113, content information acquisition unit 114, communication unit 115, and operation input unit 116 are configured by executing an information processing program according to the embodiment on the CPU 1500. However, some or all of these user information acquisition unit 110, environment information acquisition unit 111, map generation unit 112, output control unit 113, content information acquisition unit 114, communication unit 115, and operation input unit 116 may be configured by hardware circuits that work together.

[0040] The content information acquisition unit 114 acquires, for example, the purchase status and content type of content stored in the storage device 1505. The content information acquisition unit 114 acquires data from an external server (for example, a content DB 71) as needed. If, for example, the target event includes multiple types of content, the content information acquisition unit 114 can acquire information necessary for user 30 to make decisions when viewing the content from multiple external servers.

[0041] The user information acquisition unit 110 acquires information from various sensors included in the sensor unit 101 and location information determined by the positioning unit 102. The user information acquisition unit 110 aggregates and manages the acquired information.

[0042] The environmental information acquisition unit 111 acquires environmental information and map information that indicates the user 30's surrounding environment, based on information such as images captured by the outward-facing camera 1101. The environmental information acquisition unit 111 may also refer to known information from a database of floor plan information acquired separately as environmental information.

[0043] The map generation unit 112 generates a placement map for arranging content in the viewing space based on various information acquired by the environment information acquisition unit 111 and data acquired from the content information acquisition unit 114. When generating the placement map, the map generation unit 112 determines the suitability of each content to the environment. The suitability determination by the map generation unit 112 can be performed, for example, by retrieving reference environment information stored in the storage device 1505 as the ideal viewing environment for the content, and taking into account the user's preferences, etc.

[0044] The output control unit 113 presents information to the user 30 via the output unit 103. The output unit 103 includes, for example, the display units 1201L and 1201R in Figure 3, as well as the audio output unit 1202. The information presented by the output control unit 113 may include video and audio related to the content, as well as information regarding the content's placement.

[0045] The operation input unit 116 detects the user 30's input actions. The operation input unit 116 can detect the user 30's input actions using the sensors included in the sensor unit 101 described above. For example, the operation input unit 116 can detect input actions by recognizing the user 30's hand from the image captured by the outward-facing camera 1101 and using hand tracking technology to track the movement of the recognized hand. However, the operation input unit 116 may also detect user operations on this controller by connecting a controller for user 30 to the terminal device 10.

[0046] The communication unit 115 communicates with the outside of the terminal device 10 via wired or wireless communication. For example, the communication unit 115 communicates with terminal devices used by other users, devices worn by the user 30, or external information processing devices using Wi-Fi, Bluetooth (registered trademark), infrared communication, etc., to acquire information necessary for various judgments and other purposes.

[0047] In the terminal device 10, the CPU 1500 executes the information processing program according to the embodiment, thereby configuring the user information acquisition unit 110, environment information acquisition unit 111, map generation unit 112, output control unit 113, content information acquisition unit 114, communication unit 115, and operation input unit 116 described above as modules, for example, on the main memory area of ​​the RAM 1502.

[0048] The information processing program can be obtained from an external source (e.g., server 20) via a network 2 such as a LAN or the Internet, for example, through communication via the communication I / F 1508, and installed on the terminal device 10. In this case, the information processing program is stored on a storage medium owned by the server 20. However, the information processing program may also be provided stored on a removable storage medium such as a CD (Compact Disk), DVD (Digital Versatile Disk), or USB (Universal Serial Bus) memory.

[0049] [3. Processing of the object] Next, the processing according to the embodiment will be described.

[0050] (3.1. Outline of the process according to the embodiment) The process according to the embodiment will be described in general terms. The information required in the process according to the embodiment can be broadly divided into the following two categories. (A) Information about the content that user 30 views (B) Information regarding the viewing environment of user 30

[0051] (A) The information regarding the content that user 30 will view includes, for example, if the event is a live music performance, the performers, performance style, etc. The information regarding the content may also include location information indicating where the content will be presented in the distribution source space, and a timetable indicating the time period during which the content will be presented. Furthermore, (B) the environment in which user 30 will view the content includes, for example, the device used for viewing, and surrounding space information regarding the space around where user 30 will view the content (such as the type and arrangement of furniture). In this embodiment, by associating this information regarding the content with the information regarding the viewing environment, when placing the content in a physical space (such as user 30's home), a placement that is highly compatible between the content and the physical space is achieved.

[0052] To achieve a content placement that is highly compatible with the physical space, it is desirable that the content being viewed matches the viewing environment. Furthermore, if user 30 explicitly moves to a different physical space and the viewing environment changes, it is desirable that content transitions occur in accordance with that movement. For example, interruptions in viewing during content transitions would lead to a deterioration of the viewing experience.

[0053] Therefore, in this embodiment, for example, when content transitions due to user 30 movement, fade-in and fade-out processing (crossfade processing) may be inserted while considering the loading of the video (content) after the transition. Also, if it is difficult to handle video buffering due to factors such as communication speed, a lower-resolution video or audio-only output may be performed in advance.

[0054] The following describes in more detail the automatic placement of content in the viewing space and the transitions associated with the movement of the user 30, according to the embodiment. The processes described below are mainly executed by the environment information acquisition unit 111, the map generation unit 112, and the content information acquisition unit 114 in the functional block diagram of Figure 4.

[0055] Figure 5 is a flowchart illustrating an example of an information processing method according to an embodiment.

[0056] Prior to the processing of the flowchart in Figure 5, user 30 puts on the terminal device 10 (eyewear device) and starts the information processing program according to the embodiment. User 30 is also assumed to have already obtained information about the event they wish to watch and to watch the event at home. In other words, in this case, user 30's home constitutes the viewing space for watching the event. Furthermore, as described above, the event is assumed to be a collective event with multiple venues and multiple performers.

[0057] In step S10, the control unit 100 in the terminal device 10 acquires environmental information for the user 30 using the environmental information acquisition unit 111. The environmental information acquisition unit 111, for example, uses the outward-facing camera 1101 provided in the terminal device 10 to photograph the area around the user 30 and acquires captured image data. The control unit 100, for example, models the three-dimensional shape of the surroundings based on the captured image data and generates a map of the viewing space.

[0058] In the next step S11, the control unit 100 uses the content information acquisition unit 114 to acquire information about the event that user 30 will view. For example, the content information acquisition unit 114 communicates with the reference DB 72, for example via the network 2, and acquires information about the event specified by user 30 from the reference DB 72. The event information acquired by the control unit 100 may include, for example, an event map based on map information showing the locations of multiple venues prepared for the event, and a timetable showing the progress of each venue in the event.

[0059] In the next step S12, the control unit 100 acquires the user 30's planned actions for the event, for example, using the content information acquisition unit 114. For example, the content information acquisition unit 114 uses the output control unit 113 to display the event timetable on the display unit and prompts the user 30 to input their schedule. The content information acquisition unit 114 can acquire the user 30's planned actions in response to the user 30's input operation.

[0060] In the next step S13, the control unit 100 uses the map generation unit 112 to assign events to the user 30's environment. That is, the map generation unit 112 assigns the presentation position of each piece of content presented at each venue in the event to the user 30's viewing space. At this time, the map generation unit 112 can set up multiple viewing areas in the user 30's viewing space and assign each presentation position of each piece of content to each of the multiple viewing areas. The user 30 is able to move between the multiple viewing areas set up in the user 30's viewing space. The map generation unit 112 presents information indicating each presentation position of each piece of content assigned to the viewing space to the user 30 using the output unit 103 (for example, display units 1202L and 1202R).

[0061] In the next step S14, the control unit 100, using the map generation unit 112, reflects any modifications made by the user 30 to each display position of the content assigned to the viewing space in step S13.

[0062] By applying the information processing system 1 according to this embodiment, for example, the viewing space set up in the user 30's home is assigned each presentation position of content presented at multiple venues in a group event. Therefore, when the user 30 remotely watches a group event, for example, they can experience movement between multiple venues in the physical space within the viewing space set up in their home, thereby achieving a deeper sense of immersion.

[0063] (3-2. Details of the process according to the embodiment) Next, the process according to the embodiment will be described in more detail.

[0064] (3-2-1. Environmental Information Acquisition Process) First, we will explain in more detail the acquisition process for acquiring environmental information about user 30 by the information processing system 1 according to the embodiment, as described in step S10 of Figure 5. Figure 6 is a flowchart showing an example of the acquisition process for acquiring environmental information about user 30 according to the embodiment.

[0065] In step S100, the control unit 100 acquires environmental information about the user 30 using the environmental information acquisition unit 111. More specifically, based on the outputs of the sensor unit 101 and the positioning unit 102, the environmental information acquisition unit 111 acquires environmental image information, depth information, and location information of the terminal device 10 related to the user 30 as environmental information. For example, the environmental information acquisition unit 111 acquires image information as RGB information using R (red), G (green), and B (blue) color information from the outward-facing camera 1101 included in the sensor unit 101. The environmental information acquisition unit 111 also acquires depth information from the outward-facing camera 1101.

[0066] Figure 7 is a schematic diagram showing an example of image information and depth information acquired by the outward-facing camera 1101. In Figure 7, section (a) shows an example of image information. Section (b) shows an example of depth information. In section (b), depth information is acquired at the position of each dot.

[0067] Furthermore, when acquiring image information and depth information, the environmental information acquisition unit 111 may prompt the user 30 to acquire information for an appropriate area so as to cover the area (viewing space) that the user 30 is expected to use for this viewing experience, using a user interface or the like. The user interface includes at least one of a screen that presents information to the user and an input unit for the user to input information. Figure 8 is a schematic diagram showing an example of a user interface screen that prompts the user 30 to acquire image information and depth information for an appropriate area. In Figure 8, the user interface screen 300 is presented to the user 30 by the output unit 103 (for example, display units 1202L and 1202R), including a message 301 prompting the user to acquire image information and depth information for an appropriate area.

[0068] Returning to the flowchart in Figure 6, in the next step S101, the control unit 100 integrates the environmental information acquired in step S100 by the environmental information acquisition unit 111 and performs a 3D model of the environment related to user 30. The 3D modeling method will use matching of feature quantities extracted based on image information and depth information.

[0069] Furthermore, if the environmental information acquisition unit 111 cannot obtain the required accuracy in 3D modeling, it may generate a floor plan-like model using 2D information. Also, if the environmental information acquisition unit 111 can acquire existing data (2D floor plan, 3D scan data, etc.), it may not perform modeling processing on these existing data and may instead refer to them in subsequent processes.

[0070] Figure 9 is a schematic diagram showing an example of 3D model data of the environment for user 30, generated by data integration by the environmental information acquisition unit 111 in step S101. In Figure 9, the 3D model 40 made from the 3D model data corresponds to the viewing space where user 30 views the event. In the example in Figure 9, the 3D model 40 is assumed to be a 3D model of user 30's room.

[0071] In the state where the 3D model 40 has been generated, there is a lack of information such as which area within the 3D model 40 the user 30 will view the content in. Therefore, in the next step S102, the control unit 100 uses the map generation unit 112 to assign meaning to the 3D model 40 and generate a map. At this time, the map generation unit 112 may retain data on the types of furniture and other items that have been placed in advance, such as through object recognition. The map generation unit 112 may also refer to the user 30's past purchase history and use it as reference data to identify the types of items.

[0072] In the example shown in Figure 9, the 3D model 40 recognizes a living room 400, a bed 401 and furniture 403 within the living room 400, and an area 402 with a different floor condition from the living room 400. The 3D model 40 also recognizes a kitchen unit 404 and a bathroom 405. To the left of the kitchen unit 404, an area 407 with a different floor condition from the living room 400 is recognized, and furniture 406 is recognized adjacent to area 407.

[0073] In subsequent processing, when associating content with viewing space, it is important to know which area within the viewing space (which is the physical space) user 30 is most likely to view the content in. Therefore, for example, an area where user 30 is most likely to view the content may be explicitly set, and this point may be used as a reference point for placement.

[0074] For example, the map generation unit 112 may use a user interface screen to prompt the user 30 to set viewing areas from among the parts recognized in the 3D model 40 that are most likely to be viewed. The map generation unit 112 stores the reference coordinates of each set viewing area as the reference point for each viewing area.

[0075] At this time, the map generation unit 112 sets up multiple viewing areas for the viewing space based on at least one of the following: division information, role information, structural information, floor surface information, shape information, and equipment information, and incorporates the set up multiple viewing areas into the 3D model 40. Figures 10A to 10D are schematic diagrams illustrating an example of a method for setting up multiple viewing areas for a viewing space that can be applied to the embodiment.

[0076] Figure 10A shows an example of setting multiple viewing areas for a viewing space based on division information derived from structural information indicating the structure of the viewing space. In Figure 10A, the viewing space 310a is physically divided by structures 320, such as doors and thresholds. In this case, the map generation unit 112 can set separate viewing areas, Area #1 and Area #2, in the separated spaces, as shown in space 310b.

[0077] Figure 10B shows an example of setting multiple viewing areas for a viewing space based on division information derived from the surface information of the floor of the viewing space. In Figure 10B, the viewing space 311a has different floor surface conditions in area 321 and other parts, and the floor surface information differs in these areas. For example, if the parts other than area 321 have hardwood flooring and area 321 has carpet, or if the floor of area 321 is linoleum, the floor surface information will differ between area 321 and other parts. In this case, the map generation unit 112 can set area #1 and area #2, which are separate viewing areas, in area 321 and other parts, as shown in space 311b.

[0078] Figure 10C shows an example of setting multiple viewing areas for a viewing space based on division information derived from shape information indicating the shape of the viewing space. In Figure 10C, as shown in space 312a, even in environments where the surface condition of the floor is the same and there are no divisions by structures, it is possible that space 312a may be divided by walls 322a and 322b. In this case, as shown in space 312b, the map generation unit 112 can set area #1 and area #2, which are separate viewing areas, by extending walls 322a and 322b.

[0079] Figure 10D shows an example of setting up multiple viewing areas in a viewing space based on segmentation information derived from equipment information of the equipment installed in the viewing space. In Figure 10D, as shown in space 313a, it is possible that equipment 323 is placed that clearly indicates different activities in its vicinity, such as a kitchen unit. Equipment 323 includes furniture and electrical appliances that are installed and used in a fixed or movable manner in the space. In this case, as shown in space 313b, the map generation unit 112 can set up separate viewing areas, Area #1 and Area #2, around the equipment 323 as a separate space. For example, the map generation unit 112 may identify a space as a kitchen if a stove or kitchen unit is placed as equipment 323, and identify it as a living room if a sofa is placed there. Furthermore, margin areas may be set in advance according to the furniture, and the area including these margin areas may be designated as a viewing area.

[0080] The reason for setting up multiple viewing areas within a viewing space, as explained using Figures 10A to 10D, is that when arranging content within a viewing space, it is sometimes necessary to avoid placing content across elements that separate these viewing spaces. For example, even within the same viewing space, if user 30 is using it for different roles, this example should be considered. Furthermore, even in a single-room apartment, the area around the kitchen is expected to be separated as a separate space. These are stored as area information.

[0081] Returning to the flowchart in Figure 6, in step S103, the control unit 100 completes the environmental assessment by saving the area information, including the 3D model data and reference point data acquired in steps S101 and S102, as environmental data to, for example, the storage device 1505. In addition to the 3D model data, the area information may also include, for example, the reference point coordinates of each area, the area, information on the furniture to be placed, whether seating is permitted, the presence or absence of hazardous materials in the surrounding area (such as the presence or absence of heating devices), and sound insulation due to the floor material. The control unit 100 maintains each of these area information items, for example, as a list.

[0082] Figure 11 is a schematic diagram showing an example of 3D model data and area information stored by the control unit 100 according to an embodiment. Section (a) of Figure 11 shows an example in which reference positions A to D are set for each area of ​​the 3D model 40. Reference positions A to D may be, for example, the coordinates of any vertex in each area or the coordinate positions obtained by averaging the vertex coordinates.

[0083] Section (b) of Figure 11 shows an example of area information (second spatial information). For each reference position A to D, the items "reference point coordinates," "tags," and "area information" are associated. The item "reference point coordinates" shows the coordinates of reference positions A to D. The item tags are associated with the viewing area information. In the example in Figure 11, in the area information, the item "tags" indicates the meaning assigned to each area, with the area assigned to reference position A being "living room," the area assigned to reference position B being "bed," the area assigned to reference position C being "kitchen," and the area assigned to reference position D being "bathroom." Based on these meanings, it can be determined that the areas assigned to reference positions A and B are areas for relaxation, the area assigned to reference position C is an area for cooking, etc., and the area assigned to reference position D is an area for showering, bathing, etc. In other words, the information shown in the item "tags" can be said to be role information that indicates the role of the corresponding area.

[0084] In the area information, the item "Area Information" indicates the information to be set as the viewing area among the areas indicated by each reference position A to D. In the example in Figure 11, the two areas indicated by reference positions A and B are spatially continuous and have a common role, so they are integrated and set as the viewing area indicated by area information [1]. In addition, the area indicated by reference position C is set as the viewing area indicated by area information [2], and the area indicated by reference position D is set as the viewing area indicated by area information [3]. The user 30 is allowed to move between the viewing areas indicated by area information [1] to [3].

[0085] (3-2-2. Assigning the event to the viewing environment) Next, we will explain the process of acquiring event information and assigning it to the viewing environment by the information processing system 1 according to the embodiment, as described in steps S11 to S13 of Figure 5.

[0086] In an event, the content that users 30 will view includes content whose presentation time (time slot) and location are predetermined, and content whose presentation time and location are not predetermined. The following explanation will focus on content whose presentation time and location are predetermined in the event.

[0087] Furthermore, the content that this embodiment is particularly intended for is not content presented by a single performer, but rather content presented in so-called collective events where multiple performers appear in parallel at multiple venues. However, this embodiment is not limited to this; even if there is only one performer, it also envisions exhibition-style events where multiple contents are distributed in parallel. Therefore, the following explanation will assume that multiple contents are being distributed.

[0088] (Regarding weighting multiple pieces of content) Multiple content elements can be weighted. For example, a weighting system for a broadcaster could assign weights to each venue in a group event. That is, the broadcaster can give a higher weight to the importance of the main venue in a group event. Similarly, the broadcaster can also weight elements in an exhibition format.

[0089] On the other hand, the weighting on the user 30's side can be based on the user 30's preferences. For example, user 30 can weight content presented by performers they are already fans of, according to their own viewing priorities. Alternatively, user 30 can also weight the genre of content presented by performers. For example, the control unit 100 presents a user interface showing each of the items that user 30 weights. The control unit 100 stores the weighting information entered into this user interface.

[0090] (Regarding the spatial arrangement of each venue in the physical space) In group events and exhibitions, the organizer often sets the expected spatial relationships between content items. The spatial relationships between each content item are determined based on the event map provided by the organizer, which shows these relationships. For the purposes of this explanation, it is assumed that the event organizer and the content provider are the same person.

[0091] Figure 12 is a schematic diagram showing an example of a map (event map) that illustrates the spatial relationships between multiple pieces of content in a collective event.

[0092] In Figure 12, the event map 50 (first spatial information) shows Stage 500, the main stage for content presentation, and Stages 501 and 502, which are secondary stages for content presentation, respectively. In Figure 12, Stages 501 and 502 are also shown as Sub-Stage #1 and Sub-Stage #2, respectively. Stage 500 is located within Stadium 510, while Stages 501 and 502 are located on the beach 511 and hill 512, respectively. In Figure 12, with north (N) at the top, Stages 501 and 502 are located northwest and southwest of Stage 500, respectively.

[0093] Such event maps 50 are often known to users 30 in advance. Therefore, when arranging each piece of content in the viewing space, it is necessary to refer to the arrangement on the event map 50 for each stage 500-502.

[0094] Furthermore, in addition to physical placement, spatial placement may be set up specifically for distribution. For example, placement areas may be changed depending on the content genre, label, and the amount of interactive participation.

[0095] (Regarding preset maps provided by the distribution provider) Next, a preset map presented by the distribution side according to this embodiment will be described. The distribution side, for example, using server 20, generates an initial layout for arranging the display positions of each content in the user 30's viewing space, based on the positional relationship of each venue in the event and the time slots (timetable) in which each performer presents content at each venue. The distribution side, for example, using server 20, presents the generated initial layout to user 30 as a preset map.

[0096] Figure 13 is a schematic diagram showing an example of an event timetable, which is created on the distribution side and retrieved, for example, by server 20. In Figure 13, timetable 520 shows the performance times of performers A to F at each stage 500, 501, and 502 (described as the main stage, sub-stage #1, and sub-stage #2, respectively) in the event map 50 shown in Figure 12. In the example in Figure 13, timetable 520 shows that performer A will perform at stage 500 from time [10:00] to [11:00], and performer B will perform at time [11:30] to [12:30]. Similarly, the performance times of performers C to F are shown for stages 501 and 502.

[0097] As shown in Figure 12, the broadcaster pre-defines the positional relationships between each stage 500-502 for spatial arrangement. For example, when the broadcaster initially places the content to be presented in each stage 500-502 in the user 30's viewing space, they refer to the positional relationships of these stages. The broadcaster, for example, prepares a two-dimensional grid using server 20, and creates a preset map by placing the areas corresponding to each stage 500-502 on this grid.

[0098] Figure 14 is a diagram illustrating the arrangement of regions corresponding to each stage 500-502 on a grid, applicable to the embodiment. In Figure 14, section (a) shows the event map 50, and section (b) shows an example of a preset map 530. In the figure, the preset map 530 consists of a grid on a two-dimensional plane. The preset map 530 is assumed to have the upward direction in the figure corresponding to the north direction.

[0099] The region 540 shown in section (a) of Figure 14 represents the region to be placed relative to the preset map 530. In this example, regions 500lay, 501lay, and 502lay are automatically placed relative to the preset map 530 according to the grid, corresponding to the positional and size relationships of each stage 500, 501, and 502 in the event map 50.

[0100] That is, for example, server 20 places area 500lay, which corresponds to stage 500 located at the easternmost (rightmost) end of area 540 of event map 50, at the rightmost (easternmost) end of preset map 530. For example, server 20 places area 501lay, which corresponds to stage 501 located northwest of stage 500 in event map 50, at the upper left (northwest of area 500lay) of preset map 530. Similarly, for example, server 20 places area 502lay, which corresponds to stage 502 located southwest of stage 500 in event map 50, at the lower left (southwest of area 500lay) of preset map 530.

[0101] In addition, in the example shown in the figure, each region 500lay to 502lay is placed on the preset map 530 with a size corresponding to the ratio of the sizes of the corresponding stages 500 to 502.

[0102] Furthermore, the broadcaster can manually set the margins between stages for each area 500lay to 502lay that is automatically placed on the preset map 530, as shown in section (b) of Figure 14.

[0103] Furthermore, according to the timetable 520 shown in Figure 13, for example, the time period from [10:00] to [10:30] is when Stage 501 (Substage #1) is not in use. In such cases, when there are available time periods for each Stage 500 to 502, the server 20 may dynamically change the arrangement of each area 500lay to 502lay in the preset map 530, for example, by expanding area 500lay, which corresponds to the main stage Stage 500, in the preset map 530.

[0104] Furthermore, for example, server 20 may perform automatic placement of the preset map 530 without referring to the event map 50, based on the attributes of the content presented at each venue. For example, server 20 may use genre and performer fan information as content attributes, and for content with overlapping genres and fan bases, it can automatically place the areas adjacent to each other on the preset map 530. Also, for example, server 20 may perform automatic placement while considering factors such as ensuring margins between areas for content of different genres.

[0105] Furthermore, for example, server 20 may set in the preset map 530 transition areas that provide effects to content when content transitions to other content at the boundary of adjacent or nearby areas (for example, area 500lay and area 501lay). Also, for example, server 20 may place areas in the preset map 530 that do not contain areas 500lay to 502lay where content is presented, to present other content that is not actually presented in stages 500 to 502.

[0106] Figure 15 is a schematic diagram showing another example of a preset map 530 applicable to the embodiment. In Figure 15, the regions 540lay, 541lay, and 542lay placed in the preset map 530 are regions corresponding to the venues where content #1, #2, and #3 are presented, respectively, in the event. Region 543 is a region that includes the boundary between the adjacent regions 540lay and 541lay, and is a transition region for presenting a predetermined effect when content transitions to other content. The predetermined effect in the transition region includes at least one of a visual effect and an auditory effect. A specific example of such an effect is a crossfade process between the content before the transition and the content after the transition. However, it is not limited to this, and a moving video showing movement through the real space regions corresponding to regions 540lay and 541lay may be created in advance and presented in region 543 in the viewing space viewed by user 30.

[0107] In Figure 15, area 544 is an area where content different from content #1, #2, and #3 presented at each venue in the event is located, and is, for example, an area for selling merchandise such as items. Within the viewing space where user 30 is viewing, it is possible to enable the purchase of items via electronic payment within this area 544.

[0108] Server 20 saves the generated preset map 530 to, for example, the reference DB 72. However, Server 20 may also save the generated preset map 530 to its own storage medium.

[0109] Although the above description explains that the server 20 generates the preset map 530, this is not limited to this example. For example, the terminal device 10 can also generate the preset map 530. In this case, the terminal device 10 obtains the event map 50 and timetable 520 from the server 20 and generates the preset map 530 based on the obtained event map 50 and timetable 520. The terminal device 10 saves the generated preset map 530 to, for example, a storage device 1505.

[0110] (Content allocation to viewing space) Next, we will explain the correspondence between the preset map 530 and the viewing space where user 30 views the content.

[0111] Figure 16 is a flowchart illustrating an example of the content allocation process for the viewing space according to this embodiment. Each process in the flowchart of Figure 16 is executed by the control unit 100 in the terminal device 10.

[0112] In step S200, the control unit 200 reads the environmental data acquired and saved by the process shown in the flowchart in Figure 6.

[0113] In the next step S201, the control unit 100 acquires video data from the content DB 71 using the content information acquisition unit 114, and also acquires information related to the video data from the reference DB 72 via communication with the communication unit 115. The information related to the video data acquired from the reference DB 72 includes a preset map 530 and a timetable 520. Furthermore, if an event has started, for example, the control unit 100 acquires video data of the event to be distributed from the streaming server 70 via communication with the communication unit 115.

[0114] In the next step, S202, the control unit 100 separates the video data, the index for the video data, and the additional information attached to the video data from the video data acquired from the streaming server 70 and the content DB 71 in step S201. The index and additional information may include, for example, information indicating the venue (stage) where the corresponding video data was filmed.

[0115] In the next step, S203, the control unit 100 uses the map generation unit 112 to perform a matching evaluation between each venue (stage) where content is presented at the event and the environment related to user 30, i.e., user 30's viewing space, based on environmental data. In the next step, S204, the control unit 100 uses the map generation unit 112 to assign each piece of content presented at the event to the environment (viewing space) related to user 30, based on the evaluation results of the matching evaluation in step S203.

[0116] (Content allocation to each viewing area) The processes described in steps S203 and S204 above will now be explained in more detail. The matching evaluation between each venue where the content is presented and the user's viewing space can utilize the positional relationships between the content and the positional relationships between each viewing area within the viewing space. Furthermore, the degree of user engagement with the content, the size of the venue where the content is presented, and the video distribution format are also considered in the matching evaluation.

[0117] This section explains the degree of user participation. The degree of user participation varies depending on the content. Even in remote events conducted via streaming, it is expected that user participation, such as interjections and cheers, will be reflected in the content. For example, content in which user interjections, cheers, and singing are commonplace in content presentations in physical spaces will be defined as high-participation content. For high-participation content, it is expected that there will be requests for users to perform these actions in remote events, and that these will be incorporated into the content as part of the performance.

[0118] It is preferable to select a viewing area that allows participants to better enjoy highly interactive content. For example, in the case of content where users speak loudly, users may hesitate to participate in a room with a typical soundproof environment due to concerns about the impact on the surrounding environment. Therefore, when presenting highly interactive content, consideration should be given to allocating a highly soundproofed area to the viewing area.

[0119] When assigning content presentation locations to viewing areas, it is preferable to take into account the size of the venue where the content is presented. For example, content presented in the largest of several venues could be assigned to the largest viewing area among several viewing areas in the user's viewing space.

[0120] Furthermore, when assigning content presentation locations to viewing areas, it is preferable to consider the distribution method used to deliver the content's video. In other words, the user's viewing experience differs depending on whether the display device used to display the content in the viewing area is a fixed-angle display (including switching) based on an FPD, or a 360° video including a free viewpoint, such as eyewear. Therefore, it is conceivable to determine the content to be assigned to a viewing area based on whether the content distribution method in that area is, for example, a fixed-angle display or 360° video.

[0121] (Specific examples of content allocation to viewing space) Next, we will explain in more detail how content is allocated to the viewing space. For example, consider a collective event such as a music festival featuring performers W, X, Y, and Z. The room shown as the 3D model 40 in Figure 11 will be the viewing space, and the content presented in this collective event will be spatially arranged within it. Furthermore, each piece of content will be a 360° video in which the viewpoint and sound image switch according to the position and orientation of the user 30.

[0122] In this case, the room designated as the viewing space has four reference positions A to D, as shown in Figure 11. Considering the continuity of the space, the viewing area where content can be placed in this room is set to three viewing areas, as shown in section (b) of Figure 11, indicated by area information [1], [2], and [3]. Therefore, the upper limit for the number of content items that can be placed in the viewing space simultaneously is set to [3]. On the other hand, there are four content items that should be placed for the event in question, so the priority of placement is determined.

[0123] Figure 17 is a schematic diagram illustrating an example of information used to determine the placement of each piece of content, applicable to the embodiment. In the example in Figure 17, this information includes the items "Performer," "Genre," "User Preferences," "User Participation Level," and "Venue Size." The item "Placement Area" on the far right of Figure 17 indicates the viewing area where each piece of content will be placed, determined based on the aforementioned items. Figure 17 also shows information regarding content that will be presented in parallel during specific time periods at an event.

[0124] The "Performer" item indicates the performers who present content in parallel during that time slot. In the example in Figure 17, performers W, X, Y, and Z are shown. The "Genre" item is genre information indicating the type (genre) of content presented by the performers listed in the "Performer" item. In the example in Figure 17, the genre information indicating the genre of content presented by each performer W, X, Y, and Z is [Rock], [Rock], [Idol], and [Jazz], respectively.

[0125] In Figure 17, the "User Preference" item represents preference information indicating the user's preference for the performer indicated in the "Performer" item, or for the content presented by that performer. This preference information for the "User Preference" item is entered by the user 30 who views each piece of content, and indicates the user 30's degree of preference for the corresponding performer, i.e., the degree of their liking. In the example in Figure 17, the "User Preference" item shows that the preference levels for each performer W, X, Y, and Z are [High], [Low], [Medium], and [Medium], respectively. A value of [High] for the "User Preference" item indicates that the user 30 has a high degree of preference for the corresponding performer or the content presented by that performer; a value of [Low] indicates a low degree of preference; and a value of [Medium] indicates a moderate degree of preference.

[0126] In Figure 17, the information in the item "User Participation" represents the level of participation of all users who view the content presented by the performer indicated in the item "Performer." In this case, the user participation level can be obtained, for example, from the reference DB72. However, the item "User Participation" may also represent the level of participation entered by the user 30 who views the content in their own viewing space.

[0127] The information in the "Venue Size" field indicates the size of the physical venue where the content is presented by the performers specified in the "Performers" field.

[0128] The map generation unit 112 assigns weights to the content to be placed in the viewing space based on the values ​​of each item in Figure 17. This weighting will be explained in detail.

[0129] The map generation unit 112 first prioritizes the content to be placed in the viewing space according to the value of the item "User Preference" in Figure 17. In the example in Figure 17, the value of the item "User Preference" for performer X is [Low], so the map generation unit 112 excludes performer X from the content to be placed.

[0130] Next, the placement of performers W, Y, and Z is determined. As can be seen from Figure 11, the viewing area indicated by area information [1] is the largest in this space as well. Therefore, the map generation unit 112 places performer W, whose "venue size" item is [large], as indicated in the "placement area" item.

[0131] Next, for performers Y and Z, the "User Preference" item is [Medium] for both, indicating that their preferences are equivalent to those of user 30. However, the "User Participation" item is [High] for performer Y and [Low] for performer Z, indicating that performer Y is more engaged. In this case, the viewing area indicated by area information [2] is the "kitchen," and the viewing area indicated by area information [3] is the "bathroom." It is presumed that the viewing area indicated by area information [3] is more airtight than the viewing area indicated by area information [2]. Therefore, it is assumed that the viewing area indicated by area information [3] is more suitable for situations such as when user 30 is cheering for the presented content. Accordingly, the map generation unit 112 places performer Y in the viewing area indicated by area information [3] and performer Z in the viewing area indicated by area information [2], as indicated by the "Placement Area" item.

[0132] By moving through viewing spaces where content is placed in each viewing area, user 30 can experience the event in a way that is tailored to their surrounding environment.

[0133] Figure 18 is a schematic diagram showing an example of a user interface for placing content in a viewing area, applicable to the embodiment.

[0134] As mentioned above, the user interface includes at least one of a screen that presents information to the user and an input unit for the user to input information. In the examples in sections (a) to (d) of Figure 18, the user interface is configured to include these screens and input units. When an eyewear device is used as the terminal device 10, the input unit may include, for example, an outward-facing camera 1101 that photographs the user's hand 30 in order to perform hand tracking.

[0135] As explained using Figure 6, spatial information of the environment (viewing space) in which user 30 will view each piece of event content has already been acquired. User 30, for example, purchases a ticket for the content that is available for viewing and performs tasks such as signing in.

[0136] In the terminal device 10, the control unit 100 causes the output control unit 113 to display the user interface screen 600 shown in section (a) of Figure 18 on the display unit (for example, at least one of the display units 1201L and 1201R). The user interface screen 600 displays a three-dimensional model 610 of the user 30's viewing space and a message 611 prompting the user 30 to confirm the viewing area and to add or delete the viewing area.

[0137] User 30 can check the environmental information of their viewing space based on the user interface screen 600. The control unit 100 presents the maximum number of usable viewing areas based on the environmental information (spatial information) of the viewing space, and prompts User 30 to confirm the availability of each viewing area, for example, by message 611. In the example shown in the figure, the 3D model 610 shows three viewing areas based on reference positions A to C. This is because the conditions of the viewing space may differ between the time spatial information is acquired and the time it is used, and it is necessary to determine the availability of each viewing area according to the situation.

[0138] Furthermore, the control unit 100 may also enable adjustment of the viewing area at this stage. If the presented viewing area has a small layout or includes a hazardous area, the user 30 can perform adjustments such as expanding or contracting the viewing area or setting exclusion areas based on the user interface screen 600.

[0139] Next, the control unit 100 presents a user interface screen 601, shown in section (b) of Figure 18, where the user 30 inputs information indicating their preferences for each piece of content. In the example shown, the user interface screen 601 displays the event timetable 521. In this example, performer W is assigned to the main stage (main) for the time period [10:00] to [11:30], and performers X, Y, and Z are assigned to sub-stages #1 to #3 (sub-stages #1 to #3), respectively. Performer A is assigned to the main stage for the time period [11:30] to [12:30], and performers C and D are assigned to sub-stages #2 and #3, respectively. Performer B is assigned for the time period [12:00] to [12:30].

[0140] User 30 inputs preference information indicating their own preferences to the timetable 521 on the user interface screen 601, as shown in section (c) of Figure 18. In the example shown in the figure, as an operation to input preference information, User 30 performs an operation on the terminal device 10 to mark 620 the performers in the timetable 521 that are of high preference to them, i.e., performers whose content they plan to watch.

[0141] In the example in section (c) of Figure 18, presenters W, Y, and Z are marked with mark 620 during the time period [10:00] to [11:30], as shown in Figure 17, indicating that viewers are scheduled to view the content presented by presenters W, Y, and Z during that time period. In addition, in the example in the figure, presenter C is marked with mark 620 during the time period [11:30] to [12:30], and presenter B is marked with mark 620 during the time period [12:30] to [12:30], indicating that viewers are scheduled to view the content presented by presenters C and D during those time periods.

[0142] Once the settings in section (c) of Figure 18 are finalized, the control unit 100 displays a user interface screen 603 on the display unit for confirming each set viewing area, as shown in section (d) of Figure 18. In the example shown, the performers W, Y, and Z are automatically positioned at each reference position A to C, which represent each viewing area, in the three-dimensional model 610 representing the user 30's viewing space. In the example shown, a message 612 is displayed on the user interface screen 603 prompting the user 30 to confirm whether this automatically positioned content is acceptable.

[0143] Note that in the example in section (c) of Figure 18, a mark 620 indicating high preference is directly attached to the timetable 521, but this is not limited to this example. For example, user 30 may input their preferred genre and set the content to be viewed based on recommendations from the system. In this case, for example, the control unit 100 performs the weighting process described above on the genre and reflects the automatically arranged results according to the weighting results on the user interface screen 603 shown in section (d) of Figure 18. User 30 confirms the arrangement based on this user interface screen 603.

[0144] Furthermore, for example, the placement of content in each viewing area is not limited to automatic placement by the control unit 100, but may also be done manually by the user 30. For example, the user 30 may manually change the placement of content in response to operations on the user's screen.

[0145] Furthermore, for example, the control unit 100 may use the preset map 530 described above as is to automatically arrange the content for each viewing area. Alternatively, the user 30 can explicitly change the arrangement by editing the arrangement based on the preset map 530, such as by rotating or flipping it, in response to user 30's actions.

[0146] Figure 19 is a schematic diagram showing an example of the arrangement of each content for each viewing area according to the embodiment. Section (a) of Figure 19 is the same diagram as section (a) of Figure 14 described above, and shows an example of a preset map 530 in which areas 500lay to 502lay, corresponding to stages 500 to 502, are arranged in accordance with the positional relationship of stages 500 to 502.

[0147] Section (b) of Figure 19 shows an example where the arrangement of each region 500lay to 502lay in the preset map 530 is applied to the 3D model 40 representing the viewing space, maintaining the same positional relationship. In this example, each region 500lay to 502lay is displayed superimposed on the image of the 3D model 40 as a viewing area.

[0148] In the example in section (b) of Figure 19, the area 500lay, which corresponds to the main stage, stage 500, is positioned according to the reference position A.

[0149] With respect to area 500lay, area 501lay, which corresponds to stage 501 located northwest of stage 500 as substage #1, is positioned in the upper left of area 500lay, corresponding to the relative positional relationship between stage 500 and stage 501. Similarly, with respect to area 500lay, area 502lay, which corresponds to stage 502 located southwest of stage 500 as substage #2, is positioned in the lower left of area 500lay, corresponding to the relative positional relationship between stage 500 and stage 502.

[0150] Section (c) of Figure 19 shows an example in which the arrangement of each region 500lay to 502lay as shown in Section (b) of Figure 19 is changed according to the operation of the user 30. The control unit 100 displays, for example, an image as shown in Section (b) of Figure 19 on the display unit screen. The user 30 can specify the region they want to move from among regions 500lay to 502lay on that screen and move it.

[0151] In the example in section (c) of Figure 19, the position of region 502lay is moved to the right to become region 502lay', which serves as the viewing area, and region 500lay is vertically reduced in the figure so as not to overlap with region 502lay to become region 500lay', which also serves as the viewing area. Such changes in shape due to the movement of regions are controlled by the control unit 100.

[0152] [4. Viewing process according to the embodiment] Next, the content viewing process according to the embodiment will be described. Figure 20 is a flowchart illustrating an example of the content viewing process according to the embodiment.

[0153] In Figure 20, in step S300, the terminal device 10 is activated, for example, in response to an operation by user 30, and the information processing program according to the embodiment is executed, and the control unit 100 is configured. In the terminal device 10, the control unit 100 requests event information from, for example, server 20. In response to this request, server 20 transmits event information to the terminal device 10. In the terminal device 10, the control unit 100 displays a screen on the display unit for specifying the event transmitted from server 20.

[0154] In the next step S301, the control unit 100 specifies the target event in response to the user 30's operation. In response to this event specification operation, the control unit 100 requests event information from the server 20, for example, including performer information, event map 50, timetable 520, etc. In response to this request, the control unit 100 obtains the event information sent from the server 20.

[0155] In the next step, S302, the control unit 100 determines whether or not a correspondence exists between the user 30's viewing environment and each piece of content presented within the event. If the control unit 100 determines that no such correspondence exists (step S302, "No"), it proceeds to step S303. In step S303, the control unit 100 executes the process shown in the flowchart of Figure 6 described above, assigning each piece of content within the event to each viewing area in the user 30's viewing space. After that, the process proceeds to step S304.

[0156] On the other hand, if the control unit 100 determines in step S302 that an association has been made with each piece of content presented within the event (step S302, "Yes"), it skips the processing in step S303 and proceeds to step S304.

[0157] In step S304, the control unit 100 determines whether multiple content items are associated with the user 30's viewing space. If the control unit 100 determines that only one content item is associated with the user 30's viewing space (step S304, "No"), it proceeds to step S310. In step S310, the control unit 100 presents the content item in the viewing space and completes the series of processes.

[0158] On the other hand, if the control unit 100 determines in step S304 that multiple content items are associated with the user 30's viewing space (step S304, "Yes"), it proceeds to step S305. In this case, each of the multiple content items is positioned to display in one of the multiple viewing areas within the viewing space.

[0159] In the next step, S305, the control unit 100 selects content from multiple content items according to the user 30's location. Specifically, the control unit 100 obtains the user 30's current location using the positioning unit 102, and based on the obtained current location, detects which viewing area within the viewing space the user 30 is in. The control unit 100 then selects content from multiple content items according to the viewing area corresponding to the user 30's location.

[0160] In the next step, S306, the control unit 100 outputs the content selected in step S305. For example, the control unit 100 extracts the content selected in step S305 from each of the contents presented in each stage 500 to 502 transmitted from the streaming server 70. The control unit 100 outputs the extracted content in the viewing area where the user 30 is currently located.

[0161] In the next step, S307, the control unit 100 determines whether or not user 30 is attempting to move to another viewing area. The control unit 100 can detect whether or not user 30 is moving and in the direction of movement, for example, based on the positioning output from the positioning unit 102, and the outputs of the gyro sensor 1104, acceleration sensor 1105, and compass sensor 1106. Based on user 30's position and direction of movement, the control unit 100 can infer whether or not user 30 is attempting to move to another viewing area.

[0162] If the control unit 100 determines that user 30 is about to move to another viewing area (step S307, "Yes"), it proceeds to step S308. In step S308, the control unit 100 displays the transition effect when moving across viewing areas on the display unit. Specific examples of the transition effect will be described later. After displaying the transition effect in step S308, the control unit 100 returns to step S305.

[0163] On the other hand, if the control unit 100 determines in step S307 that user 30 is not attempting to move to another viewing area (step S307, "No"), it proceeds to step S320. In step S320, the control unit 100 determines whether user 30 has finished viewing all the content of the event. If the control unit 100 determines that viewing has ended, for example, in response to a predetermined operation on the terminal device 10 (step S320, "Yes"), it terminates the series of processes shown in the flowchart of Figure 20.

[0164] On the other hand, if the control unit 100 determines that user 30 has not finished viewing the content (step S320, "No"), it returns to step S306.

[0165] (Regarding the display of transition effects) Next, we will explain the display of the transition effect in step S308 described above. As an example, consider the case where user 30 moves from viewing area A, where content A is being viewed, to viewing area B, where content B is being viewed. In this case, the moment user 30 enters viewing area B from viewing area A, the display on the display unit will switch from content A to content B. Such an abrupt switch in displayed content may impair immersion. Therefore, it is preferable to provide a transition area at the boundary between viewing area A and viewing area B, for example, shown as area 543 in Figure 15, and to apply a predetermined video processing that indicates the transition from content A to content B as a transition effect in this transition area.

[0166] As an example of a transition effect, the control unit 100 pre-loads the content to be output in the destination viewing area B, fades out content A and fades in content B in the transition area, and processes content A and content B. In this case, the control unit 100 may set the amount of fade-in and fade-out according to the direction and speed of movement of the user 30.

[0167] As another example of a transition effect, the control unit 100 may output pre-prepared content for movement in the transition area. The content for movement may be, for example, video reconstructed using a 3D model, which is output in the transition area. For example, the control unit 100 may reconstruct the scenery along the movement path between venues in real space as a 3D model. Since the amount of movement from viewing area A to viewing area B is less than the amount of movement in the actual venue, in these cases, the playback speed of the video output in the transition area may be set to high speed.

[0168] Here, the control unit 100 may adjust the amount of movement in the transition effect according to the size of the area where the user 30 is placed when the user moves within the viewing area. For example, even if the user's actual movement is the same, the relative amount of movement within the viewing area will differ depending on whether the set viewing area is large or small. Therefore, it is preferable for the control unit 100 to adjust the amount of movement within the content, taking into account the size of the set viewing area and the size of the venue in the physical space where the content assigned to the viewing area is presented.

[0169] Furthermore, if it is difficult to spatially separate viewing area A and viewing area B, the control unit 100 may, for example, prompt the user 30 to change the lighting environment to promote a sense of separation between viewing area A and viewing area B. For example, the brightness of the lighting could be made different for viewing area A and viewing area B.

[0170] Furthermore, when multiple people view the content presented at an event within the same viewing area, if multiple users enter the same viewing area, the size of the viewing area may be expanded beyond the initially set size. In other words, the map generation unit 112 can set the viewing area according to the number of users included in that viewing area. In addition, the map generation unit 112 can dynamically set the viewing area in response to changes in the number of users included in that viewing area.

[0171] Figure 21 is a schematic diagram showing an example of setting up viewing areas according to the number of users in a viewing area according to an embodiment. Initially, as shown in section (a) of Figure 21, user 31, who has his terminal device 10 installed, is in viewing area 550a (area #1), and user 30, who has his terminal device 10 installed, is in viewing area 551a (area #2). The control unit 100 sets the size and other properties of viewing area 551a, for example, assuming that there is one user.

[0172] Consider the case where user 31 moves from viewing area 550a to viewing area 551a where user 30 is located. In this case, there will be two users, users 30 and 31, in viewing area 551a. If viewing area 551a is set up assuming only one user 30, there is a possibility that users 30 and 31 may come into contact or collide if they move. Therefore, as shown in section (b) of Figure 21, the control unit 100 expands viewing area 551a to become viewing area 551b, which accommodates both users 30 and 31. Viewing area 550a, where user 31 is no longer present and the number of users becomes [0], is reduced to the viewing area 550b.

[0173] Furthermore, if multiple users view the content of a collective event in their respective environments (viewing spaces), the amount of adjustment required for movement within each environment and the time required for physical movement will differ depending on the environment. Therefore, in order to ensure that each user has the experience of moving between content together in the same event, the terminal device 10 worn by each user may perform processing to adjust the movement speed within the environment among users and ensure consistency in movement time. In this case, the adjustment of movement time among users may be performed on the server 20, or it may be performed by direct communication between each terminal device 10.

[0174] Although the above description explains that the information processing system 1 according to the embodiment targets content presented at events held in real space, this is not limited to this example. That is, the information processing system 1 according to the embodiment is also applicable to events held in virtual space. Furthermore, the information processing system 1 is also applicable to events held in two-dimensional space that present content in the form of two-dimensional information, such as videos and paintings.

[0175] As described above, according to the information processing system 1 of this embodiment, users can virtually experience movement between multiple venues by moving through each viewing area provided within the viewing space. Therefore, it is possible to provide users with a greater sense of immersion in remote events delivered via streaming.

[0176] Furthermore, according to the information processing system 1 of this embodiment, the display position of each piece of content in the user's viewing space is set in accordance with the spatial relationship of each venue in the real world. As a result, the user can virtually experience moving between each venue in the real world within their viewing space, making it possible to achieve a greater sense of immersion in remote events delivered via streaming.

[0177] Furthermore, the information processing system 1 according to this embodiment can provide a transition effect to content transitions when the user moves across viewing areas. Therefore, even when the user moves across viewing areas to watch different content, the discomfort the user may feel due to abrupt content changes can be suppressed. As a result, users can experience greater immersion in remote events delivered via streaming.

[0178] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0179] Furthermore, this technology can also be configured as follows. (1) A control unit that determines the presentation position of the multiple contents in the viewing space based on first spatial information relating to a distribution source space to which multiple contents are associated, and second spatial information relating to a viewing space in which the multiple contents are viewed. Equipped with, Information processing device. (2) The aforementioned viewing space includes multiple viewing areas, The control unit, Based on the first spatial information and the second spatial information, the presentation position of each of the multiple content items relative to each of the multiple viewing areas is determined. The information processing device described in (1) above. (3) The aforementioned multiple viewing areas are such that a user viewing the multiple contents can move between them. The information processing device described in (2) above. (4) The first spatial information includes first positional information indicating the location of a plurality of presentation areas where each of the plurality of contents is presented, The second spatial information includes second positional information indicating the locations of the plurality of viewing areas, The control unit, Based on the first and second location information, the presentation position of each of the multiple content items is determined for each of the multiple viewing areas, each of which has a positional relationship with the respective presentation area. The information processing device described in (2) or (3) above. (5) The first spatial information includes preference information indicating the degree of preference of a user viewing each of the multiple pieces of content, The control unit, Based on the preference information included in the first spatial information, a content is selected from among the multiple content items to be presented in the multiple viewing areas. An information processing device as described in any of (2) to (4) above. (6) The first spatial information includes information indicating the type of each of the multiple contents, The control unit, Based on the information indicating the type included in the first spatial information, control is used to determine which of the multiple viewing areas the presentation position of each of the multiple contents will be in. An information processing device as described in any of (2) to (5) above. (7) The control unit, The plurality of viewing areas are defined for the viewing space based on at least one of the following: division information, role information, structural information, floor surface information, shape information, and equipment information. An information processing device as described in any of (2) to (6) above. (8) The control unit, In the region including the boundary between two adjacent viewing areas among the aforementioned plurality of viewing areas, a predetermined effect is presented. An information processing device as described in any of (2) to (7) above. (9) The aforementioned predetermined effect includes at least one of a visual effect and an auditory effect. The information processing device described in (8) above. (10) The predetermined effect is a crossfade of the content presented in each of the two viewing areas. The information processing apparatus described in (8) or (9) above. (11) The aforementioned predetermined effect is the presentation of pre-prepared content for movement. The information processing apparatus described in (8) or (9) above. (12) The aforementioned content for transport is The content is based on the scenery along the travel path when one of the two viewing areas moves between the location where the content designated as the presentation position is presented in the distribution source space, and the other viewing area of ​​the two viewing areas moves between the location where the content designated as the display position is presented in the distribution source space. The information processing device described in (11) above. (13) The aforementioned content for transport is This content is a reconstruction of the aforementioned scenery using a 3D model. The information processing device described in (12) above. (14) The control unit, The moving content is presented at a speed corresponding to the distance between the viewing area of ​​one of the two viewing areas of the plurality of content being at the position where the content designated as the presentation position is presented in the distribution source space, and the viewing area of ​​the other of the two viewing areas of the plurality of content being at the position where the content designated as the display position is presented in the distribution source space. An information processing device as described in any of (11) to (13) above. (15) The control unit, Depending on the time period during which the aforementioned multiple contents are presented in the distribution source space, the correspondence between each of the aforementioned multiple contents and each of the aforementioned multiple viewing areas is changed. An information processing device as described in any of (2) to (14) above. (16) The control unit, A user interface is presented for editing the correspondence between each of the aforementioned multiple content items and each of the aforementioned multiple viewing areas. An information processing device as described in any of (2) to (15) above. (17) The control unit, At least one of the aforementioned multiple viewing areas is set based on the number of users in that viewing area. An information processing device as described in any of (2) to (16) above. (18) The control unit, The viewing area is set according to the change in the number of users. The information processing device described in (17) above. (19) Executed by the processor, A control step of determining the presentation position of the multiple contents in the viewing space based on first spatial information relating to a distribution source space to which multiple contents are associated, and second spatial information relating to a viewing space in which the contents are viewed. Having, Information processing methods. (20) On the computer, A control step of determining the presentation position of the multiple contents in the viewing space based on first spatial information relating to a distribution source space to which multiple contents are associated, and second spatial information relating to a viewing space in which the contents are viewed. A computer-readable recording medium on which an information processing program for executing [the specified action] is recorded. [Explanation of symbols]

[0180] 1. Information Processing System 10 Terminal devices 20 servers 30,31 users 40,610 3D models 50 Event Map 70 Streaming Servers 71 Content Database 72 Reference DB 100 Control Unit 101 Sensor section 102,1509 Positioning Unit 103 Output section 110 User Information Acquisition Unit 111 Environmental Information Acquisition Department 112 Map Generation Unit 113 Output Control Unit 114 Content Information Acquisition Unit 115 Communications Department 116 Operation Input Section 300, 600, 601, 603 User Interface Screens Stages 500, 501, 502 500lay,500lay',501lay,502lay,502lay',540lay,541lay,542lay,543,544 area 520, 521 Timetable 530 Preset Maps 620 marks 1101 Outward-facing camera 1104 Gyro Sensor 1105 Accelerometer 1106 Directional sensor 1201L,1201R Display section 1505 Storage device

Claims

1. The system includes a control unit that determines the presentation position of the multiple contents in the viewing space based on first spatial information relating to a distribution source space to which multiple contents are associated, and second spatial information relating to a viewing space including multiple viewing areas in which the multiple contents are viewed. The control unit, Based on the first spatial information and the second spatial information, it is determined which of the multiple viewing areas each of the multiple contents will be placed in. Information processing device.

2. Each of the aforementioned multiple contents is associated with a priority, The control unit determines the area where each of the multiple contents will be placed based on the priority. The information processing apparatus according to claim 1.

3. The control unit, If, due to constraints based on the size of a specific viewing area, there is content that cannot be arranged according to its positional relationship in the distribution source space, some of the content will be placed in a different viewing area according to the priority. The information processing apparatus according to claim 2.

4. The first spatial information includes first positional information indicating the location of a plurality of presentation areas where each of the plurality of contents is presented, The second spatial information includes second positional information indicating the locations of the plurality of viewing areas, The control unit, Based on the first and second location information, the presentation position of each of the multiple content items is determined for each of the multiple viewing areas, each of which has a positional relationship with the respective presentation area. The information processing apparatus according to claim 1.

5. The first spatial information includes preference information indicating the degree of preference of a user viewing each of the multiple contents, The control unit, Based on the preference information included in the first spatial information, a content is selected from among the multiple content items to be presented in the multiple viewing areas. The information processing apparatus according to claim 1.

6. The first spatial information includes information indicating the type of each of the multiple contents, The control unit, Based on the information indicating the type included in the first spatial information, control is used to determine which of the multiple viewing areas the presentation position of each of the multiple contents will be placed in. The information processing apparatus according to claim 1.

7. The control unit, The plurality of viewing areas are defined for the viewing space based on at least one of the following: division information, role information, structural information, floor surface information, shape information, and equipment information. The information processing apparatus according to claim 1.

8. The control unit, In the region including the boundary between two adjacent viewing regions among the aforementioned plurality of viewing regions, a predetermined effect is presented. The information processing apparatus according to claim 1.

9. The aforementioned predetermined effect includes at least one of a visual effect and an auditory effect. The information processing apparatus according to claim 8.

10. The predetermined effect is a crossfade of the content presented in each of the two viewing areas. The information processing apparatus according to claim 8.

11. The aforementioned predetermined effect is the presentation of pre-prepared content for movement. The information processing apparatus according to claim 8.

12. The aforementioned content for transport is The content is based on the scenery of the travel path when one of the two viewing areas moves between the location where the content designated as the presentation position is presented in the source space, and the other viewing area moves between the location where the content designated as the presentation position is presented in the source space. The information processing apparatus according to claim 11.

13. The aforementioned content for transport is This content is a three-dimensional reconstruction of the aforementioned scenery. The information processing apparatus according to claim 12.

14. The control unit, The moving content is presented at a speed corresponding to the distance between the viewing area of ​​one of the two viewing areas of the plurality of content being at the position in the source space where the content designated as the presentation position is presented, and the viewing area of ​​the other of the two viewing areas of the plurality of content being at the position in the source space where the content designated as the presentation position is presented. The information processing apparatus according to claim 11.

15. The control unit, Depending on the time period during which the aforementioned multiple contents are presented in the distribution source space, the correspondence between each of the aforementioned multiple contents and each of the aforementioned multiple viewing areas is changed. The information processing apparatus according to claim 1.

16. The control unit, A user interface is presented for editing the correspondence between each of the aforementioned multiple content items and each of the aforementioned multiple viewing areas. The information processing apparatus according to claim 1.

17. The control unit, At least one of the aforementioned multiple viewing areas is set based on the number of users in that viewing area. The information processing apparatus according to claim 1.

18. The control unit, The viewing area is set according to the change in the number of users. The information processing apparatus according to claim 17.

19. Executed by the processor, The control step includes determining the presentation position of the multiple contents in the viewing space based on first spatial information relating to a distribution source space to which multiple contents are associated, and second spatial information relating to a viewing space including multiple viewing areas in which the multiple contents are viewed. The control step is, Based on the first spatial information and the second spatial information, it is determined which of the multiple viewing areas each of the multiple contents will be placed in. Information processing methods.

20. On the computer, A computer-readable recording medium on which a computer-readable recording medium is recorded, which contains an information processing program for executing a control step of determining the presentation position of the multiple contents in the viewing space, based on first spatial information relating to a distribution source space to which multiple contents are associated, and second spatial information relating to a viewing space including multiple viewing areas in which the multiple contents are viewed. The control step is, Based on the first spatial information and the second spatial information, it is determined which of the multiple viewing areas each of the multiple contents will be placed in. Recording medium.

Images