Information processing device, information processing method, and program

Abstract

This information processing device comprises: an acquisition unit that acquires information about an event outside of first content generated around a user who views or listens to the first content; a determination unit that, on the basis of the information about the event, makes a determination relating to whether it is necessary to issue a notification to the user regarding the event; and a notification control unit that performs a process relating to the notification to the user on the basis of the result of the determination.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program.

[0002] Devices for enhancing user experience (UX) have been actively developed. For example, in recent years, various types of head-mounted displays (HMDs) have been developed to allow users to be more immersed in virtual reality (VR) content.

[0003] Japanese Patent Application Laid-Open No. 2016-214822

[0004] To achieve a high user experience, a high degree of isolation from the outside world is desirable. For example, to allow a user to immerse themselves in VR content, it is desirable for the user to be able to enjoy the VR content while being isolated from the outside world (e.g., the real world outside the VR content). However, if the degree of isolation from the outside world is high, even events in the outside world that require the user's attention are blocked. In this case, the user is unable to respond to events that require attention, resulting in a decrease in the convenience of the device.

[0005] Therefore, the present disclosure proposes an information processing device, an information processing method, and a program that can achieve a high user experience without significantly impairing convenience.

[0006] In order to solve the above problem, one form of information processing device according to the present disclosure includes an acquisition unit that acquires information about an event outside of the first content that occurs around a user who is watching or listening to the first content, a determination unit that determines whether or not to notify the user about the event based on the event information, and a notification control unit that performs processing related to the notification to the user based on the result of the determination.

[0007] 1 is a diagram for explaining an overview of the present embodiment. FIG. 1 is a diagram for explaining an example of an output device. FIG. 2 is a diagram for explaining an example of an output device. FIG. 3 is a diagram for explaining an example of an output device. FIG. 4 is a diagram for explaining an example of an output device. FIG. 5 is a diagram for explaining an example of a configuration of an information processing system according to an embodiment of the present disclosure. FIG. 6 is a diagram for explaining an example of a configuration of a server according to an embodiment of the present disclosure. FIG. 7 is a diagram for explaining an example of a configuration of a terminal device according to an embodiment of the present disclosure. FIG. 8 is a diagram for explaining federated learning. FIG. 9 is a diagram for explaining a first example of a configuration of an information processing system. FIG. 10 is a diagram for explaining a second example of a configuration of an information processing system. FIG. 11 is a diagram for explaining a third example of a configuration of an information processing system. FIG. 12 is a flowchart for explaining learning data generation processing. FIG. 13 is a flowchart for explaining learning processing. FIG. 14 is a flowchart for explaining event notification processing. FIG. 15 is a diagram for explaining additional content data.

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0009] Additionally, in this description / specification, the phrase "at least one of" following a list of elements is understood to mean that the listed elements are optional. For example, "at least one of A, B, and C" means "(A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C)." "At least one of A, B, or C" and "at least one of A, B, and / or C" are similar to "at least one of A, B, and C." Here, A, B, and C are all arbitrary expressions (e.g., words, phrases, terms, or items).

[0010] In addition, in this specification and drawings, multiple components having substantially the same functional configuration may be distinguished by adding different numbers or letters to the same reference numeral. For example, multiple components having substantially the same functional configuration may be distinguished by adding different numbers or letters to the same reference numerals and letters to the terminal device 20 as needed. １ , 20 ２ , and 20 ３ However, when there is no need to particularly distinguish between multiple components having substantially the same functional configuration, only the same reference numerals are used. For example, the terminal device 20 １ , 20 ２ , and 20 ３ When there is no need to particularly distinguish between them, they will be simply referred to as terminal devices 20.

[0011] The present disclosure will be described in the following order: 1. Overview of the present disclosure 1-1. Overview of the problem 1-2. Overview of the solution 1-3. Determining whether or not to notify the user 2. Configuration of the information processing system 2-1. Configuration of the server 2-2. Configuration of the terminal device 2-3. Learning model 2-4. Specific configuration example of the information processing system 3. Operation of the information processing system 3-1. Learning data generation process 3-2. Learning process 3-3. Event notification process 3-4. Determining whether or not to notify based on context 3-5. Generation process of second content 4. Modifications 4-1. Modification related to device mode 4-2. Modification related to events 4-3. Modification related to event detection 4-4. Modification related to second content 4-5. Modification related to notification to the user 4-6. Other modifications 5. Conclusion

[0012] <<1. Overview of the Present Disclosure>> First, an overview of the present disclosure will be described.

[0013] <1-1. Overview of the Issue> There has been active development of devices to enhance user experience (UX). For example, in recent years, various types of head-mounted displays (HMDs) have been developed to allow users to be more immersed in virtual reality (VR) content.

[0014] In recent years, content production using these devices has been actively carried out. For example, in recent years, immersive content production using HMDs has been actively carried out. Examples of immersive content include VR games and metaverses (e.g., metaverse-type social networking services (SNSs)). Immersive content has great value because it allows users to escape from the real world and experience a completely different world.

[0015] On the other hand, even while playing immersive content, a user needs to respond to events that occur outside the immersive content (e.g., events in the real world). For example, if a family member or friend calls out to the user or there is a knock on the door, the user needs to respond. Also, if a fire alarm sounds, an emergency earthquake alert is broadcast, or the sound of glass breaking is heard, the user needs to respond immediately. Also, if an electrical appliance makes an operating sound (e.g., the sound of a microwave or washing machine finishing operation), the user needs to respond to the sound, although not immediately. This is not limited to sound, but also applies to light (e.g., a warning light or fire light).

[0016] The user experience (e.g., a sense of immersion) improves as the sound and / or light from the outside world (e.g., the real world) is blocked to a greater extent, but blocking sound and / or light that is necessary for the user reduces the convenience of the device.

[0017] On the other hand, reducing the degree of blocking of external sound and / or light may increase the convenience of the device, but may also allow unwanted sound and / or light to reach the user. For example, if the degree of blocking of external sound is reduced to the point where the above-mentioned sounds (e.g., the voices of family members and / or the operating sounds of electrical appliances) can be heard, unwanted sounds (e.g., children making noise on the street) may also reach the user. This may impair the user experience (e.g., the user's sense of immersion).

[0018] <1-2. Overview of Solution> In this embodiment, the above-described problem is solved by the following means.

[0019] Fig. 1 is a diagram for explaining an overview of this embodiment. The information processing system of this embodiment includes one or more information processing devices. In the example of Fig. 1, the information processing system includes a server and multiple terminal devices as information processing devices.

[0020] In the example of FIG. 1, the server is connected to at least one of a plurality of terminal devices via a network. Also, in the example of FIG. 1, the plurality of terminal devices are configured to operate in cooperation with one another. One of the plurality of terminal devices is a master device, and the other of the plurality of terminal devices is a slave device. The master device is, for example, a main device (e.g., a personal computer or a game console), and the slave device is a sub-device (e.g., an output device such as a head-mounted display, headphones, or earphones) that operates upon receiving a signal from the main device. In the example of FIG. 1, the slave device is a head-mounted display that is worn by a user U.

[0021] The form of the information processing device of this embodiment is not limited to the above. For example, in the example of FIG. 1, the master device and the slave device are shown as separate information processing devices (terminal devices). However, the master device and the slave device can be considered as a single information processing device (terminal device). Furthermore, the information processing device of this embodiment may be a standalone device. For example, the information processing device of this embodiment may be a standalone output device (e.g., a standalone head-mounted display) that does not have a master device. Furthermore, the information processing device of this embodiment does not necessarily have a communication function.

[0022] 2A and 2B are diagrams showing an example of an output device. In the example of Fig. 2A and Fig. 2B, the output device is a head-mounted display worn by a user U. Fig. 2A is a diagram of the output device as seen from the front, and Fig. 2B is a diagram of the output device as seen from the side.

[0023] 2A and 2B, the output device includes a display, a speaker, and a camera. The display is, for example, a VR display. The speaker is, for example, a speaker included in a headphone-type speaker unit. The camera is, for example, a fisheye camera. Note that the output device does not necessarily have to include a speaker and a camera.

[0024] The output device may include a sensor for detecting an event occurring around the user U. In the examples of FIGS. 2A and 2B , the output device includes a microphone as a sensor for detecting an event. Note that, although the output device includes multiple microphones in the examples of FIGS. 2A and 2B , the output device may include only one microphone. The output device may also include a communication function (e.g., a wireless communication function). In this case, the output device may transmit sound information detected by the microphone to another device. For example, if the output device is a terminal device functioning as a slave device, the output device may transmit sound information to a terminal device functioning as a master device. The output device may also transmit sound information to a server via a network.

[0025] It should be noted that the output device of this embodiment is not limited to a head-mounted display. For example, the output device of this embodiment may be an audio device without a display. Figures 3A and 3B are diagrams showing other examples of the output device. In the examples of Figures 3A and 3B, the output device is headphones worn by a user U. Figure 3A is a diagram showing the output device as seen from the front, and Figure 3B is a diagram showing the output device as seen from the side.

[0026] 3A and 3B, the output device includes a speaker, such as a speaker included in a headphone-type speaker unit.

[0027] The output device may include a sensor for detecting an event occurring around the user U, similar to the head-mounted display described above. In the examples of FIGS. 3A and 3B , the output device includes a microphone as a sensor for detecting an event. Note that, although the output device includes multiple microphones in the examples of FIGS. 3A and 3B , the output device may include only one microphone. The output device may also include a communication function (e.g., a wireless communication function). In this case, the output device may transmit sound information detected by the microphone to another device. For example, if the output device is a terminal device functioning as a slave device, the output device may transmit sound information to a terminal device functioning as a master device. The output device may also transmit sound information to a server via a network.

[0028] The user U uses such an output device to view or listen to content (hereinafter referred to as first content). For example, the user U uses a head-mounted display to view VR content (e.g., a VR game or the Metaverse). If the VR content includes sound and the head-mounted display has speakers, the user may not only view but also listen to the VR content. Alternatively, the user U uses headphones to listen to audio content (e.g., music or learning content).

[0029] Here, "a user watching or listening to content" refers to at least one of the following states: a state in which a user is watching content, a state in which a user is listening to content, and a state in which a user is watching and listening to content.

[0030] As described above, when the user U uses the output device to view or listen to the first content, the greater the degree of isolation from the outside world, the better the user experience. However, the convenience of the device is reduced because even sounds and / or light necessary for the user U are blocked. On the other hand, reducing the degree of isolation from the outside world improves the convenience of the device. However, the user experience is impaired because even unnecessary sounds and / or light reach the user.

[0031] Therefore, the information processing device of this embodiment operates as follows, thereby enabling realization of a high level of user experience without significantly impairing convenience.

[0032] In this embodiment, the outside world refers to the world outside the content (e.g., the first content). For example, if the content is VR content, the outside world is the world outside the virtual reality (e.g., virtual three-dimensional space) created by the VR content (i.e., the real world). The content is not limited to VR content. The content may be XR (Extended Reality / Cross Reality) content other than VR content (e.g., AR (Augmented Reality) content or MR (Mixed Reality) content). Alternatively, the content may be video content, still image content, or text content. In this case, the outside world may be the world outside the world created by these contents on the display (or in the user's head) (i.e., the real world). Furthermore, the content is not limited to visual content. The content may be sound content such as music. In this case, the outside world may be the world outside the world created by the sound content in the user's head (i.e., the real world).

[0033] The information processing device that performs the operations described below may be a terminal device or a server. The terminal device may be an output device (e.g., a head-mounted display or headphones), or may be a terminal device that functions as a master device of the output device (e.g., a personal computer or a game console).

[0034] Furthermore, the operations described below may be performed by a single information processing device, or may be performed by multiple information processing devices working together. For example, one terminal device may perform some of the operations described below, and another terminal device may perform the remaining operations. Alternatively, one or more terminal devices may perform some of the operations described below, and one or more servers may perform the remaining operations.

[0035] First, the information processing device acquires information on an event outside the first content that has occurred around the user U (hereinafter referred to as event information). For example, one or more information processing devices may acquire sound information detected by a microphone as the event information. Furthermore, one or more information processing devices may acquire light information detected by an optical sensor or an image sensor as the event information. Note that the event detected by the information processing device may be an event that has occurred outside the room where the user U is present. Even if an event has occurred outside the room where the user U is present, if the user U can detect the event when the first content is not being output, the event is an event that has occurred around the user U.

[0036] Here, the information processing device may acquire sound information detected by a microphone as event information. For example, one or more information processing devices may acquire human voice information (e.g., information on the sound of a family member or friend calling out to user U) as event information, or may acquire sound information other than human voice information (e.g., information on the sound of a door knock, a fire alarm, the sound of glass breaking, or the sound of an electrical appliance operating) as event information. One or more information processing devices may acquire sound information detected by a microphone of another information processing device via communication. If the information processing device itself has a microphone, the information processing device may acquire sound information from the microphone included in the information processing device itself.

[0037] Furthermore, the information processing device may acquire, as event information, information on light detected by an optical sensor or an image sensor. For example, one or more information processing devices may acquire, as event information, information on the light of a warning light or a fire. In this case, one or more information processing devices may acquire, via communication, information on light detected by an optical sensor or an image sensor of another information processing device. If the information processing device itself includes an optical sensor or an image sensor, the information processing device may acquire light information from the optical sensor or image sensor included in the information processing device itself.

[0038] Then, based on the event information, the information processing device determines whether or not it is necessary to notify the user U of the event. For example, the information processing device determines whether or not it is necessary for the user U to respond to the event indicated by the event information.

[0039] For example, if the event indicated by the event information satisfies a predetermined criterion, the information processing device determines that it is necessary to notify the user U about the event. For example, if the event indicated by the event information is a call from a family member or friend to the user U, the information processing device determines that it is necessary to notify the user U about the event. Furthermore, if the event indicated by the event information does not satisfy a predetermined criterion, the information processing device determines that it is not necessary to notify the user U about the event. For example, if the event indicated by the event information is, for example, the sound of children making noise on the road, the information processing device determines that it is not necessary to notify the user U about the event.

[0040] The information processing device may determine whether or not a notification is necessary using a learning model that has learned the behavior of user U (or one or more other users). The learning model will be described later. The method for determining whether or not a notification is necessary is not limited to a method that uses the learning model. The information processing device may also determine whether or not a notification is necessary without using the learning model.

[0041] Next, based on the result of the determination, the information processing device performs processing related to notification to the user U. For example, when the information processing device determines that it is necessary to notify the user U about the event, it notifies the user U about the event.

[0042] Note that, if the information processing device is not an output device, the information processing device may control the output device to notify the user U. For example, the information processing device may transmit a control signal to the output device to notify the user U about the event.

[0043] Note that the information processing device may provide a notification regarding an event within the first content in order to enhance the user experience (e.g., to prevent the user U from losing immersion in the first content). For example, the information processing device generates second content for notifying the user U based on event information. At this time, the information processing device may generate, as the second content, content corresponding to the content of the first content. For example, the information processing device may generate, as the second content, content in which a character corresponding to the content of the first content provides the notification. Then, the information processing device may play back the second content within the first content.

[0044] As a result, events requiring a low response are not notified to the user U, and events requiring a high response are notified to the user U. As a result, the information processing device can achieve a high user experience without significantly impairing convenience.

[0045] <1-3. Determining Whether User Notification is Necessary> In order to notify a user of an event in the external world without impairing the user experience, it is necessary to determine which events require the user to respond. As described above, an information processing device can use a learning model to determine whether a user notification is necessary. However, training the learning model requires a large amount of learning data. However, creating such data is not easy.

[0046] For example, to create training data, it is necessary to collect a large amount of sensor information according to the expected use case, and at the same time, to assign labels to the collected information (for example, binary values ​​such as "user needs to respond" or "user does not need to respond"). Labeling a large amount of information requires a lot of manpower, which increases costs.

[0047] Therefore, in this embodiment, this problem is solved by replacing manual labeling with automatic labeling.

[0048] For example, the information processing device of this embodiment acquires information about an event that has occurred around a user U, and at the same time, acquires information about the user's behavior in response to the event (for example, information about at least one of the user's voice and movement). Then, the information processing device labels the event information based on the information about the user's behavior.

[0049] For example, the information processing device determines whether the user U responded to the event indicated by the event information based on information about the user U's behavior (e.g., information about at least one of the user U's voice and movement). For example, the information processing device acquires information about the user's behavior B (e.g., replying, turning toward the voice, or not responding) immediately after the event A (e.g., calling out to the user U) is performed.

[0050] If the information processing device determines that the user's action B is a response to the event A, it associates response information indicating that "the user needs to respond" with the event information of the event A. On the other hand, if the information processing device determines that the user's action B is not a response to the event A, it associates response information indicating that "the user does not need to respond" with the event information of the event A.

[0051] The information processing device uses the information collected in this manner (event information and response information) as learning data to learn a learning model. The information processing device uses the learning model learned in this manner to determine whether or not a notification to user U is necessary.

[0052] Note that the learning model used by the information processing device to determine whether or not a notification to the user U is necessary is not limited to a model that has learned the behavior of the user U. The information processing device may determine whether or not a notification to the user U is necessary using a model that has learned the behavior of one or more other users.

[0053] The outline of this embodiment has been described above, and the information processing system 1 according to this embodiment will now be described in detail.

[0054] <<2. Configuration of Information Processing System>> First, the overall configuration of the information processing system 1 will be described.

[0055] 4 is a diagram illustrating a configuration example of an information processing system 1 according to an embodiment of the present disclosure. The information processing system 1 is a system for executing processing related to predetermined content. For example, the information processing system 1 is a system for executing processing related to XR content such as VR content.

[0056] The VR content may be, for example, immersive content such as a VR game or a metaverse (e.g., a metaverse-type SNS). The XR content is not limited to VR content. For example, the XR content may be AR content or MR content. The predetermined content is not limited to XR content. The predetermined content may be video content, still image content, text content, or sound content such as music.

[0057] The information processing system 1 includes one or more information processing devices. In the example of Fig. 4, the information processing system 1 includes a server 10 and a terminal device 20. Note that the devices in the figure may be considered devices in a logical sense. In other words, some of the devices in the figure may be realized by a virtual machine (VM), a container, a docker, or the like, and these may be physically implemented on the same hardware.

[0058] The server 10 and the terminal device 20 may each have a communication function. The server 10 and the terminal device 20 may be connected via a network N. In this case, the server 10 and the terminal device 20 can be referred to as communication devices. Although only one network N is shown in the example of Fig. 4, multiple networks N may exist.

[0059] Here, the network N is a communication network such as a LAN (Local Area Network), a WAN (Wide Area Network), a cellular network, a fixed telephone network, a regional IP (Internet Protocol) network, or the Internet. The network N may include a wired network or a wireless network. The network N may also include a core network. The core network is, for example, an EPC (Evolved Packet Core) or a 5GC (5G Core network). The network N may also include a data network other than the core network. The data network may be a service network of a telecommunications carrier, for example, an IMS (IP Multimedia Subsystem) network. The data network may also be a private network such as an in-house network.

[0060] The following describes in detail the configuration of each device that makes up the information processing system 1. Note that the configuration of each device shown below is merely an example. The configuration of each device may be different from the configuration shown below.

[0061] <2-1. Server Configuration> First, the configuration of the server 10 will be described.

[0062] The server 10 is an information processing device (computer) that performs processing related to predetermined content.

[0063] The predetermined content is typically immersive content such as a VR game or a metaverse. However, the predetermined content is not limited to immersive content. The predetermined content may be XR content other than immersive content (VR content, AR content, or MR content). Furthermore, the predetermined content may be video content, still image content, text content, or sound content such as music.

[0064] Any type of computer can be used as the server 10. The server 10 may be an application server or a web server. The server 10 may be a cloud server or an edge server. The server 10 may be a PC server, a mid-range server, or a mainframe server. The server 10 may be an information processing device (computer) that performs data processing (edge ​​processing) near a user or a terminal. For example, the server 10 may be an information processing device (computer) attached to or built into a base station or a roadside device. The server 10 may be an information processing device (computer) that performs cloud computing.

[0065] Fig. 5 is a diagram illustrating an example configuration of the server 10 according to an embodiment of the present disclosure. The server 10 includes a communication unit 11, a storage unit 12, and a control unit 13. Note that the configuration illustrated in Fig. 5 is a functional configuration, and the hardware configuration may be different from this. Furthermore, the functions of the server 10 may be distributed and implemented in multiple physically separated configurations. For example, the server 10 may be configured by multiple server devices.

[0066] The communication unit 11 is a communication interface for communicating with other devices. For example, the communication unit 11 is a LAN (Local Area Network) interface such as a NIC (Network Interface Card). The communication unit 11 may be a wired interface or a wireless interface. The communication unit 11 communicates with, for example, the terminal device 20 or another server 10 under the control of the control unit 13.

[0067] The storage unit 12 is a data readable / writable storage device such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, or a hard disk. The storage unit 12 stores, for example, event information, response information, and a learning model. There may be multiple pieces of event information, response information, and learning models. Furthermore, the storage unit 22 may not store all or some of the event information, response information, and learning models.

[0068] The event information is information about an event that occurs outside of a content such as immersive content (hereinafter referred to as a first content) that occurs around a user who is watching or listening to the first content. For example, the event information is information about sounds (e.g., audio information of family / friends calling out) that occur around a user who is watching or listening to the first content (e.g., VR content). The response information is information about the user's response to the event. For example, the response information is information indicating whether the user has responded to the event. The learning model will be described later.

[0069] The control unit 13 is a controller that controls each component of the server 10. The control unit 13 may be implemented by a processor such as a central processing unit (CPU) or a micro processing unit (MPU). Specifically, the control unit 13 may be implemented by a processor executing various programs stored in a storage device within the server 10 using a random access memory (RAM) or the like as a work area. The control unit 13 may be implemented by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The control unit 13 may also be implemented by a graphics processing unit (GPU). A CPU, an MPU, an ASIC, an FPGA, and a GPU can all be considered controllers. The control unit 13 may be configured by multiple physically separated entities. For example, the control unit 13 may be configured by multiple semiconductor chips.

[0070] The control unit 13 includes at least one block selected from the group consisting of an acquisition unit 131, a determination unit 132, a notification control unit 133, a generation unit 134, a detection unit 135, a storage unit (storage control unit) 136, and a learning unit 137. Each block constituting the control unit 13 (e.g., the acquisition unit 131 to the learning unit 137) is a functional block that represents a function of the control unit 13. These functional blocks may be software blocks or hardware blocks. For example, each of the above-described functional blocks may be a software module implemented by software (including a microprogram), or may be a circuit block on a semiconductor chip (die). Of course, each functional block may be a processor or an integrated circuit. The control unit 13 may be configured with functional units different from the above-described functional blocks. The method of configuring the functional blocks is arbitrary.

[0071] The control unit 13 may be configured with functional units different from the functional blocks described above. Furthermore, some or all of the operations of the blocks (e.g., the acquisition unit 131 to the learning unit 137) constituting the control unit 13 may be performed by another device. For example, some or all of the operations of the blocks constituting the control unit 13 may be performed by the control unit 23 of the terminal device 20 or the control unit 13 of another server 10.

[0072] <2-2. Configuration of Terminal Device> Next, the configuration of the terminal device 20 will be described.

[0073] The terminal device 20 is an information processing device that performs processing related to predetermined content.

[0074] The predetermined content is typically immersive content such as a VR game or a metaverse. However, the predetermined content is not limited to immersive content. The predetermined content may be XR content other than immersive content (VR content, AR content, or MR content). Furthermore, the predetermined content may be video content, still image content, text content, or sound content such as music.

[0075] The terminal device 20 may be a cooperative information processing device that operates in conjunction with one or more other information processing devices. For example, the terminal device 20 may be a master device (e.g., a personal computer or a game console) that controls a slave device, or a slave device (e.g., an output device such as a head-mounted display or headphones) that operates in response to a control signal from the master device. The terminal device 20 may also be a standalone information processing device that operates independently. In this case, the terminal device 20 does not necessarily have a communication function.

[0076] The terminal device 20 is typically a wearable device worn by a user. For example, the terminal device 20 may be an XR (Extended Reality / Cross Reality) device such as an AR (Augmented Reality) device, a VR (Virtual Reality) device, or an MR (Mixed Reality) device. In this case, the XR device may be a glasses-type device such as AR glasses or MR glasses, or a head-mounted device such as a VR head-mounted display. When the terminal device 20 is an XR device, the terminal device 20 may be a standalone device consisting only of a part worn by the user (e.g., a glasses part). Alternatively, the terminal device 20 may be a terminal-linked device consisting of a part worn by the user (e.g., a glasses part) and a terminal part linked to the part (e.g., a smart device).

[0077] The terminal device 20 may be headphones, a headset, or earphones. The terminal device 20 may be an intercom (intercommunication system) or a neckband speaker. The terminal device 20 may be a wearable device such as a smart watch.

[0078] The terminal device 20 is not limited to the above-described devices. For example, the terminal device 20 may be a television, a speaker, a music player, or a personal computer. The terminal device 20 may also be a mobile terminal such as a mobile phone, a smart device (smartphone or tablet), a PDA (Personal Digital Assistant), or a notebook PC. The terminal device 20 may also be an IoT (Internet of Things) device. Alternatively, any type of computer may be used as the terminal device 20.

[0079] FIG. 6 is a diagram illustrating an example configuration of a terminal device 20 according to an embodiment of the present disclosure. The terminal device 20 includes a communication unit 21, a storage unit 22, a control unit 23, an input unit 24, an output unit 25, and a sensor unit 26. Note that the configuration illustrated in FIG. 8 is a functional configuration, and the hardware configuration may be different. Furthermore, the functions of the terminal device 20 may be distributed and implemented in multiple physically separated components. Furthermore, the terminal device 20 may not include some of the above configurations. For example, the terminal device 20 may not include at least one of the communication unit 21, the input unit 24, the output unit 25, and the sensor unit 26.

[0080] The communication unit 21 is a communication interface for communicating with other devices. For example, the communication unit 21 is a LAN interface such as a NIC. The communication unit 21 may be a wired interface or a wireless interface. If the communication unit 21 has a wireless interface, the communication unit 21 may be configured to connect to the network N or other communication devices using a radio access technology (RAT) such as LTE (Long Term Evolution), NR (New Radio), Wi-Fi, or Bluetooth (registered trademark). The communication unit 21 communicates with, for example, the server 10 or other terminal devices 20 under the control of the control unit 23.

[0081] The storage unit 22 is a data readable / writable storage device such as a DRAM, an SRAM, a flash memory, or a hard disk. The storage unit 22 stores, for example, event information, response information, and a learning model. There may be multiple pieces of event information, response information, and learning model. Furthermore, the storage unit 22 may not store all or some of the event information, response information, and learning model.

[0082] The event information is information about an event that occurs outside of content such as VR content (hereinafter referred to as first content) around a user who is watching or listening to the first content. For example, the event information is information about a sound (e.g., audio information of a family member / friend calling out) that occurs around a user who is watching or listening to the first content (e.g., VR content).

[0083] The first content is typically immersive content such as a VR game, a metaverse (for example, a metaverse-type SNS), etc. However, the first content is not limited to immersive content. The first content may be XR content other than immersive content (VR content, AR content, or MR content). Furthermore, the first content is not limited to XR content. The first content may be video content, still image content, text content, or sound content such as music.

[0084] The response information is information about the user's response to the event. For example, the response information is information indicating whether or not the user has responded to the event. The learning model will be described later.

[0085] The control unit 23 is a controller that controls each component of the server 10. The control unit 23 may be implemented by a processor such as a central processing unit (CPU) or a micro processing unit (MPU). Specifically, the control unit 23 may be implemented by a processor executing various programs stored in a storage device inside the terminal device 20 using a random access memory (RAM) or the like as a work area. The control unit 23 may be implemented by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The control unit 23 may also be implemented by a graphics processing unit (GPU). A CPU, an MPU, an ASIC, an FPGA, and a GPU can all be considered controllers. The control unit 23 may be configured by multiple physically separated entities. For example, the control unit 23 may be configured by multiple semiconductor chips.

[0086] The control unit 23 includes at least one block from among an acquisition unit 231, a determination unit 232, a notification control unit 233, a generation unit 234, a detection unit 235, a storage control unit (storage unit) 236, and a learning unit 237. Each block constituting the control unit 23 (e.g., the acquisition unit 231 to the learning unit 237) is a functional block that indicates the function of the control unit 23. These functional blocks may be software blocks or hardware blocks. For example, each of the above-described functional blocks may be a software module realized by software (including a microprogram), or may be a circuit block on a semiconductor chip (die). Of course, each functional block may be a processor or an integrated circuit. The control unit 23 may be configured in functional units different from the above-described functional blocks. The method of configuring the functional blocks is arbitrary.

[0087] The control unit 23 may be configured with functional units different from the above-described functional blocks. Also, some or all of the operations of each block (e.g., the acquisition unit 231 to the learning unit 237) constituting the control unit 23 may be performed by another device. For example, some or all of the operations of each block constituting the control unit 23 may be performed by the control unit 13 of the server 10 or the control unit 23 of another terminal device 20.

[0088] The input unit 24 is an input device that accepts various inputs from the outside. For example, the input unit 24 is an operation device such as a keyboard, a mouse, or operation keys that allows the user to perform various operations. If a touch panel is employed in the terminal device 20, the touch panel is also included in the input unit 24. In this case, the user performs various operations by touching the screen with a finger or a stylus.

[0089] The output unit 25 is a device that outputs various types of information to the outside, such as sound, light, vibration, and image. The output unit 25 includes a display unit that displays various types of information. The display unit is, for example, a display device such as a liquid crystal display or an organic EL (Electro Luminescence) display. If a touch panel is employed in the terminal device 20, the display unit may be a device integrated with the input unit 24. If the terminal device 20 is an XR device, the terminal device 20 may be a transparent device that projects an image onto glasses, or a retinal projection device that projects an image directly onto the user's retina. The output unit 25 outputs various types of information to the user under the control of the control unit 23.

[0090] The sensor unit 26 is configured with one or more sensors. The one or more sensors included in the sensor unit 26 may include a sensor that detects the surroundings of the terminal device 20. For example, the one or more sensors included in the sensor unit 26 may include at least one of a sound sensor, an image sensor, a geomagnetic sensor, an illuminance sensor, a distance sensor (e.g., a ToF (Time of Flight) sensor), a barometric pressure sensor, a temperature sensor, a smoke sensor, and a light sensor.

[0091] The sensors included in the sensor unit 26 are not limited to sensors that detect the surroundings of the terminal device 20. The one or more sensors included in the sensor unit 26 may include a sensor that detects the position or attitude of the terminal device 20. For example, the one or more sensors included in the sensor unit 26 may include an acceleration sensor and / or a gyro sensor. For example, the one or more sensors included in the sensor unit 26 may include a 6DoF (Six degrees of freedom) sensor or a 3DoF (Three degrees of freedom) sensor. Furthermore, the one or more sensors included in the sensor unit 26 may include a positioning sensor such as a GNSS (Global Navigation Satellite System) sensor. The GNSS sensor may be a GPS (Global Positioning System) sensor, a GLONASS sensor, a Galileo sensor, or a QZSS (Quasi-Zenith Satellite System) sensor.

[0092] The one or more sensors included in the sensor unit 26 may include a sensor unit configured by combining multiple sensors. For example, the one or more sensors included in the sensor unit 26 may include an inertial measurement unit (IMU) configured by combining multiple sensors selected from a positioning sensor (e.g., a GNSS sensor), an acceleration sensor, and a gyro sensor. The sensor unit can also be considered a type of sensor.

[0093] Furthermore, one or more sensors included in the sensor unit 26 may include a device / component configured using a sensor. For example, one or more sensors included in the sensor unit 26 may include at least one of a camera (e.g., a visible light camera, an infrared camera, or a light field camera), a LiDAR (Light Detection and Ranging), a radar (e.g., a microwave radar or a millimeter wave radar), a microphone, and an imaging device. A device / component configured using a sensor can also be considered a type of sensor.

[0094] In addition, the one or more sensors provided in the sensor unit 26 may include a sensor that detects at least one of the color of an object, the speed of an object, the acceleration of an object, the reflectivity of an object, the transmittance of an object, the distance to the object, the temperature of the object / environment, geomagnetism, illuminance, air pressure, light, and sound.

[0095] Furthermore, the one or more sensors provided in the sensor section 26 may include a sensor / sensor unit / device / component configured by combining two or more sensors selected from the above-mentioned plurality of sensors.

[0096] <2-3. Learning Model> Next, the learning model used in this embodiment will be described.

[0097] As described above, the memory unit 12 of the server 10 or the memory unit 22 of the terminal device 20 stores a learning model. The learning model is a model that learns the relationship between events occurring around a user and the user's responses to the events. The learning model can also be called an artificial intelligence (AI) model, a machine learning (ML) model, an AI / ML model, or the like.

[0098] The learning model is, for example, a machine learning model such as a neural network model. A neural network model is composed of layers called an input layer, an intermediate layer (or hidden layer), and an output layer, each of which includes a plurality of nodes, and each node is connected via an edge. Each layer has a function called an activation function, and each edge is weighted. The learning model has one or more intermediate layers (or hidden layers). When the learning model is a neural network model, learning the learning model means, for example, setting the number of intermediate layers (or hidden layers), the number of nodes in each layer, or the weight of each edge.

[0099] Here, the neural network model may be a model based on deep learning. In this case, the neural network model may be a model in a form called a deep neural network (DNN). Furthermore, the neural network model may be a convolution neural network (CNN), a recurrent neural network (RNN), a long short-term memory (LSTM), a fully-connected neural network, or a model in a form called an autoencoder. Of course, the neural network model is not limited to these types of models.

[0100] Furthermore, the learning model is not limited to a neural network model. For example, the learning model may be a model based on reinforcement learning. In reinforcement learning, behaviors (settings) that maximize value are learned through trial and error. Alternatively, the learning model may be a logistic regression model.

[0101] The learning model may be composed of multiple models. For example, the learning model may be composed of multiple neural network models. More specifically, the learning model may be composed of multiple neural network models selected from, for example, DNN, CNN, RNN, LSTM, fully-connected neural network, and autoencoder. When the learning model is composed of multiple neural network models, these multiple neural network models may be in a subordinate relationship or a parallel relationship.

[0102] As described above, the information processing device of this embodiment (e.g., the server 10 or the terminal device 20) stores, for example, event information, response information, or a learning model. The event information is information about an event that occurs outside the first content around a user who is watching or listening to the first content. The response information is information about the user's response to the event. When an event occurs around the user, the learning model is used to determine whether or not a notification about the event needs to be sent to the user. The learning model will be described in detail below.

[0103] The learning model is, for example, a learning model (trained model) that has been trained using event information as input data and response information to the event indicated by the event information as a correct label (teaching data). When an information processing device inputs event information to the learning model, the learning model outputs, for example, information indicating whether or not a notification regarding the event indicated by the event information (for example, a notification indicating that an event has occurred) needs to be sent to the user (for example, 0 (no notification necessary), 1 (notification necessary)).

[0104] In Figures 5 and 6, text information such as "learning model" is described as information stored in memory unit 12 and memory unit 22, but in reality, memory unit 12 and memory unit 22 store character strings, numerical values, etc. that indicate the structure of the model and connection coefficients.

[0105] The learning model may be a model that has been trained to use data pairs of event information and response information as learning data and, when event information is input, output information indicating whether a user needs to be notified of an event indicated by the event information (hereinafter referred to as necessity information). In this case, the learning model may include an input layer that receives the event information, an output layer that outputs the necessity information, a first element that belongs to a layer other than the output layer and that is any layer between the input layer and the output layer, and a second element whose value is calculated based on the first element and a weight of the first element, and may be a model that causes a computer to function to output the necessity information from the output layer in accordance with the event information input to the input layer by performing a calculation based on the first element and the weight of the first element, using each element belonging to each layer other than the output layer as the first element, for the information input to the input layer.

[0106] Here, it is assumed that the learning model is realized by a neural network having one or more hidden layers, such as a DNN. In this case, the first element included in the learning model corresponds to any node in the input layer or hidden layer. The second element corresponds to a next-stage node, which is a node to which a value is transmitted from the node corresponding to the first element. The weight of the first element corresponds to a connection coefficient, which is a weight taken into account for the value transmitted from the node corresponding to the first element to the node corresponding to the second element.

[0107] Also, assume that the learning model is realized as a regression model expressed as "y = a1 * x1 + a2 * x2 + ... + ai * xi". In this case, the first element included in the learning model corresponds to input data (xi) such as x1 or x2. The weight of the first element corresponds to the coefficient ai corresponding to xi. Here, the regression model can be regarded as a simple perceptron having an input layer and an output layer. When each model is regarded as a simple perceptron, the first element corresponds to one of the nodes in the input layer, and the second element can be regarded as a node in the output layer.

[0108] The information processing device calculates the information to be output using a model with any structure, such as a neural network or a regression model. Specifically, coefficients are set in the learning model so that necessity information is output when event information is input. For example, the server 10 sets the coefficients based on the similarity between the value obtained by inputting the event information into the learning model and the response information. The server 10 uses such a learning model to generate necessity information from the event information.

[0109] In the above example, a model that outputs necessity information when event information is input is shown as an example of the learning model. However, the learning model according to the embodiment may be a model that is generated based on results obtained by repeatedly inputting and outputting data to the learning model.

[0110] Furthermore, when the information processing device performs learning or generates output information using GAN (Generative Adversarial Networks), the learning model may be a model that constitutes part of the GAN.

[0111] In addition, the information processing device (learning device) that learns the learning model (for example, the learning model) may be the server 10, the terminal device 20, or another information processing device.

[0112] For example, assume that the information processing device (learning device) is the server 10. In this case, the server 10 learns a learning model and stores the learned learning model in the storage unit 12. More specifically, the server 10 sets the connection coefficients of the learning model so that, when event information is input to the learning model, the learning model outputs necessity information.

[0113] Also, for example, assume that the information processing device (learning device) is the terminal device 20. In this case, the terminal device 20 learns a learning model and stores the learned learning model in the storage unit 22. More specifically, the terminal device 20 sets the connection coefficients of the learning model so that, when event information is input to the learning model, the learning model outputs necessity information.

[0114] For example, an information processing device (learning device) inputs event information to a node in the input layer of a learning model, and propagates the data through each intermediate layer to the output layer of the learning model, thereby outputting necessity information.The information processing device then corrects the connection coefficients of the learning model based on the difference between the necessity information actually output by the learning model and the response information used as the correct label (teaching data).At this time, the information processing device may correct the connection coefficients using a technique such as backpropagation.At this time, the information processing device may correct the connection coefficients based on the cosine similarity between a vector representing the teaching data and a vector representing the value actually output by the learning model.

[0115] Any learning algorithm may be used for the learning. For example, the information processing device (learning device) may perform learning of the learning model using a learning algorithm such as a neural network, a support vector machine, clustering, reinforcement learning, a random forest, or a decision tree.

[0116] Furthermore, the learning algorithm used in this embodiment may be one in which each information processing device (learning device) learns independently, or one in which multiple information processing devices learn in cooperation with each other. An example of a learning algorithm in which learning devices learn in cooperation with each other is federated learning. Federated learning will be described below.

[0117] Federated learning is a type of algorithm that optimizes machine learning models, allowing for training of learning models without exposing the private data of individual devices.

[0118] FIG. 7 is a diagram illustrating federated learning. In federated learning, multiple terminal devices 20 with data and a server 10 that oversees them cooperate to advance learning. First, each of the multiple terminal devices 20 downloads a model from the server 10 and updates the model by learning it using its own data. The terminal devices 20 then upload the updated model to the server 10. The server 10 aggregates a large number of updated models to generate a single new model. The server 10 then distributes the new model to the multiple terminal devices 20. Each of the multiple terminal devices 20 learns using the new model. The multiple terminal devices 20 and the server 10 repeat these processes. In federated learning, a model is updated within the terminal device 20 and uploaded to the server 10, making it possible to perform learning without disclosing private data in the terminal devices 20.

[0119] 2-4. Specific Configuration Example of Information Processing System Next, a specific configuration example of the information processing system 1 will be described.

[0120] The information processing system 1 shown below (the information processing system 1 shown in FIG. 8, FIG. 9, or FIG. 10) may be configured by one information processing device. For example, the information processing system 1 shown below may be configured by one server 10, one terminal device 20, or one other information processing device.

[0121] Furthermore, the information processing system 1 shown below (the information processing system 1 shown in FIG. 8, FIG. 9, or FIG. 10) may be configured with a plurality of information processing devices. For example, the information processing system 1 shown below may be configured with a plurality of servers 10, a plurality of terminal devices 20, or a plurality of other information processing devices. For example, the information processing system 1 shown below may be configured with a plurality of information processing devices selected from one or more servers 10, one or more terminal devices 20, and one or more other information processing devices.

[0122] The configuration of the information processing system 1 shown in Figure 8, Figure 9, or Figure 10 is merely an example. The information processing system 1 is not limited to the configuration shown in Figure 8, Figure 9, or Figure 10. In this case, the information processing system 1 may be configured by one information processing device, or may be configured by multiple information processing devices. When the information processing system 1 is configured by one information processing device, the description of "information processing system 1" described above or below can be replaced with "information processing device."

[0123] 8 is a diagram illustrating a first configuration example of the information processing system 1. The information processing system 1 according to the first configuration example generates learning data to be used for training a learning model. Specifically, the information processing system 1 according to the first configuration example detects an event and a user's response to the event, and stores the detection results as learning data.

[0124] The information processing system 1 according to the first configuration example may be the same system as the information processing system 1 according to the second configuration example described below and / or the information processing system 1 according to the third configuration example described below, or may be a different system.

[0125] The information processing system 1 according to the first configuration example includes a microphone, a camera, a gyro sensor, an acceleration sensor, a speaker, a display, a control unit, an event detection unit, a response detection unit, and a memory unit.

[0126] One or more sensors (in the example of FIG. 8 , at least one of a microphone, a camera, a gyro sensor, and an acceleration sensor) included in the information processing system 1 correspond to one or more sensors included in the sensor unit 26 of the terminal device 20. The speaker and the display correspond to the output unit 25 of the terminal device 20. The control unit corresponds to the control unit 13 of the server 10 or the control unit 23 of the terminal device 20. The event detection unit and the response detection unit correspond to the detection unit 135 of the server 10 or the detection unit 235 of the terminal device 20. The memory unit corresponds to the memory unit 12 of the server 10 or the memory unit 22 of the terminal device 20.

[0127] One or more sensors (in the example of Figure 8, at least one of a microphone, a camera, a gyro sensor, and an acceleration sensor) included in the information processing system 1 transmit detected sensor information (also called detection information or sensing content) to the control unit.

[0128] The control unit transmits sensor information from one or more sensors to the event detection unit. For example, the control unit transmits sensor information from one or more microphones to the event detection unit.

[0129] The event detection unit detects an event based on sensor information from one or more sensors. For example, the event detection unit may detect an acoustic event based on sensor information from one or more microphones. For example, the event detection unit may detect information on the type of acoustic event and the time period of the acoustic event from information on the sound detected by the microphones. In this case, the event detection unit may detect the acoustic event by comparing the information on the sound detected by the microphones with pre-stored sound patterns. A learning model that learns sound characteristics of a predetermined event may be used to detect the acoustic event. Note that the event detection unit may estimate the direction in which the sound originates based on sensor information from multiple microphones. Then, the event detection unit may detect the acoustic event based on the sound pattern and the direction in which the sound originates. Alternatively, the event detection unit may detect the acoustic event based on a change in sound pressure, etc. The acoustic event detection method is not limited to the above, and various known methods may be used.

[0130] An acoustic event may be, for example, a call from a family member or friend, a knock on the door, a fire alarm, an earthquake early warning, the sound of glass breaking, or the sound of an electrical appliance operating. Of course, acoustic events are not limited to the above.

[0131] Note that the event detection method described here is merely an example. The information processing system 1 / information processing device can detect events using various known methods. The events detected by the event detection unit are not limited to acoustic events. For example, the events detected by the event detection unit may be events detected from image information (e.g., still image and / or video information) detected by a camera / image sensor.

[0132] The control unit acquires, as the event information, information related to the event detected by the event detection unit. For example, the control unit may acquire, as the event information, sensor information when the event detection unit detects the event. For example, the control unit may acquire, as the event information, sound information for a time period when the event detection unit detects the event. Furthermore, the control unit may acquire, as the event information, information indicating the content of the event detected by the event detection unit (for example, information indicating the type of event, such as "human voice," "operation sound of a home appliance," or "alarm sound") associated with sensor information when the event was detected.

[0133] The control unit transmits sensor information (sensing content) of one or more sensors (in the example of FIG. 8, at least one of a microphone, a camera, a gyro sensor, and an acceleration sensor) to the response detection unit.

[0134] The response detection unit detects the user's response based on the sensor information. For example, the response detection unit may detect the user's neck movement and / or the user's vocalization as the user's response. More specifically, the response detection unit may detect the user's head movement (e.g., nodding and / or shaking of the head) and / or detect a voice activity from audio information detected by directivity toward the user's mouth, thereby detecting the user's neck movement and / or the user's vocalization.

[0135] The response detection unit determines that the user has responded to the event detected by the event detection unit when a user response (for example, at least one of nodding, shaking of the head, and vocalization) is detected, and determines that the user has not responded to the event detected by the event detection unit when a user response is not detected.

[0136] The method of detecting a user's response shown here is merely an example, and the information processing system 1 / information processing device can detect a user's response using various known methods.

[0137] The control unit acquires the detection result of the response detection unit as response information. The detection result of the response detection unit is, for example, a determination result indicating whether or not the user has responded to the event detected by the event detection unit. The response information may include sensor information when the user's response is detected or not.

[0138] The control unit stores the event information and response information for the event indicated by the event information in association with each other in the storage unit.

[0139] 9 is a diagram showing a second configuration example of the information processing system 1. The information processing system 1 according to the second configuration example performs learning of a learning model.

[0140] The information processing system 1 according to the second configuration example may be the same system as the information processing system 1 according to the first configuration example described above and / or the information processing system 1 according to the third configuration example described below, or may be a different system.

[0141] The information processing system 1 according to the second configuration example includes a control unit, a learning unit, and a storage unit. The control unit corresponds to the control unit 13 of the server 10 or the control unit 23 of the terminal device 20. The learning unit corresponds to the learning unit 137 of the server 10 or the learning unit 237 of the terminal device 20. The storage unit corresponds to the storage unit 12 of the server 10 or the storage unit 22 of the terminal device 20.

[0142] The control unit acquires data (hereinafter referred to as learning data) used to learn the learning model. For example, the control unit acquires data (event information and response information) generated by the information processing system 1 according to the first configuration example as learning data. The control unit may acquire the learning data from the storage unit, or may acquire the learning data from another information processing system / another information processing device via communication. The control unit transmits the acquired learning data to the learning unit.

[0143] The learning unit learns the learning model using the learning data (event information and response information) acquired from the control unit. When the learning unit completes learning the learning model, it transmits the learned learning model to the control unit.

[0144] The learning unit stores the learned learning model in the memory unit.

[0145] 10 is a diagram showing a third configuration example of the information processing system 1. The information processing system 1 according to the third configuration example performs processing related to notifying a user who is watching or listening to the first content of an event that has occurred around the user. Specifically, the information processing system 1 according to the third configuration example determines whether or not the user needs to be notified about the event, and performs processing related to notifying the user based on the result of the determination.

[0146] The first content is typically immersive content such as a VR game or a metaverse. However, the first content is not limited to immersive content. The first content may be XR content other than immersive content (VR content, AR content, or MR content). Furthermore, the first content is not limited to XR content. The first content may be video content, still image content, text content, or sound content such as music.

[0147] The information processing system 1 according to the third configuration example may be the same system as the information processing system 1 according to the first configuration example described above and / or the information processing system 1 according to the second configuration example described above, or may be a different system.

[0148] The information processing system 1 according to the third configuration example includes a microphone, a camera, a gyro sensor, an acceleration sensor, a speaker, a display, a control unit, an event detection unit, a response detection unit, and a memory unit.

[0149] The microphone, camera, gyro sensor, and acceleration sensor correspond to one or more sensors included in the sensor unit 26 of the terminal device 20. The speaker and display correspond to the output unit 25 of the terminal device 20. The control unit corresponds to the control unit 13 of the server 10 or the control unit 23 of the terminal device 20. The event detection unit corresponds to the detection unit 135 of the server 10 or the detection unit 235 of the terminal device 20. The determination unit corresponds to the determination unit 132 of the server 10 or the determination unit 232 of the terminal device 20. The generation unit corresponds to the generation unit 134 of the server 10 or the generation unit 234 of the terminal device 20. The storage unit corresponds to the storage unit 12 of the server 10 or the storage unit 22 of the terminal device 20.

[0150] One or more sensors (in the example of Figure 10, at least one of a microphone, a camera, a gyro sensor, and an acceleration sensor) included in the information processing system 1 transmit detected sensor information (also called detection information or sensing content) to the control unit.

[0151] The control unit transmits sensor information from one or more sensors to the event detection unit. For example, the control unit transmits sensor information from one or more microphones to the event detection unit.

[0152] The event detection unit detects an event based on sensor information from one or more sensors. For example, the event detection unit may detect an acoustic event based on sensor information from one or more microphones.

[0153] The control unit acquires, as event information, information related to the event detected by the event detection unit. For example, the control unit may acquire, as event information, sensor information when the event detection unit detects the event. The control unit may also acquire, as event information, information that associates information indicating the content of the event detected by the event detection unit with the sensor information when the event was detected. The control unit transmits the acquired event information to the determination unit.

[0154] The control unit acquires a learning model that has learned the relationship between events occurring around the user and the user's responses to those events. For example, the control unit acquires the learning model acquired by the information processing system 1 according to the second configuration example. The control unit may acquire the learning model from the storage unit, or may acquire the learning model from another information processing system / another information processing device via communication. The control unit transmits the acquired learning model to the determination unit.

[0155] The determination unit determines whether or not a notification regarding the event needs to be sent to the user based on the event information. For example, the information processing device determines whether or not a user needs to respond to an event indicated by the event information. For example, the determination unit inputs the event information into a learning model to determine whether or not a notification regarding the event needs to be sent to the user. The determination unit transmits the determination result to the control unit.

[0156] When a notification about an event needs to be sent to the user, the control unit sends event information (e.g., an acoustic event) to the generation unit, which may include sensor information at the time the event was detected.

[0157] The information processing device generates second content for notifying the user based on the event information. At this time, the generation unit generates content corresponding to the content of the first content (e.g., VR content) as the second content. For example, the information processing device may generate content in which a character corresponding to the content of the first content makes the notification as the second content. The second content will be described in detail later. The generation unit transmits the second content to the control unit.

[0158] The control unit outputs the sound and / or image of the second content to the speaker and / or the display. Note that if the second content corresponds to the content of the first content, the control unit may play the second content within the first content.

[0159] <<3. Operation of Information Processing System>> The configuration of the information processing system 1 has been described above. Next, the operation of the information processing system 1 having such a configuration will be described.

[0160] The processes executed by the information processing system 1 are divided into a learning data generation process, a learning process, and an event notification process.

[0161] At least one of these processes (the learning data generation process, the learning process, and the event notification process) may be executed by a single information processing device. For example, at least one of these processes (e.g., the learning data generation process, the learning process, and the event notification process) may be executed by a single server 10 or a single terminal device 20. Of course, a single information processing device may execute all of these processes.

[0162] When one information processing device executes processing, the above-mentioned or below-mentioned description of "information processing system 1" can be replaced with "information processing device," "server 10," or "terminal device 20."

[0163] Furthermore, at least one of these processes (the learning data generation process, the learning process, and the event notification process) may be performed cooperatively by multiple information processing devices. For example, at least one of these processes may be performed cooperatively by multiple servers 10 or multiple terminal devices 20. At least one of these processes may be performed cooperatively by multiple information processing devices selected from one or more servers 10 and one or more terminal devices 20.

[0164] When a plurality of information processing devices cooperate to execute processing, the term "information processing device" described above or below can be replaced with "information processing system 1."

[0165] <3-1. Learning Data Generation Processing> First, the learning data generation processing will be described.

[0166] FIG. 11 is a flowchart showing a training data generation process. The training data generation process is a process for generating training data to be used for training a training model. Note that not all of the processes shown in FIG. 11 are necessarily required processes for this embodiment. In other words, each of the processes shown in FIG. 11 can be performed independently.

[0167] In the following description, it is assumed that one information processing device executes the learning data generation process. Here, the information processing device executing the learning data generation process may be the server 10 or the terminal device 20.

[0168] Note that multiple information processing devices may cooperate to execute the training data generation process. For example, one server 10 may execute part of the training data generation process, and another server 10 may execute other parts of the training data generation process. For example, one terminal device 20 may execute part of the training data generation process, and another terminal device 20 may execute other parts of the training data generation process. For example, one server 10 may execute part of the training data generation process, and one terminal device 20 may execute other parts of the training data generation process. Of course, three or more information processing devices may cooperate to execute the training data generation process. For example, three or more information processing devices selected from one or more servers 10 and one or more terminal devices 20 may execute the training data generation process.

[0169] When multiple information processing devices cooperate to execute the learning data generation process, the term "information processing device" described below can be replaced with "information processing system 1."

[0170] Furthermore, when multiple information processing devices cooperate to execute the learning data generation process, the description of "information processing device" below can be replaced with "Nth information processing device" as appropriate, where N is an arbitrary integer. For example, when multiple information processing devices cooperate to execute the learning data generation process, the description of "information processing device" below can be replaced with "first information processing device" or "second information processing device" as appropriate.

[0171] Furthermore, the description of "information processing device" below can be replaced with "server 10" or "terminal device 20." For example, if one or more servers 10 and one or more terminal devices 20 cooperate to execute the learning data generation process, the description of "information processing device" below can be replaced with "one or more servers 10" or "one or more terminal devices 20" as appropriate.

[0172] In addition, the description of "information processing device" below refers to "server 10 Ｎ " or "Terminal device 20 Ｍ " where N and M are any integers. For example, if two servers 10 cooperate to execute the learning data generation process, the description of "information processing device" below may be appropriately changed to "server 10 １ " or "Server 10 ２ Furthermore, for example, if two terminal devices 20 cooperate to execute the learning data generation process, the description of "information processing device" below can be appropriately replaced with "terminal device 20" １ " or "Terminal device 20 ２ For example, if two servers 10 and two terminal devices 20 cooperate to execute the learning data generation process, the description of "information processing device" below can be appropriately replaced with "server 10" １ ", "Server 10 ２ ", "Terminal device 20 １ " or "Terminal device 20 ２ " can be replaced with ".

[0173] The learning data generation process is executed, for example, by a control unit of an information processing device (for example, the control unit 13 of the server 10 or the control unit 23 of the terminal device 20). Note that the learning data generation process may be executed in parallel with a learning process (described later) and / or an event notification process (described later). Of course, the learning data generation process may be executed regardless of the execution timing of the learning process and / or the event notification process. The learning data generation process will be described below with reference to the flowchart of FIG. 11.

[0174] First, a detection unit of the information processing device (e.g., the detection unit 135 of the server 10 or the detection unit 235 of the terminal device 20) executes a process for detecting an event occurring around a user who is watching or listening to predetermined content, outside the predetermined content (step S101). The predetermined content may be the first content described above or below (e.g., XR content such as VR content). The event outside the predetermined content may be, for example, a sound occurring around the user (e.g., a human voice).

[0175] Then, the control unit of the information processing device determines whether an event has been detected in step S101 (step S102).

[0176] The processes of steps S101 and S202 may be performed during the execution of the event notification process described below. In this case, the processes of steps S101 and / or S202 may be common processes with the processes of steps S301 and / or S302 of the event notification process.

[0177] If an event has not been detected (step S102: No), the control unit returns the process to step S101. If an event has been detected (step S102: Yes), the control unit acquires information related to the event detected in step S101 as event information (step S103). For example, the control unit may acquire, as event information, sensor information when the event detection unit detected the event. Alternatively, the control unit may acquire, as event information, information indicating the content of the event detected by the event detection unit (for example, information indicating the type of event, such as "human voice," "operation sound of a home appliance," or "alarm sound") associated with the sensor information when the event was detected.

[0178] Next, a detection unit of the information processing device (for example, the detection unit 135 of the server 10 or the detection unit 235 of the terminal device 20) detects a user's response to the event detected in step S101 (step S104). For example, the detection unit may detect the user's neck movement and / or the user's vocalization as the user's response.

[0179] The control unit of the information processing device then acquires the detection result of step S103 as response information (step S105). Here, the response information may be a determination result indicating whether or not the user has responded to the event detected in step S101. The response information may include sensor information when the user's response is detected or not.

[0180] Next, a detection unit of the information processing device (for example, the detection unit 135 of the server 10 or the detection unit 235 of the terminal device 20) detects a user's response to the event detected in step S101 (step S103). For example, the detection unit may detect the user's neck movement and / or the user's vocalization as the user's response.

[0181] Next, the storage control unit of the information processing device (for example, the storage unit 136 of the server 10 or the storage control unit 236 of the terminal device 20) associates the event information acquired in step S102 with the response information acquired in step S104 and stores them in the memory unit as learning data.

[0182] When the saving is completed, the control unit of the information processing device returns the process to step S101.

[0183] <3-2. Learning Process> Next, the learning process will be described.

[0184] FIG. 12 is a flowchart showing a learning process. The learning process is a process for learning a learning model. The learning performed in the learning process may be a re-learning of a learned learning model. Note that not all of the processes shown in FIG. 12 are necessarily required processes for this embodiment. In other words, each of the processes shown in FIG. 12 can be performed independently.

[0185] In the following description, it is assumed that one information processing device executes the learning process. Here, the information processing device executing the learning process may be the server 10 or the terminal device 20.

[0186] It should be noted that a plurality of information processing devices may cooperate to execute the learning process. For example, one server 10 may execute a part of the learning process, and another server 10 may execute another part of the learning process. Also, for example, one terminal device 20 may execute a part of the learning process, and another terminal device 20 may execute another part of the learning process. Also, for example, one server 10 may execute a part of the learning process, and one terminal device 20 may execute another part of the learning process. Of course, three or more information processing devices may cooperate to execute the learning process. For example, three or more information processing devices selected from one or more servers 10 and one or more terminal devices 20 may execute the learning process.

[0187] When multiple information processing devices cooperate to execute the learning data generation process, the term "information processing device" described below can be replaced with "information processing system 1."

[0188] Furthermore, when multiple information processing devices cooperate to execute the learning data generation process, the description of "information processing device" below can be replaced with "Nth information processing device" as appropriate, where N is an arbitrary integer. For example, when multiple information processing devices cooperate to execute the learning data generation process, the description of "information processing device" below can be replaced with "first information processing device" or "second information processing device" as appropriate.

[0189] Furthermore, the description of "information processing device" below can be replaced with "server 10" or "terminal device 20." For example, if one or more servers 10 and one or more terminal devices 20 cooperate to execute the learning data generation process, the description of "information processing device" below can be replaced with "one or more servers 10" or "one or more terminal devices 20" as appropriate.

[0190] In addition, the description of "information processing device" below refers to "server 10 Ｎ " or "Terminal device 20 Ｍ " where N and M are any integers. For example, if two servers 10 cooperate to execute the learning data generation process, the description of "information processing device" below may be appropriately changed to "server 10 １ " or "Server 10 ２ Furthermore, for example, if two terminal devices 20 cooperate to execute the learning data generation process, the description of "information processing device" below can be appropriately replaced with "terminal device 20" １ " or "Terminal device 20 ２ For example, if two servers 10 and two terminal devices 20 cooperate to execute the learning data generation process, the description of "information processing device" below can be appropriately replaced with "server 10" １ ", "Server 10 ２ ", "Terminal device 20 １ " or "Terminal device 20 ２ " can be replaced with ".

[0191] The learning process is executed, for example, by a control unit of an information processing device (for example, the control unit 13 of the server 10 or the control unit 23 of the terminal device 20). The learning process may be executed in parallel with the learning data generation process described above and / or the event notification process described below. Of course, the learning data generation process may be executed regardless of the execution timing of the learning process and / or the event notification process. The learning process will be described below with reference to the flowchart of FIG. 12.

[0192] First, an acquisition unit of the information processing device (for example, the acquisition unit 131 of the server 10 or the acquisition unit 231 of the terminal device 20) acquires learning data for learning / relearning a learning model (step S201). The learning data may be, for example, learning data generated in a learning data generation process (information that associates event information with response information).

[0193] Next, the learning unit of the information processing device (for example, the learning unit 137 of the server 10 or the learning unit 237 of the terminal device 20) executes learning / relearning of the learning model based on the learning data (step S202).

[0194] Then, the storage control unit of the information processing device (for example, the storage unit 136 of the server 10 or the storage control unit 236 of the terminal device 20) stores the learning model learned in step S202 in a memory unit.

[0195] When the saving is completed, the control unit of the information processing device ends the learning process.

[0196] <3-3. Event Notification Processing> Next, the event notification processing will be described.

[0197] 13 is a flowchart showing an event notification process. The event notification process is a process for notifying a user who is watching or listening to the first content about an event that has occurred around the user. Note that not all of the processes shown in FIG. 13 are necessarily essential processes for this embodiment. In other words, each process shown in FIG. 13 can be performed independently.

[0198] The first content is typically immersive content such as a VR game or a metaverse. However, the first content is not limited to immersive content. The first content may be XR content other than immersive content (VR content, AR content, or MR content). Furthermore, the first content is not limited to XR content. The first content may be video content, still image content, text content, or sound content such as music.

[0199] In the following description, a user who views or listens to the first content is referred to as a user U. In the following description, it is assumed that one information processing device executes the event notification process. Here, the information processing device that executes the event notification process may be the server 10 or the terminal device 20.

[0200] It should be noted that multiple information processing devices may cooperate to execute the event notification process. For example, one server 10 may execute part of the event notification process, and another server 10 may execute other parts of the event notification process. Also, for example, one terminal device 20 may execute part of the event notification process, and another terminal device 20 may execute other parts of the event notification process. Also, for example, one server 10 may execute part of the event notification process, and one terminal device 20 may execute other parts of the event notification process. Of course, three or more information processing devices may cooperate to execute the event notification process. For example, three or more information processing devices selected from one or more servers 10 and one or more terminal devices 20 may execute the event notification process.

[0201] When multiple information processing devices cooperate to execute the learning data generation process, the term "information processing device" described below can be replaced with "information processing system 1."

[0202] Furthermore, when multiple information processing devices cooperate to execute the learning data generation process, the description of "information processing device" below can be replaced with "Nth information processing device" as appropriate, where N is an arbitrary integer. For example, when multiple information processing devices cooperate to execute the learning data generation process, the description of "information processing device" below can be replaced with "first information processing device" or "second information processing device" as appropriate.

[0203] Furthermore, the description of "information processing device" below can be replaced with "server 10" or "terminal device 20." For example, if one or more servers 10 and one or more terminal devices 20 cooperate to execute the learning data generation process, the description of "information processing device" below can be replaced with "one or more servers 10" or "one or more terminal devices 20" as appropriate.

[0204] In addition, the description of "information processing device" below refers to "server 10 Ｎ " or "Terminal device 20 Ｍ " where N and M are any integers. For example, if two servers 10 cooperate to execute the learning data generation process, the description of "information processing device" below may be appropriately changed to "server 10 １ " or "Server 10 ２ Furthermore, for example, if two terminal devices 20 cooperate to execute the learning data generation process, the description of "information processing device" below can be appropriately replaced with "terminal device 20" １ " or "Terminal device 20 ２ For example, if two servers 10 and two terminal devices 20 cooperate to execute the learning data generation process, the description of "information processing device" below can be appropriately replaced with "server 10" １ ", "Server 10 ２ ", "Terminal device 20 １ " or "Terminal device 20 ２ " can be replaced with ".

[0205] The event notification process is executed, for example, by a control unit of an information processing device (for example, the control unit 13 of the server 10 or the control unit 23 of the terminal device 20). Note that the event notification process may be executed in parallel with the aforementioned learning data generation process and / or the aforementioned learning process. Of course, the learning data generation process may be executed regardless of the execution timing of the learning process and / or the event notification process. The event notification process will be described below with reference to the flowchart of FIG. 13.

[0206] First, a detection unit of the information processing device (e.g., the detection unit 135 of the server 10 or the detection unit 235 of the terminal device 20) executes a process for detecting an event outside the first content that has occurred around a user who is watching or listening to the first content (step S301). An event outside the first content is, for example, a sound (e.g., a human voice) that has occurred around the user U. For example, an event outside the first content is an event outside the VR content that has occurred around the user U who is watching or listening to the VR content on a head-mounted display.

[0207] Then, the control unit of the information processing device determines whether an event has been detected in step S301 (step S302).

[0208] The processes of steps S301 and S302 may be performed while the aforementioned learning data generation process and / or the aforementioned learning process are being executed. That is, the collection of learning data and / or the learning / relearning of the learning model may be performed while the user U is viewing the first content or using the device. In this case, the processes of steps S101 and / or S202 may be common processes with the processes of steps S301 and / or S302 of the event notification process.

[0209] If an event is not detected (step S302: No), the control unit returns the process to step S301. If an event is detected (step S302: Yes), an acquisition unit of the information processing device (e.g., the acquisition unit 131 of the server 10 or the acquisition unit 231 of the terminal device 20) acquires information about the event detected in step S301 as event information (step S303). For example, the acquisition unit acquires information about an event outside the VR content that has occurred around the user U who is watching or listening to the VR content on the head-mounted display.

[0210] The acquisition unit may acquire, as the event information, sensor information when the detection unit detects an event. The acquisition unit may also acquire, as the event information, information that associates information indicating the content of the event detected by the event detection unit (e.g., information indicating the type of event, such as "human voice," "operation sound of a home appliance," or "alarm sound") with the sensor information when the event was detected.

[0211] Note that the information processing device that detects an event (the information processing device having a detection unit) and the information processing device that acquires the event information (the information processing device having an acquisition unit) may be the same device or different devices. When the information processing device that detects an event and the information processing device that acquires the event information are the same device, the operation of the information processing device to detect an event can be considered as the operation to acquire the event information.

[0212] Next, a determination unit of the information processing device (for example, the determination unit 132 of the server 10 or the determination unit 232 of the terminal device 20) determines whether or not it is necessary to notify the user U about the event detected in step S301 (step S304). For example, the determination unit determines whether or not it is necessary for the user U to respond to the event detected in step S301.

[0213] Here, the determination unit may determine whether or not a notification is necessary using a learning model that has learned the behavior of the user U. The learning model may be a model that has learned the relationship between events that occur around the user U and the user U's responses to those events. For example, the learning model may be a learning model learned through the above-described learning process. Learning also includes relearning.

[0214] The learning model is not limited to a learning model that has learned the behavior of user U. The learning model may also be a learning model that has learned the behavior of one or more other users. The learning model may also be a model that has learned the relationship between events that occur around one or more other users and the responses of one or more other users to the events. For example, the learning model may be a learning model that has been learned through the above-mentioned learning process. Learning also includes relearning.

[0215] As described above, the information on events outside the first content may include information on sounds (acoustic event information) occurring around the user U who is watching or listening to the first content. In this case, the learning model may be a model that learns the relationship between sounds and responses of users (user U and / or one or more other users) to the sounds.

[0216] Furthermore, the determination by the determination unit as to whether or not a notification is necessary is not limited to a determination using a learning model that has learned the behavior of user U (or one or more other users). For example, assume that information about an event outside the first content includes human voice information. In this case, the determination unit may determine whether or not a notification is necessary based on the context of the voice information. For example, the discrimination unit may determine whether or not a notification is necessary for user U using a learning model that determines whether or not a notification is necessary for user U based on the context of the voice information. This context-based determination process will be described in detail later.

[0217] If it is determined that notification to the user U is not necessary (step S304: No), the control unit of the information processing device returns the process to step S301.

[0218] If it is determined that notification to the user U is necessary (step S304: Yes), the generation unit of the information processing device (for example, the generation unit 134 of the server 10 or the generation unit 234 of the terminal device 20) generates content for notification to the user U based on the event information (step S305). In the following description, the content for notification to the user U is referred to as second content.

[0219] Here, the generation unit may generate, as the second content, content corresponding to the details of the first content. For example, the generation unit may generate, as the second content, content in which a character corresponding to the details of the first content makes a notification. The character may be a character appearing in the first content, or may be a character that fits the concept of the first content (e.g., a butler-like fictional character in the case of content depicting a medieval aristocratic society). The process of generating the second content will be described in detail later.

[0220] As described above, multiple information processing devices may cooperate to execute the event notification process. For example, the information processing device that executes the second content generation process (step S305) of the event notification process and the information processing device that executes other processes of the event notification process (e.g., at least one step of steps S301 to S304 and step S306) may be different devices. For example, the second content generation process of the event notification process may be executed by one or more servers 10, and the other processes may be executed by one or more terminal devices 20.

[0221] Next, the notification control unit of the information processing device (for example, the notification control unit 133 of the server 10 or the notification control unit 233 of the terminal device 20) performs processing related to notification to the user U (step S306). For example, the notification control unit controls the output unit to notify the user U about the event. Note that if the information processing device that executes step S306 is not an output device having an output unit that outputs to the user U, the notification control unit may control the output device to notify the user U. For example, the information processing device may send a control signal to the output device to notify the user U about the event.

[0222] Note that the notification control unit may provide a notification regarding an event within the first content in order to enhance the user experience (e.g., to prevent the user U from losing immersion in the first content). For example, the notification control unit may perform processing for playing the second content generated in step S305 within the first content. For example, the notification control unit may control the output unit to play the second content within the first content. If the information processing device is not an output device having an output unit that outputs to the user, the notification control unit may cause the output device to play the second content within the first content.

[0223] When the notification is completed, the control unit of the information processing device returns the process to step S301.

[0224] <3-4. Determination Process of Notification Necessity Based on Context> Next, the determination process of step S304 will be described in detail.

[0225] Here, a process for determining whether or not to notify the user U when voice information is acquired as event information will be described. As described above, the information processing device may determine whether or not to notify the user U based on the context of the voice information. The determination of whether or not to notify the user U is, for example, a determination of whether or not the user U needs to respond to the event.

[0226] The determination process shown below is merely an example, and the determination process in step S304 is not limited to the process shown below.

[0227] First, the determination unit of the information processing device converts acquired voice information (e.g., sensor information detected by a microphone) into a text string using a voice recognition process. For example, the determination unit of the information processing device converts the acquired voice information (e.g., sensor information detected by a microphone) into a text string using a voice recognition process. Here, the determination unit of the information processing device may convert the acquired voice information (e.g., sensor information detected by a microphone) into a text string using a natural language learning model.

[0228] Then, the determination unit inputs the text string into a learning model for determination, thereby determining whether or not a notification to the user U is required. Here, the determination unit may use a transformer-based encoder model as the learning model for determination. The transformer-based encoder model may be, for example, BERT (Bidirectional Encoder Representations from Transformers) or RoBERTa (Robustly Optimized BERT pre-training Approach). Of course, the determination unit may determine whether or not a notification to the user U is required using another transformer-based encoder model.

[0229] A learning unit of the information processing device (for example, the learning unit 137 of the server 10 or the learning unit 237 of the terminal device 20) may acquire a learning model for discrimination by fine-tuning a pre-trained model based on collected data. The information processing device that trains / relearns the learning model and the information processing device that uses the learning model to determine whether or not a notification is necessary may be the same device or different devices.

[0230] During fine tuning, for example, speech recognition results with the sentence-initial symbol "CLS" and the sentence-final symbol "SEP" added are given, as shown below.

[0231] [CLS] Hey [SEP]

[0232] The learning unit of the information processing device trains the learning model by passing the embedding vector of the output part corresponding to this "CLS" through a linear layer and applying a sigmoid activation function so that it can distinguish between 0 (no notification / response required) and 1 (notification / response required).

[0233] In addition to the above, there are various other types of encoder models, such as DistillBERT, XLM, or ELECTRA. These models can also be used to determine whether or not a notification is required, similar to the above.

[0234] Furthermore, the determination unit of the information processing device may use a model having a decoder such as a Generative Pretrained Transformer (GPT) or a Text-to-Text Transfer Transformer (T5) as a learning model for discrimination. In this case, the determination unit of the information processing device may input, for example, the following to the learning model for discrimination. Then, the determination unit may obtain an output from the learning model for discrimination as a discrimination result.

[0235] (Example 1) Classify the following sentences as "Response required" or "No response required" Input: Come here for a moment Output: Response required

[0236] (Example 2) Classify the following sentences as "Response required" or "No response required." Input: This room is cold. Output: No response required.

[0237] The determination unit of the information processing device may perform speaker identification. Then, the determination unit of the information processing device may determine whether a response is required based on the result of the speaker identification. When performing speaker identification, the relationship with the user determined from the speaker identification result (e.g., "family member," "roommate," "acquaintance," "stranger," etc.) may be added as a one-hot vector to an input layer immediately before outputting the determination result.

[0238] When the learning model for discrimination is a decoder-type model, the determination unit of the information processing device may modify the input to the model as follows.

[0239] Input: (acquaintance) Hey!

[0240] When the learning model for discrimination is a decoder-type model, the determination unit of the information processing device may provide, for example, the following as a "prompt" for input to the model. This enables the determination unit to control the output from the model.

[0241] (Example) Please be aware of the following points when making your decision: Do not be rude to the person who called out to you. Generally, you should respond to personal calls.

[0242] <3-5. Processing for Generating Second Content> Next, the processing for generating second content in step S305 will be described.

[0243] The generation unit of the information processing device may generate the second content by, for example, selecting content corresponding to the detected event from a plurality of pre-prepared contents. Alternatively, the generation unit of the information processing device may select content data corresponding to the detected event from a plurality of pre-prepared content generation data, and generate the second content based on the selected content generation data.

[0244] In the following description, data of content prepared in advance or data for generating content will be referred to as additional content data.

[0245] Here, the information processing device may store a plurality of additional content data in a storage unit in advance. Then, the generation unit of the information processing device may select the additional content data from the plurality of content data according to the event information acquired in step S303. Furthermore, the generation unit may acquire the additional content data according to the event information acquired in step S303 from another information processing device.

[0246] Fig. 14 is a diagram for explaining additional content data. More specifically, Fig. 14 is a diagram for explaining a sound event and additional content data associated with the sound event.

[0247] In the example of Fig. 14, a knocking sound event is associated with content data in which a butler walks over and says, "Someone's here." Also in the example of Fig. 14, a voice calling out (e.g., "Heyyy") event is associated with content data in which a character in the game approaches and calls out to the player. Also in the example of Fig. 14, a chime sound event is associated with content data in which a door appears in the sky and a chime rings. Also in the example of Fig. 14, a home appliance end sound event (e.g., a microwave end sound) is associated with content data in which a robot approaches and says, "It's done."

[0248] The generation unit acquires the additional content data as data of the second content or data for generating the second content, and generates the second content based on the additional content data, for example, by converting the additional content data into data corresponding to the first content.

[0249] The notification control unit of the information processing device performs output control to output the second content generated in this manner to the user. For example, the notification control unit performs output control to play the second content generated in step S305 within the first content.

[0250] When the second content is played within the first content, it is expected that it will be difficult to distinguish the second content from normal content. Therefore, the information processing device may use a specific character for the notification content (second content). Furthermore, the information processing device may perform output control, such as displaying a specific image in the corner of the field of view or playing specific music, in accordance with the output of the notification content (second content).

[0251] In the example of FIG. 14 , additional content data corresponding to an event is prepared in advance, but the generation unit of the information processing device may change this additional content data depending on the situation in the first content (e.g., the situation in the game). For example, if a location in the first content (e.g., a game stage) is cold, the generation unit may dress the notification character in warm clothing. Furthermore, if the location in the first content is a party venue, the generation unit may dress the notification character in clothing appropriate for the party. Furthermore, if the location in the first content is a live music venue, the generation unit may feature a person such as a musician as the notification character. The generation unit may then use the additional content changed depending on the situation in the first content as the second content.

[0252] The information processing device may also perform speaker identification when detecting an acoustic event. In this case, the generation unit of the information processing device may generate the second content based on, for example, the result of the speaker identification. In this case, the information processing device may generate the second content, for example, as follows.

[0253] First, the information processing device acquires setting information in which a character and / or voice corresponding to a family member (e.g., the mother) is set. The setting information may be information set by the user U on a setting screen for the first content (e.g., a game setting screen). The setting information may also be information that the information processing device has previously stored as default settings. The character may also be a character corresponding to the first content. For example, the character may be a character that appears in the first content (e.g., a game). Then, when a family member (e.g., the mother) calls out to the information processing device in the outside world, the generation unit converts the call into a call from the set character. The generation unit acquires the content generated by the conversion process as the second content.

[0254] The information processing device may also perform voice recognition when detecting an acoustic event. In this case, the generation unit of the information processing device may generate second content based on the results of the voice recognition. For example, when a call is received from the outside world, the generation unit converts the content of the call into a call from a character. In this case, the character may correspond to the first content. For example, the character may be a character appearing in the first content (e.g., a game).

[0255] When generating the second content based on the results of the voice recognition, the generation unit may generate content in which the content and / or speech style of the actual call is reproduced by a character as is as the second content. Alternatively, the generation unit may generate content in which the content and / or speech style of the call is changed as the second content. For example, the generation unit may generate content in which the content and / or speech style of the call is changed to content and / or speech style corresponding to the first content and / or character. For example, suppose the actual content of the call is "Hey!" In this case, the generation unit may generate content in which a character says, "May I have a moment?" as the second content.

[0256] <<4. Modifications>> The above-described embodiment is merely an example, and various modifications and applications are possible.

[0257] <4-1. Modifications Related to Device Configuration> For example, in the above-described embodiment, a head-mounted display is exemplified as an output device (terminal device 20) that outputs to a user. However, the output device is not limited to a head-mounted display. The output device may be a sound output device without a display, such as headphones or earphones.

[0258] Even if the output device that outputs to the user is a sound output device that does not have a display, the above-described learning data generation process, learning process, and event notification process (event detection process, process for determining whether or not a notification is necessary, and process for generating second content) are applicable. For example, the second content generated by the generation unit of the information processing device may be sound content.

[0259] Furthermore, the output device (terminal device 20) that outputs to the user is not limited to a device that can be worn by the user (wearable device), such as a head-mounted display, headphones, or earphones. It may also be a device that is installed and used on a structure or a mobile object, such as a television, a car navigation system, or various operation panels. The output device may also be a mobile terminal, such as a smartphone, a mobile phone, a personal computer, a music player, or a portable television.

[0260] 4-2. Modifications Related to Events In the above-described embodiment, various processes have been described using acoustic events as an example. For example, in the above-described learning data generation process (steps S101 to S106), the event for which learning data is generated is an acoustic event, as an example. Also, in the above-described learning process (steps S201 to S203), the event for which learning is performed is an acoustic event, as an example. Also, in the above-described event notification process (steps S301 to S306), the event for which notification is sent to user U is an acoustic event, as an example. However, the events for which notification is sent to user U in this embodiment (at least one of the events for which learning data is generated, the events for which learning is performed, and the events for which notification is sent to user U) are not limited to acoustic events.

[0261] For example, the event may be an event detected based on information about an image of the user's surroundings. For example, the event may be an event of a person approaching. The acquisition unit of the information processing device may acquire information about an image of the user's surroundings captured by a camera or an image sensor as event information (event information used to generate learning data and / or information about an event outside the first content). The image may be a still image or a video.

[0262] Furthermore, the event to be notified to the user U may be an event detected based on an image and sound. For example, the event may be an event in which someone calls out to the user U while waving their hand. The acquisition unit of the information processing device may acquire, as event information, information on an image of the user U's surroundings captured by a camera or an image sensor and information on sounds around the user U captured by a microphone. The image may be a still image or a video.

[0263] The information processing device may perform at least one of a learning data generation process, a learning process, and an event notification process based on the event information (image and / or sound information). Furthermore, the generation unit of the information processing device may perform at least one of an event detection process, a determination process regarding whether or not a notification is necessary, and a second content generation process based on the event information (image and / or sound information).

[0264] Furthermore, the event detected by the information processing device may be an event that occurs outside the room where the user is present. For example, the event may be a call to the user from outside the room where the user is present. This also applies when the output device that outputs to the user is a wearable device such as a head-mounted display, headphones, or earphones. Even if an event occurs outside the room where the user is present, an event that can be detected by the user when no content is being output is an event that occurred around the user.

[0265] Note that the event detected by the information processing device is not limited to an event occurring in the real space around the user. For example, the event detected by the information processing device may be an event on a network. For example, the event may be the arrival of a message at an electronic device (e.g., a server 10 such as a mail server and / or a terminal device 20 owned by the user). Here, the message is a message sent and / or received via a network. The message may be an email, an SMS (Short Message Service) message, an MMS (Multimedia Messaging Service) message, or an SNS message. The information processing device may perform at least one of a learning data generation process, a learning process, and an event notification process based on information of the arrived message (e.g., at least one of the message content, subject, sender, destination, and CC (Carbon Copy)).

[0266] Alternatively, the event detected by the information processing device may be an event that occurs outside the space surrounding the user. For example, the event detected by the information processing device may be an event that the user cannot detect even if there is no content output (for example, an event that occurs in a room away from the room where the user is present).

[0267] 4-3. Modifications Related to Event Detection In the above-described embodiment (for example, step S101 or step S301), the information processing device detects an event occurring around the user as an event outside the first content. Here, the event occurring around the user is not limited to an event occurring in a room where the user is present.

[0268] For example, the information processing device may detect an event occurring outside the room where the user is present as an event outside the first content. In this case, the information processing device (or one or more sensors included in the sensor unit) that detects the event may be located outside the room where the user is present. Of course, the information processing device (or one or more sensors included in the sensor unit) that detects the event may be installed in the room where the user is present.

[0269] Alternatively, the information processing device may detect an event based on information from another information processing device in the house (or another sensor in the house) via a home network, which allows the information processing device to detect an event using information on the status of a home appliance such as a washing machine (for example, information on the end-of-operation sound and / or abnormal sound of the home appliance) as sensor information.

[0270] Furthermore, the sensor information is not limited to information on the status of home appliances. For example, the information processing device may detect an event using, as sensor information, sound information picked up by a microphone of another information processing device in the house and / or an image captured by a camera / image sensor of another information processing device in the house. For example, if an abnormal situation is captured in a camera image of an intercom at the front door, the information processing device may detect the content of the abnormal situation as an event.

[0271] In this case, the information processing device may generate second content for the user U to confirm the front door. Then, the information processing device may output the second content to the user. For example, assume that a pseudo television is presented in a space (first content) in which the user U is immersed using a head-mounted display. The information processing device may display an image of the front door (second content) on this pseudo television. In this case, the second content may be a news program-style content announcing that a suspicious person has been spotted. Of course, the second content may simply be an image of the front door that has been converted so that it can be displayed on the pseudo television.

[0272] <4-4. Modifications Related to Second Content> In the above-described embodiment (e.g., step S305), the information processing device generates content corresponding to the details of the first content as the second content for notification to the user. However, the second content is not limited to content corresponding to the details of the first content. The second content may simply be content corresponding to the type / format of the first content.

[0273] For example, if the first content is XR content (e.g., VR content), the information processing device may generate the XR content (e.g., VR content) as the second content. Furthermore, for example, if the first content is video / still image content, the information processing device may generate video / still image content as the second content. Furthermore, for example, if the first content is text content, the information processing device may generate text content as the second content. Furthermore, for example, if the first content is sound content (e.g., music), the information processing device may generate sound content as the second content.

[0274] Furthermore, the second content is not limited to content corresponding to the type / format of the first content. The second content does not have to be content corresponding to the type / format of the first content. For example, suppose the first content is content accompanied by visual information (e.g., XR content such as VR content, video / still image content, or text content). In this case, the information processing device may generate sound content as the second content. Furthermore, suppose the first content is content not accompanied by visual information (e.g., sound content). In this case, the information processing device may generate content accompanied by visual information (e.g., XR content such as VR content, video / still image content, or text content) as the second content.

[0275] <4-5. Modifications Related to Notification to User> In the above-described embodiment (for example, step S306), the information processing device performs control to play back, within the first content, the second content for notification to the user U. However, the information processing device does not necessarily have to play back the second content within the first content.

[0276] For example, the information processing device may control an output unit (e.g., a display and / or a speaker) or an output device (e.g., another terminal device 20) so that the second content is output independently of the first content. For example, the information processing device may control the output unit or the output device so that the second content is output superimposed on the output of the first content. Furthermore, for example, the information processing device may control the output unit or the output device so that the second content is output while the output of the first content is suppressed / paused / muted.

[0277] The information processing device may also control an output unit / output device other than the output unit / output device that outputs the first content so that the second content is output by the other output unit / output device.

[0278] Furthermore, in the above-described embodiment (e.g., step S306), the information processing device outputs the second content as a notification to the user U. However, the notification to the user U is not limited to outputting the second content. For example, the information processing device may output the detected event to the user U as it is.

[0279] For example, it is assumed that the event information acquired in step S303 includes information about a sound generated around the user U. In this case, if the sound generated around the user U satisfies a predetermined criterion, the information processing device may perform output control so that the sound generated around the user U is output to the user U as is.

[0280] More specifically, in step S304, the information processing device may determine the certainty of the determination result of whether or not a notification is necessary, in addition to determining whether or not a notification is necessary. For example, the learning model may be configured to output the certainty of the determination result of whether or not a notification is necessary, in addition to determining whether or not a notification is necessary. The information processing device may then determine the certainty based on the output of the learning model. In step S306, if the certainty is equal to or less than a predetermined threshold, the information processing device controls output so that sounds generated around the user U are output to the user U as is. The information processing device may output this sound together with the second content, or may output the sound without outputting the second content. When the determination result is uncertain, the information processing device can leave it to the user U to determine whether or not a response is necessary.

[0281] Furthermore, in step S304, the information processing device may determine the importance of the event in addition to determining whether or not a notification is required to the user U. For example, the learning model may be configured to output the importance of the event in addition to determining whether or not a notification is required. For example, in the case of an event that is important to the user U, such as a fire, the learning model may be configured to output a large value as the importance. The information processing device may then determine the importance based on the output of the learning model. In step S306, if the importance is equal to or greater than a predetermined threshold, the information processing device controls output so that sounds generated around the user U are output to the user U as is. The information processing device may output this sound together with the second content, or may output this sound without outputting the second content. In this way, when the event is determined to be important, the information processing device can reliably inform the user U of the importance of the event.

[0282] This can also be applied to events other than sound. For example, assume that the event information includes sensor information (e.g., information about an image (still image / video)). In this case, if the event indicated by the event information satisfies a predetermined criterion, the information processing device may output the sensor information to the user U as is.

[0283] More specifically, in step S304, the information processing device may determine the certainty of the determination result of whether or not a notification is necessary, in addition to determining whether or not a notification is necessary. For example, the learning model may be configured to output the certainty of the determination result of whether or not a notification is necessary, in addition to determining whether or not a notification is necessary. The information processing device may then determine the certainty based on the output of the learning model. In step S306, if the certainty is equal to or less than a predetermined threshold, the information processing device outputs the sensor information (e.g., image (still image / video) information) included in the event information to the user U as is. This allows the information processing device to leave the determination of whether or not a response is necessary to the user U when the determination result is uncertain.

[0284] Furthermore, in step S304, the information processing device may determine the importance of the event in addition to determining whether or not a notification is required to the user U. For example, the learning model may be configured to output the importance of the event in addition to determining whether or not a notification is required. For example, in the case of an event that is important to the user U, such as a fire, the learning model may be configured to output a large value as the importance. The information processing device may then determine the importance based on the output of the learning model. In step S306, if the importance is equal to or greater than a predetermined threshold, the information processing device outputs the sensor information (e.g., image (still image / video) information) included in the event information to the user U as is. This allows the information processing device to reliably inform the user U of the importance of the event when the event is determined to be important.

[0285] 4-6. Other Modifications The server 10 or the control device that controls the terminal device 20 of this embodiment may be realized by a dedicated computer system or a general-purpose computer system.

[0286] For example, a communication program for executing the above-described operations may be stored and distributed on a computer-readable recording medium such as an optical disk, a semiconductor memory, a magnetic tape, or a flexible disk. Then, for example, the program may be installed on a computer and the above-described processing may be executed to configure a control device. In this case, the control device may be a device (e.g., a personal computer) external to the server 10 or the terminal device 20. Alternatively, the control device may be a device (e.g., a control unit 13 or a control unit 23) internal to the server 10 or the terminal device 20.

[0287] The communication program may also be stored in a disk device provided in a server device on a network such as the Internet, and may be downloaded to a computer. The above-described functions may also be realized by a combination of an operating system (OS) and application software. In this case, the components other than the OS may be stored on a medium and distributed, or may be stored in a server device and downloaded to a computer.

[0288] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0289] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0290] The above-described embodiments can be combined as appropriate within the scope of the present invention without causing any inconsistency in the processing content. The order of the steps shown in the flowcharts of the above-described embodiments can be changed as appropriate.

[0291] Furthermore, for example, the present embodiment can be implemented as any configuration constituting an apparatus or system. For example, the present embodiment can be implemented as a processor as a system LSI (Large Scale Integration), a module using multiple processors, a unit using multiple modules, or a set in which a unit further has additional functions. In other words, the present embodiment can also be implemented as a part of the configuration of an apparatus.

[0292] In this embodiment, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

[0293] Furthermore, for example, this embodiment can have a cloud computing configuration in which one function is shared and processed jointly by a plurality of devices via a network.

[0294] <<5. Conclusion>> As described above, according to one embodiment of the present disclosure, the information processing device / information processing system 1 acquires information about an event that occurs outside the first content and occurs around the user U who is watching or listening to the first content. Then, the information processing device / information processing system 1 determines whether or not it is necessary to notify the user U about the event based on the event information. Then, the information processing device / information processing system 1 performs processing related to notifying the user U based on the result of the determination. As a result, the user U is not notified of events that require a low response, and is notified of events that require a high response. As a result, the information processing device can provide the user with a high user experience without significantly impairing convenience.

[0295] Furthermore, when a notification to user U is necessary, the information processing device / information processing system 1 performs processing for providing the notification within the first content. For example, the information processing device / information processing system 1 generates second content for providing notification to user U based on event information. Then, the information processing device / information processing system 1 performs processing for playing the second content within the first content. This allows the information processing device / information processing system to provide notification of an event to user U without significantly impairing the user experience (for example, user U's immersion in the first content).

[0296] Furthermore, the information processing device / information processing system 1 generates, as the second content, content that corresponds to the details of the first content. For example, the information processing device / information processing system 1 generates, as the second content, content in which a character that corresponds to the details of the first content makes a notification. This allows the information processing device / information processing system to notify the user U about an event while maintaining an even higher user experience.

[0297] Furthermore, the information processing device / information processing system 1 uses a learning model that has learned the behavior of user U or one or more other users to determine whether a notification is necessary (for example, whether a response from user U is necessary). The learning model is, for example, a model that has learned the relationship between an event that occurs around user U or one or more other users and the response of user U or one or more other users to the event. This allows the information processing device / information processing system 1 to accurately determine whether a notification is necessary.

[0298] The information about the event outside the first content may include information about human voices occurring around the user U. In this case, the information processing device / information processing system 1 may determine whether or not a notification is necessary (e.g., whether or not a response from the user U is necessary) based on the context of the information about the human voices. This allows the information processing device / information processing system 1 to accurately determine whether or not a notification is necessary.

[0299] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0300] Furthermore, the effects of each embodiment described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0301] Note that the present technology may also be configured as follows. (1) An information processing device comprising: an acquisition unit that acquires information about an event that occurs outside of a first content around a user who watches or listens to the first content; a determination unit that determines whether or not a notification of the event is necessary to the user based on the event information; and a notification control unit that performs processing related to the notification to the user based on a result of the determination. (2) The information processing device according to (1), wherein the notification control unit performs processing for providing the notification within the first content. (3) The information processing device according to (2), wherein the information processing device comprises: a generation unit that generates second content for providing the notification to the user based on the event information, and the notification control unit performs processing for playing the second content within the first content. (4) The information processing device according to (3), wherein the generation unit generates, as the second content, content corresponding to the content of the first content. (5) The information processing device according to (4), wherein the generation unit generates, as the second content, content in which a character corresponding to the content of the first content provides the notification. (6) The information processing device according to any one of (3) to (5), wherein the information about the event outside the first content includes information about a sound generated around the user. (7) The information processing device according to (6), wherein the sound information includes information about a human voice. (8) The information processing device according to (7), wherein the determination unit determines whether or not a notification to the user is necessary based on a context of the information about the human voice. (9) The information processing device according to any one of (6) to (8), wherein the notification control unit outputs the sound generated around the user to the user as it is if the sound generated around the user satisfies a predetermined criterion. (10) The information processing device according to any one of (6) to (9), wherein the determination unit further determines a certainty of a determination result of whether or not a notification is necessary, and the notification control unit outputs the sound generated around the user to the user as it is if the certainty is equal to or less than a predetermined threshold.(11) The information processing device according to any one of (6) to (10), wherein the determination unit further determines the importance of the event, and the notification control unit outputs the sound generated around the user to the user as is when the importance is equal to or greater than a predetermined threshold. (12) The information processing device according to any one of (1) to (11), wherein the determination unit makes the determination as to whether the notification is necessary using a learning model that has learned the behavior of the user or one or more other users. (13) The information processing device according to (12), wherein the learning model is a model that has learned a relationship between an event that has occurred around the user or the one or more other users and a response of the user or the one or more other users to the event. (14) The information processing device according to (13), wherein the information on the event outside the first content includes information on a sound that has occurred around the user who is watching or listening to the first content, and the learning model is a model that has learned a relationship between the sound and the user's response to the sound. (15) The information processing device according to any one of (1) to (14), wherein the information about the event outside the first content includes information about an image of the user's surroundings captured by a camera or an image sensor. (16) The information processing device according to any one of (1) to (15), wherein the first content is XR (Extended Reality / Cross Reality) content. (17) The information processing device according to (16), wherein the first content is VR (Virtual Reality) content. (18) The information processing device according to (17), wherein the acquisition unit acquires information about the event outside the VR content that has occurred around the user who watches or listens to the VR content on a head-mounted display. (19) An information processing method, comprising: acquiring information about an event outside the first content that has occurred around the user who watches or listens to the first content; determining whether or not to notify the user of the event based on the event information; and performing processing related to the notification to the user based on a result of the determination.(20) A program for causing a computer to function as: an acquisition unit that acquires information about an event outside the first content that occurs around a user who is watching or listening to the first content; a judgment unit that determines whether or not the user needs to be notified about the event based on the event information; and a notification control unit that performs processing related to the notification to the user based on the result of the judgment.

[0302] REFERENCE SIGNS LIST 1 Information processing system 10 Server 20 Terminal device 11, 21 Communication unit 12, 22 Storage unit 13, 23 Control unit 24 Input unit 25 Output unit 26 Sensor unit 131, 231 Acquisition unit 132, 232 Determination unit 133, 233 Notification control unit 134, 234 Generation unit 135, 235 Detection unit 136 Storage unit 236 Storage control unit 137, 237 Learning unit U User

Claims

1. An information processing device comprising: an acquisition unit that acquires information about an event outside a first content that occurs around a user who is watching or listening to the first content; a judgment unit that judges whether or not the user needs to be notified about the event based on the event information; and a notification control unit that performs processing regarding the notification to the user based on the result of the judgment.

2. The information processing device according to claim 1, wherein the notification control unit performs processing for making the notification within the first content.

3. An information processing device as described in claim 2, further comprising a generation unit that generates second content for the notification to the user based on information about the event, wherein the notification control unit performs processing to play the second content within the first content.

4. The information processing device according to claim 3, wherein the generation unit generates, as the second content, a content corresponding to the details of the first content.

5. The information processing device according to claim 4, wherein the generation unit generates, as the second content, content in which a character corresponding to the details of the first content makes the notification.

6. The information processing device according to claim 3, wherein the information about the event outside the first content includes information about a sound generated around the user.

7. The information processing device according to claim 6, wherein the sound information includes information of a human voice.

8. The information processing device according to claim 7, wherein the determination unit determines whether or not a notification to the user is required based on a context of the information on the human voice.

9. The information processing device according to claim 6, wherein the notification control unit outputs the sound generated around the user to the user as is when the sound satisfies a predetermined criterion.

10. The information processing device of claim 6, wherein the determination unit further determines the certainty of the determination result of whether or not a notification is necessary, and the notification control unit outputs the sound generated around the user to the user as is if the certainty is equal to or lower than a predetermined threshold value.

11. The information processing device of claim 6, wherein the determination unit further determines the importance of the event, and the notification control unit outputs the sound generated around the user to the user as is if the importance is equal to or greater than a predetermined threshold.

12. The information processing device according to claim 1, wherein the determination unit determines whether or not the notification is necessary using a learning model that has learned the behavior of the user or one or more other users.

13. The information processing device according to claim 12, wherein the learning model is a model that learns the relationship between events occurring around the user or the one or more other users and responses of the user or the one or more other users to the events.

14. The information processing device of claim 13, wherein the information on the event outside the first content includes information on sounds occurring around the user who is watching or listening to the first content, and the learning model is a model that learns the relationship between the sounds and the user's response to the sounds.

15. The information processing device according to claim 1, wherein the information about the event outside the first content includes information about an image of the user's surroundings captured by a camera or an image sensor.

16. The information processing device according to claim 1, wherein the first content is an XR (Extended Reality / Cross Reality) content.

17. The information processing device according to claim 16, wherein the first content is a VR (Virtual Reality) content.

18. The information processing device according to claim 17, wherein the acquisition unit acquires information about the event occurring outside the VR content around the user who is watching or listening to the VR content on a head-mounted display.

19. An information processing method comprising: acquiring information about an event occurring outside a first content around a user who is watching or listening to the first content; determining whether or not to notify the user about the event based on the event information; and performing processing related to the notification to the user based on the result of the determination.

20. A program for causing a computer to function as: an acquisition unit that acquires information about events outside of a first content that occur around a user who is watching or listening to the first content; a judgment unit that makes a judgment as to whether or not the user needs to be notified about the event based on the event information; and a notification control unit that performs processing regarding the notification to the user based on the result of the judgment.

Images