Information processing device, method, and program

Abstract

To provide an information processing apparatus and a method by which a speech that a user intends can be easily obtained in a virtual space.SOLUTION: Information processing apparatuses (2, 3) output speeches in a virtual space in which an avatar representing a user moves, from an output unit for the user. The information processing apparatuses include: input units (23, 33) for receiving user operation for setting a specific object to a predetermined voice processing target, in objects arranged in the virtual space; and control units (21, 31) for causing the output unit to output a speech obtained by applying image processing to the voice from the specific object in the virtual space based on the user operation.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program for performing information processing related to a virtual space. Background Art

[0002] Cited Document 1 discloses an image display method for displaying a computer graphics image representing a three-dimensional virtual space. This image display method sets an element that changes the depth of field for a camera installed in the three-dimensional virtual space, and calculates a blur amount for an object based on the set element that changes the depth of field and the object to be displayed in the computer graphics image. Thus, the method performs image processing to express predetermined blur on objects not included in the depth of field range according to the depth of field set in the three-dimensional virtual space, thereby enabling expression similar to that of a live-action image captured by a camera. Prior Art Documents Patent Documents

[0003] Patent Document 1 Japanese Unexamined Patent Publication No. 2000-322594 Summary of the Invention Problem to be Solved by the Invention

[0004] The present disclosure provides an information processing apparatus and method that can make it easier to obtain audio matching a user's intention in a virtual space. Means for Solving the Problem

[0005] The information processing device in this disclosure is an information processing device that outputs audio from a user output unit for a virtual space in which an avatar representing a user moves, and comprises an input unit that receives a user operation to set a specific object as the target of predetermined audio processing among the objects placed in the virtual space, and a control unit that outputs audio from an output unit in which audio processing has been applied to the audio from the specific object in the virtual space based on the user operation.

[0006] The information processing method in this disclosure is an information processing method that causes an output unit for a user to output sound in a virtual space in which an avatar representing a user moves, and includes the steps of: receiving a user operation to set an object to be subjected to predetermined sound processing in an object placed in the virtual space; and, based on the user operation, causing an output unit to output sound that has been processed by sound processing applied to the sound from the object set in the virtual space. [Effects of the Invention]

[0007] According to the information processing device and method disclosed herein, it is possible to easily obtain voices in a virtual space that align with the user's intentions. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram illustrating the virtual space system according to Embodiment 1 of this disclosure. [Figure 2] A diagram illustrating the configuration of a virtual space server according to Embodiment 1. [Figure 3] A diagram illustrating the configuration of a personal terminal according to Embodiment 1. [Figure 4] A flowchart illustrating the registration process for metaverse travel in a virtual space system. [Figure 5] A diagram illustrating a registration screen for metaverse travel in a virtual space system. [Figure 6] A diagram to explain the display of avatars inside and outside the group. [Figure 7]A flowchart illustrating actions during metaverse travel in a virtual space system. [Figure 8] A diagram illustrating live view in a virtual space system. [Figure 9] A flowchart illustrating group shooting processing in a virtual space system. [Figure 10] Figure showing an example of display during group shooting processing. [Figure 11] A diagram illustrating the individual photography service for the virtual space system. [Figure 12] Diagram illustrating the warp travel service of Embodiment 2 [Figure 13] Flowchart illustrating the operation of the warp travel service in Embodiment 2 [Modes for carrying out the invention]

[0009] The embodiments will be described in detail below, with reference to the drawings as appropriate. However, detailed explanations of already well-known matters and redundant explanations of substantially identical configurations may be omitted. The accompanying drawings and the following description are provided so that those skilled in the art can fully understand this disclosure, and are not intended to limit the subject matter described in the claims.

[0010] (Embodiment 1) 1. Structure 1-1. System Overview A virtual space system according to Embodiment 1 of this disclosure will be described with reference to Figure 1.

[0011] The virtual space system 1 according to this embodiment comprises a virtual space server 2 and a plurality of personal terminals 3, as shown in Figure 1, for example. This system 1 is applied to a virtual space 10 such as a metaverse, where a user 5 obtains a desired experience through their avatar 15. The avatar 15 is an example of an object that represents an individual, such as the user themselves, in the virtual space 10.

[0012] In the present system 10, the virtual space server 2 and each individual terminal 3 are connected to enable data communication via a communication network 4 such as the Internet. The virtual space server 2 executes various types of information processing (for example, image processing and audio processing) for realizing each user's experience in the virtual space 10. Each individual terminal 3 is used, for example, for enabling different users 5 to operate their own avatars 15 in the virtual space 10 or view and experience the virtual space. The user 5 may use a head-mounted display (HMD), a hand controller, and the like connected to the individual terminal 3 via communication, or may experience the virtual space 10 by viewing content via the individual terminal 3 alone. Furthermore, an HMD or the like may configure the individual terminal 3.

[0013] 1-2. Server Configuration The configuration of the virtual space server 2 in the present embodiment will be described with reference to Figure 2.

[0014] Figure 2 is a diagram illustrating an example of the configuration of the virtual space server 2. The virtual space server 2 is an example of an information processing device configured by one or more computers, for example. The virtual space server 2 illustrated in Figure 2 includes a control unit 21, a storage unit 22, and a communication unit 23.

[0015] The control unit 21 includes, for example, a CPU, a GPU, and the like that implement predetermined functions in cooperation with software, and controls the operation of the virtual space server 2. The control unit 21 reads out data and programs stored in the storage unit 22, performs various arithmetic processes, and implements various functions.

[0016] For example, as a functional configuration of the virtual space server 2, the control unit 21 includes an image processing unit 21a and an audio processing unit 21b. The image processing unit 21a executes various types of image processing for generating various images in the virtual space 10. The audio processing unit 21b executes various types of audio processing for generating various audio in the virtual space 10. The control unit 21 executes, for example, a program including a group of instructions for implementing each of the above functions. The above program may be provided from a communication network such as the Internet, or may be stored in a portable recording medium.

[0017] Furthermore, the control unit 21 may be a dedicated electronic circuit or a hardware circuit such as a reconfigurable electronic circuit designed to realize each of the above functions. The control unit 21 may be composed of various semiconductor integrated circuits such as a CPU, MPU, GPU, GPGPU, TPU, microcontroller, DSP, FPGA, and ASIC. Also, the image processing unit 21a and the audio processing unit 21b may each be composed of dedicated circuits.

[0018] The memory unit 22 is a storage medium that stores the programs and data necessary to realize the functions of the virtual space server 2. As shown in Figure 2, the memory unit 22 includes a storage unit 22a and a temporary storage unit 22b.

[0019] The storage unit 22a stores parameters, data, and control programs for realizing predetermined functions. The storage unit 22a is composed of, for example, an HDD or an SSD. For example, the storage unit 22a stores the above-mentioned programs and various image data.

[0020] The temporary storage unit 22b is composed of RAM such as DRAM or SRAM, and temporarily stores (i.e., holds) data. For example, the temporary storage unit 22b holds image data that is being edited. The temporary storage unit 22b may also function as a work area for the control unit 21, or it may be composed of a storage area in the internal memory of the control unit 21.

[0021] The communication unit 23 is a module (circuit) that connects to an external device according to a predetermined communication standard in wired or wireless communication. The predetermined communication standard includes, for example, IEEE802.3, IEEE802.11a / 11b / 11g / 11ac / 11ax, USB, HDMI (registered trademark), IEEE1394, Wi-Fi, Bluetooth, etc. The communication unit 23 may also connect to a virtual space server 2 on a communication network such as the Internet. The communication unit 23 is an example of an input unit that receives various information from an external device or a communication network.

[0022] 1-3. Configuration of personal devices The configuration of the personal terminal 3 in this embodiment will be explained with reference to Figure 3.

[0023] Figure 3 is a diagram illustrating the configuration of a personal terminal 3. The personal terminal 3 is an example of an information processing device composed of, for example, a personal computer, a smartphone, a tablet device, or a wearable device. The personal terminal 3 illustrated in Figure 3 includes a control unit 31, a storage unit 32, an operation unit 33, a display unit 34, a communication unit 35, a microphone 36, a speaker 37, and a camera 38.

[0024] The control unit 31 includes, for example, a CPU or MPU that works in cooperation with software to realize predetermined functions. The control unit 31 controls, for example, the overall operation of the personal terminal 3. The control unit 31 reads data and programs stored in the storage unit 32, performs various calculations, and realizes various functions.

[0025] The control unit 31 executes a program that includes a set of instructions for realizing each of the above functions. This program may be provided via a communication network such as the Internet, or it may be stored on a portable recording medium. The control unit 31 may also be a dedicated electronic circuit or a hardware circuit such as a reconfigurable electronic circuit designed to realize each of the above functions. The control unit 31 may be composed of various semiconductor integrated circuits such as a CPU, MPU, GPU, GPGPU, TPU, microcontroller, DSP, FPGA, and ASIC.

[0026] The memory unit 32 is a storage medium that stores programs and data necessary to realize the functions of the personal terminal 3. As shown in Figure 3, the memory unit 32 includes a storage unit 32a and a temporary storage unit 32b.

[0027] The storage unit 32a stores parameters, data, and control programs for realizing predetermined functions. The storage unit 32a is composed of, for example, an HDD or an SSD. For example, the storage unit 32a stores the above-mentioned programs, as well as various image data and audio data.

[0028] The temporary storage unit 32b is composed of RAM such as DRAM or SRAM, and temporarily stores (i.e., holds) data. For example, the temporary storage unit 32b holds image data and audio data. The temporary storage unit 32b may also function as a work area for the control unit 31, or it may be composed of a storage area in the internal memory of the control unit 31.

[0029] The operation unit 33 is a general term for the operating components that the user operates. The operation unit 33 may be, for example, a keyboard, mouse, touchpad, touch panel, buttons, and switches. The operation unit 33 also includes various GUIs such as virtual buttons, icons, cursors, and objects displayed on the display unit 34. The operation unit 33 is an example of an input unit in the personal terminal 3. The input unit may be an interface circuit that receives user operations from external operating components such as a hand controller.

[0030] The display unit 34 is an example of an output unit, which may be composed of, for example, a liquid crystal display or an organic EL display. The display unit 34 may display various types of information, such as various GUIs for operating the operation unit 33 and information input from the operation unit 33.

[0031] The communication unit 35 is a module (circuit) that connects to an external device according to a predetermined communication standard in wired or wireless communication. The predetermined communication standard includes, for example, USB, HDMI, IEEE802.11, Wi-Fi, Bluetooth, etc. The communication unit 35 may also connect to a communication network such as the Internet. The communication unit 35 is an example of an acquisition unit that receives various information from an external device or a communication network.

[0032] The microphone 36 is an example of a detection unit that includes one or more microphone elements built into the personal terminal 3, for example, and picks up external sound from the personal terminal 3. The microphone 36 outputs an audio signal indicating the picked-up sound to the control unit 31. An external microphone may be used in the personal terminal 3. The personal terminal 3 may be equipped with a connection part, such as a terminal for connecting to an external microphone, as a detection unit, in place of or in addition to the built-in microphone 36.

[0033] The speaker 37 is an example of an output unit that includes one or more speaker elements built into the personal terminal 3 and outputs sound to the outside of the personal terminal 3 under control from the control unit 31. The personal terminal 3 may use an external speaker or earphones. The personal terminal 3 may also be equipped with a connection unit for connecting to an external speaker or the like as an output unit, in place of or in addition to the built-in speaker 37.

[0034] Camera 38 is positioned in a location on the personal terminal 3, for example, where it can capture images of the user's face or other objects viewing the display unit 34, and includes an image sensor and an optical system. Camera 38 is, for example, a visible light camera capable of capturing images in the visible wavelength range. In addition to or instead of this, camera 38 may be an infrared light camera capable of capturing images in the infrared wavelength range. In this case, the personal terminal 3 may further include an infrared light source as a light source for the camera 38 to capture images.

[0035] The configuration of the personal terminal 3 described above is just one example, and the configuration of the personal terminal 3 is not limited to this. For example, the acquisition unit in the personal terminal 3 may be realized through cooperation with various software in the control unit 31, etc. The input unit in the personal terminal 3 may acquire various information by reading various information stored in various storage media (for example, storage unit 32a) into the work area of ​​the control unit 31 (for example, temporary storage unit 32b).

[0036] Furthermore, the display unit 34 of the personal terminal 3 may be a variety of display devices, such as an HMD and a projector. Also, if an external display device is used, for example, the display unit 34 of the personal terminal 3 may be an output interface circuit for video signals, such as one compliant with the HDMI standard.

[0037] 2.Operation The operation of the virtual space system 1 of this embodiment, which is configured as described above, will be explained below.

[0038] This system 1 provides a journey through various locations in a virtual space 10 such as the metaverse, i.e., metaverse travel, by allowing the user 5 to operate an avatar 15. The operation of this system 1 will be explained using Figure 1.

[0039] In this system 1, as illustrated in Figure 1, virtual tourist spots 11 are constructed by placing various 3D objects in specific locations in the virtual space 10, for example, to recreate real-world tourist attractions. Metaverse travel can be set up to be an experience such as visiting various real-world tourist spots 11 around the world in the virtual space 10.

[0040] In Figure 1, during a metaverse journey in the virtual space 10, avatars 15 of multiple users 5 are being guided to tourist spots 11 by a guide 12. Guide 12 is, for example, an object that acts as a guide for the metaverse journey, and is, for example, the avatar of the user acting as the guide. Users 5 may participate in the metaverse journey individually, or as a group including multiple users.

[0041] This system 1 enables multiple users 5 to share experiences such as metaverse travel in a virtual space 10 through data communication between multiple personal terminals 3 and a virtual space server 2. Each personal terminal 3 inputs various operations of the user 5 via its operation unit 33 and captures the user 5's voice via its microphone 36, and transmits the obtained input information to the virtual space server 2.

[0042] The virtual space server 2 generates various images and sounds in the virtual space 10 using the image processing unit 21a and the sound processing unit 21b based on the input information received from each personal terminal 3, and transmits data indicating the generated images and sounds to each personal terminal 3. Each personal terminal 3 displays images of the virtual space 10 on the display unit 34 or outputs sounds from the speaker 37 based on the data received from the virtual space server 2.

[0043] 2-1. Metaverse Travel Registration Process System 1 accepts applications from users to participate in a metaverse trip, for example, before the trip described above is implemented, and performs pre-configurations according to the user's requests. The process for accepting applications for a metaverse trip in this embodiment will be explained using Figures 4 and 5.

[0044] Figure 4 is a flowchart illustrating the registration process for a metaverse trip in System 1. Figure 5 illustrates the registration screen for a metaverse trip in System 1. The flow illustrated in Figure 4 is executed, for example, by the control unit 21 of the virtual space server 2. The registration screen illustrated in Figure 5 is displayed, for example, on the display unit 34 of the personal terminal 3 while the flow in Figure 4 is being executed. Below, we will describe an example of the process when a group organizer applies for a metaverse trip.

[0045] First, for example, the control unit 21 of the virtual space server 2 determines which trip the user will participate in based on the user operation information received from the personal terminal 3 (S1). The user inputs a user operation to the operation unit 33, selecting a desired trip name from various pre-prepared trip names on the reception screen displayed on the display unit 34 of the personal terminal 3, as shown in Figure 5. The control unit 31 of the personal terminal 3 transmits the selection result from the user operation to the virtual space server 2 via the communication unit 23 (S1).

[0046] Furthermore, the control unit 21 sets the user's participation mode, nickname, and avatar based on user operations on the personal terminal 3 (S2). In this system 1, as shown in Figure 5, users can select "individual" or "group" as the mode of participation in the metaverse journey. In addition, the user can, for example, enter a desired nickname or specify the appearance of various avatars on the operation unit 33 of the personal terminal 3. This user operation input information is transmitted from the personal terminal 3 to the virtual space server 2 (S2).

[0047] For example, if the user's participation is in a "group" (YES in S3), the control unit 21 of the virtual space server 2 assigns a group ID to identify whether or not the user is a member of the group, and a member ID to identify each member within the group (S4).

[0048] Furthermore, the control unit 21 accepts the setting of "real name" and "real face" for users whose participation is in a group via the personal terminal 3 (S5). The "real name" is a name that can be set separately from the nickname (S2), and together with the "real face," it can be kept secret from anyone other than group members. The "real face" is an appearance that can be set separately from the appearance of the avatar set in step S2, and can, for example, be a real photograph or a 3D scan of the user. This information is also transmitted from the personal terminal 3 to the virtual space server 2 in the same manner as above (S5).

[0049] On the other hand, if the user's participation is as an "individual" and not as a "group" (NO in S3), the control unit 21 proceeds to step S6 without performing the processing in steps S4 and S5. In this case, the control unit 21 may assign a personal ID to identify the individual user instead of performing step S4.

[0050] Next, the control unit 21 receives the settings for various services in the metaverse journey via the personal terminal 3 (S6). In this system 1, as shown in Figure 5, for example, guide services, group services, individual photography services, warp travel services, and voice conversion services are provided. The various services will be described later. When a user inputs a user operation to activate a desired service on the personal terminal 3, for example, this system 1 makes the service corresponding to that user available in the metaverse journey.

[0051] The control unit 21 associates the various pieces of information set in steps S1 to S6 above, stores them in the storage unit 22 as the user's reception information, and terminates the process shown in this flow.

[0052] With the metaverse travel registration process described above, System 1 can obtain the user's requests in advance before the metaverse travel, making it easier to realize a metaverse travel that aligns with the user's intentions.

[0053] Figures 6(A) and 6(B) illustrate the display of avatars 15 inside and outside a group in this system 10. For example, based on the reception information set as described above, the virtual space server 2 of this system 1 switches the name displayed on the user's avatar 15 and the appearance of the avatar 15 between inside and outside the group.

[0054] For example, in the live view on the display unit 34 of a personal terminal 3 of a member of the same group as the user, the real name set in step S5 is displayed on the user's avatar 15, as shown in Figure 6(A). In addition, the user's avatar 15 is displayed with their real face.

[0055] On the other hand, in the live view on a personal terminal 3 that is not a member of the user group, the nickname is displayed on the user's avatar 15, as shown in Figure 6(B). In addition, the user's avatar 15 is displayed with the appearance set in step S2.

[0056] As described above, by switching the information displayed on avatar 15 inside and outside the group, it is possible to ensure the privacy of users when participating in a metaverse journey as a group, and to make it easier to experience virtual spaces 10 such as metaverse journeys.

[0057] 2-2. Actions during metaverse travel Based on the reception information set as described above, the operation of this system 1 when conducting a metaverse trip will be explained using Figures 7 and 8.

[0058] Figure 7 is a flowchart illustrating the operation during metaverse travel in this system 10. Figure 8 is a diagram illustrating the live view in this system 10.

[0059] The process shown in Figure 7 is initiated, for example, when a metaverse journey begins in System 1. Figure 7 illustrates the process that the virtual space server 2 performs during a user's metaverse journey, for example, in response to input from a specific user's personal terminal 3.

[0060] In this system 1, for example, the control unit 21 of the virtual space server 2 determines whether the voice conversion service is ON (enabled) for a given user based on reception information managed for each user in the storage unit 22 (S11). The voice conversion service is a service that converts the sound quality and volume of the user's own voice or the voice of another person to meet the user's requirements.

[0061] For example, as shown in Figure 5, the user can select either "avatar sound quality" or "volume optimization" or both as the voice conversion service. If the voice conversion service is ON (YES in S11), the control unit S11 sets the voice conversion process according to the user's selection to the voice processing unit 21b (S12). On the other hand, if the voice conversion service is OFF (disabled) instead of ON (NO in S11), the process in step S12 is not performed.

[0062] "Avatar sound quality" is a voice conversion service that changes the sound quality of the user's voice to match the gender and age of the user's avatar, for example. For example, if a male user uses a female avatar, the control unit 21 sets the voice processing unit 21b to perform voice processing to convert the male voice to a female voice (S12). This avatar sound quality voice conversion service can reduce the unnaturalness of the user's voice coming from an avatar.

[0063] "Volume optimization" is a voice conversion service that adjusts the volume of voices a user hears from other people's avatars, or adjusts the volume of a user's voice when others hear it. For example, the control unit 21 sets the voice processing unit 21b to perform voice processing that adjusts the volume of input and output voices to a volume set in advance by the user (S12). For example, even if the user speaks softly or the distance between avatars is large, the voice can be made easier to hear.

[0064] Furthermore, the control unit 21 determines whether the guide service is ON or OFF based on the received information (S13). The guide service is a service that makes it possible to view the information presented by the guide 12 in, for example, the live view of the personal terminal 3. The user can select either the image or the audio of the guide 12, or both, as shown in Figure 5, for example.

[0065] If the guide service is OFF (NO in S13), the control unit 21 starts a live view that visualizes the real-time situation in the virtual space 10 on, for example, the personal terminal 3, without adding any guide information (S14). On the other hand, if the guide service is ON (YES in S13), the control unit 21 starts the live view with guide information according to the user's selection (S14). The live view of this system 1 will be explained using Figures 8(A) to (C).

[0066] Figures 8(A) and 8(B) illustrate the live view screen when no guide information is added (S14). In the example in Figure 8(A), the display unit 34 of the personal terminal 3 displays a first-person live view screen corresponding to the field of view of the user's own avatar 15. In the example in Figure 8(B), the display unit 34 displays a third-person live view screen corresponding to a field of view different from that of the user's own avatar 15.

[0067] System 1 allows the user to switch between various live view screens, such as those shown in Figures 8(A) and (B), by giving instructions from the control panel 33 of the personal terminal 3, for example, during a metaverse journey. In addition, along with these screen displays, System 1 outputs various sounds with volumes corresponding to the relative positions of each avatar in the virtual space 10, for example, from the speaker 37 of the personal terminal 3.

[0068] In the live view described above, it is possible that the user's avatar 15 may not be facing the direction of the guide 12, or the guide 12 may be far away, resulting in the guide's image not being displayed or its voice being difficult to hear. Therefore, this system 1 provides a guide service that can avoid such situations (Figure 8(C)).

[0069] Figure 8(C) illustrates the live view screen when guide information is added (S15). In this example, since "guide image" is selected as the guide information (Figure 5), the control unit 21 controls the image processing unit 21a to superimpose the image of guide 12 onto the live view screen. In addition to or instead of this, if "guide audio" is selected, the control unit 21 controls the audio processing unit 21b to maintain the volume of guide 12 in the audio data output during live view. With this guide service, the user can stably view the guide information regardless of their position relative to guide 12, for example.

[0070] Returning to Figure 7, the control unit 21 determines whether the warp movement service is ON or OFF based on the received information (S16). The warp movement service is a service that warps the avatar 15 in the virtual space 10. In this embodiment, an example is described in which the warp movement service is performed using location information within a group (see Figure 5).

[0071] If the warp travel service is ON (YES in S16), the control unit 21 determines whether or not a help request has been received from a personal terminal 3 during a metaverse trip (S17). For example, in a metaverse trip, a schedule is predetermined, including the time periods for visiting various locations. According to the warp travel service of this embodiment, a user can send a help request from their personal terminal 3 to the virtual space server 2 to resolve a situation where, for example, an avatar 15 of a group member is absent from the meeting place in the schedule.

[0072] When the control unit 21 of the virtual space server 2 receives a help request from a personal terminal 3 (YES in S17), it sends location information indicating the location of the meeting place in the above schedule to each member's personal terminal 3 for warping (S18). By using this warp location information, absent members can warp to the meeting place indicated by the location information in the virtual space. The transmission of the warp location information (S18) may be performed by identifying the absent member based on the status of the virtual space at the time of the help request.

[0073] This transmission of location information for warp (S18) is not performed when the warp travel service is OFF (NO in S16) or when there is no help request (NO in S17).

[0074] Furthermore, the control unit 21 determines whether the group service is ON or OFF based on the reception information (S19). The group service is a service that makes it easier for users to take videos with members of their group. If the group service is ON (YES in S19), the control unit 21 performs group shooting processing according to the user's selection (S20). On the other hand, if the group service is OFF (NO in S19), the group shooting processing (S20) is not performed.

[0075] For example, as shown in Figure 5, users can select either "remove subject" or "remove sound" or both in the group service. In the group shooting process (S20), "remove subject" allows for the bulk removal of avatars that are not members of the group from the video, and "remove sound" allows for the bulk removal of sound from avatars that are not members of the group. Details of the group shooting process (S20) in this embodiment will be described later.

[0076] The control unit 21, for example, repeatedly executes the processes in steps S16 to S20 until the metaverse journey is complete (S21), and terminates the processing of this flow once the metaverse journey is finished.

[0077] According to the operation of System 1 described above, in response to various service requests from users, it is possible to realize viewing experiences of images and sounds that cannot be obtained in the real world, for example, during live view and video recording while conducting a metaverse trip.

[0078] Steps S11 and S12 described above illustrate an example of the voice conversion service shown in Figure 5. The voice conversion service (S11, S12) of System 1 is not limited to the example described above and may configure various sound conversion processes. For example, the voice tone of a specific avatar 15 may be changed from the standpoint of ensuring privacy, or the natural voice may be output only within a group. Furthermore, the volume of a specific object may be changed from the standpoint of increasing the volume of an avatar 15 that you want to stand out more, or reducing the volume of distracting ambient noise.

[0079] Furthermore, System 1 may perform voice zoom on an avatar 15 that is speaking, for example. In the virtual space 10, the positional relationship with the avatar 15 that is speaking can be accurately determined, so voice zoom can be achieved with greater precision than in the real world. In addition, System 1 may provide a service to translate the speech of a specific avatar 15, and may output the translated audio or display it as subtitles.

[0080] Furthermore, steps S16-S18 described an example in which location information is transmitted in response to a help request in a warp travel service that uses location information. In this system 1, the transmission of location information for warping (S18) is not limited to a user's help request, but may also be performed automatically, for example, by detecting an avatar 15 that is absent during the time slot when it is supposed to gather at a meeting place, from the perspective of a metaverse travel schedule.

[0081] 2-2-1. Group shooting processing The details of the group imaging process in step S20 of Figure 7 will be explained using Figures 9-10.

[0082] Figure 9 is a flowchart illustrating the group imaging process (S20) in System 1. Figure 10 shows an example of the display in the group imaging process (S20). The process shown in Figure 9 is executed each time step S20 in Figure 7 is repeated, for example, during a metaverse trip.

[0083] First, the control unit 21 of the virtual space server 2 changes the display mode of the object to be deleted to a predetermined state, such as semi-transparent, for each live view on the personal terminal 3, after a predetermined period (e.g., 30 minutes) has elapsed since the start of the metaverse journey (YES in S31) (S32). An example of the display in step S32 is shown in Figure 10(A).

[0084] In the example shown in Figure 10(A), the control unit 21 changes the display of avatars 151 and 152 that are not members of the same group as the user to semi-transparent based on the user's group ID (S32). Such semi-transparent display may be applied not only to avatars but also to various objects in the virtual space 10. Furthermore, the display mode for deletion candidates is not limited to semi-transparent display, and various display modes may be adopted.

[0085] The control unit 21 determines, for example, whether a user has instructed a user to return from the deletion candidates via operation on the operation unit 33 of the personal terminal 3 (S33). For example, if the user selects a desired avatar 151 from among the semi-transparently displayed avatars 151 and 152 on the live view screen via touch operation, the personal terminal 3 sends a command to the virtual space server 2 to return the selected avatar 151 (S33). For example, during such user operations, the personal terminal 3 may enlarge the display on the live view screen to make it easier for the user to select the desired avatar 151.

[0086] If the control unit 21 receives a recovery instruction from, for example, the personal terminal 3 (YES in S33), it restores the display mode of the avatar 151 selected by the instruction in the live view of the personal terminal 3 from semi-transparent to its original state, and restores the avatar 15 from the list of candidates for deletion (S34). An example of the display in step S34 after Figure 10(A) is illustrated in Figure 10(B).

[0087] On the other hand, if there is no specific instruction to return (NO in S33), the control unit 21 proceeds to step S35 without performing the processing in step S34. When the control unit 21 receives, for example, an instruction to start video recording from the personal terminal 3 (YES in S35), it controls the image processing unit 21a and the audio processing unit 21b regarding the deletion candidates as a preparatory process for video recording (S36-S40). The control unit 21 can identify these deletion candidates based on various ID information such as the group ID (and whether or not there is an instruction to return).

[0088] For example, if the "Delete Subject" group service in the reception information (Figure 5) is ON (YES in S36), the control unit 21 controls the image processing unit 21a to delete the avatar 152, which has been designated as a deletion candidate in the semi-transparent display, in the recorded video (S37). An example of the display in step S37 after Figure 10(B) is shown in Figure 10(C). In this case, in the live view of personal terminal 3, for example, the semi-transparent display of avatar 152 is switched to a transparent display. On the other hand, if "Delete Subject" is OFF (NO in S36), the process in step S37 is not performed, and avatar 152, which is a candidate for deletion, will not be deleted and will be visible in the recorded video, for example.

[0089] Furthermore, if the "Sound Deletion" group service in the reception information (Figure 5) is ON (YES in S38), the control unit 21 controls the audio processing unit 21b to delete the audio from the semi-transparent avatar 152 in the recorded video (S39). In this case, the audio from the avatar 152 that is subject to sound deletion may also be deleted from the audio output during live view on the personal terminal 3.

[0090] Furthermore, the control unit 21 displays an indicator in the live view display of the personal terminal 3 to identify whether or not each avatar 15 is subject to sound deletion processing (S40). An example of the display in step S40 is shown in Figure 10(D). In this example, the identification indicator is made by adding a predetermined icon 60 to avatars 15 that are not subject to sound deletion to indicate that sound is being recorded. Such identification indicators are not limited to icons 60 and may be displayed in various forms. Also, the identification indicator is displayed in live view but does not need to be displayed in recorded video.

[0091] The control unit 21 performs the above preprocessing (S36-S40) and then executes video recording in the virtual space 10 (S41). This video recording continues, for example, until the personal terminal 3 instructs it to stop recording. The recorded video may be recorded in the storage unit 22 of the virtual space server 2 or the storage unit 32 of the personal terminal 3, or it may be distributed via the communication network 4, etc. For example, the video being recorded may be superimposed on the live view and displayed in PIP (Picture-in-Picture) on the display unit 34 of the personal terminal 3 of a group member of the user.

[0092] The control unit 21 terminates the group shooting process (S20 in Figure 7) in response to the completion of video recording (S41), for example, and proceeds to step S21.

[0093] Furthermore, if a user issues a shooting instruction (YES in S35) before a predetermined period has elapsed since the start of the metaverse journey (NO in S31), the control unit 21 processes steps S36 to S41, for example, by considering everyone outside the user's group as a candidate for deletion. Also, if no shooting instruction is issued (NO in S35), the control unit 21 terminates step S20 in Figure 7 without performing steps S36 to S41, for example, and proceeds to step S21.

[0094] The group shooting process described above (S20) makes it easy to shoot videos that are suitable for group travel but difficult to shoot in the real world, such as intentionally excluding members of the user's group from the video or removing their audio.

[0095] Furthermore, according to the semi-transparent processing in steps S31 to S34 described above, the user can, for example, record a video without deleting someone they have become close to outside of the group during a predetermined period from the start of the metaverse journey by issuing a restore instruction to that person.

[0096] The above description illustrates an example in which the semi-transparent processing (S31-S34) is performed after a predetermined period has elapsed since the start of the metaverse journey. However, System 1 is not limited to this example; for example, it may be configured to perform the semi-transparent processing (S31-S34) in response to instructions from a user or a member of their group.

[0097] Furthermore, System 1 may perform the various processes described above during video recording in step S41. For example, the warp movement process (S16-S18 in Figure 7) may be performed during video recording.

[0098] 2-3. About our individual photography service As described above, the group service of System 1 allows users to, for example, collectively set all users who are not members of a group as targets for deletion during video recording. Alternatively, or in addition to, this group service, System 1 may also provide an individual recording service that allows users to specify deletion targets individually (see Figure 5). The individual recording service of System 1 will be explained using Figure 11.

[0099] Figure 11 illustrates the live view during metaverse travel when the individual shooting service is ON. In this system 1, for example, the control unit 21 sets whether subject deletion and sound deletion can be performed in the live view during virtual travel, according to the ON / OFF status of "subject deletion" and "sound deletion" in the individual shooting service reception information (Figure 11). In the example in Figure 11, since both are set to be executable, the subject deletion button 61 and the sound deletion button 62 are displayed on the live view screen of the personal terminal 3.

[0100] For example, the user touches the subject deletion button 61 on the personal terminal 3, and then further touches the live view screen to specify the object to be deleted. The control unit 2 then obtains the result of these operations from the personal terminal 3 and, for example, instructs the image processing unit 21a to delete the specified object, similar to step S37 in Figure 9.

[0101] Furthermore, the user touches the sound delete button 62 on the personal terminal 3, and then further touches the live view screen to specify the object whose sound they want to delete. The control unit 2 then obtains the result of these operations from the personal terminal 3 and, for example, instructs the audio processing unit 21b to delete the sound of the specified object, similar to step S39 in Figure 9. Alternatively, the control unit 21 may, for example, display an identification of the object to be deleted on the live view screen in response to the operation of the sound delete button 62, similar to step S40.

[0102] Furthermore, in System 1, semi-transparency processing may also be performed in the individual shooting service, similar to steps S32 to S34 of the group shooting process (Figure 9). In the individual shooting service of System 1, the target of the semi-transparency display may also be individually specified by the user operation on the personal terminal 3 for the semi-transparency processing (S32 to S34).

[0103] Furthermore, when deleting a subject using the subject deletion button 61 described above, this system 1 may also delete the audio of the object deleted from the image at the same time. In addition, it may be possible to give a command to restore the audio of the deleted object.

[0104] Furthermore, for objects that are not subject to sound removal but are subject to subject removal, an icon 60 similar to that in Figure 10(D) may be displayed, or in addition to or instead of this, an identification indicator (e.g., a musical note mark) indicating the location of a sound source may be displayed. For example, if a user does not want the guide 12 to be visible but wants to hear the sound, the guide 12 can be excluded from the subject removal but not the sound removal.

[0105] Furthermore, in this system 1, the user operation to instruct the execution of subject deletion / sound deletion is not limited to the examples described above. For example, in the individual shooting service, instead of touching the subject deletion button 61 / sound deletion button 62, an upward / downward flick operation may be used, or user operations such as eye-tracking input may be used.

[0106] 3. Summary As described above, in this embodiment, the virtual space server 2 and the personal terminal 3 are information processing devices that display images in the virtual space 10, where an avatar representing the user moves, on a display unit 34 for the user. The information processing devices (2,3) include an input unit (23,33) that receives a user operation to set a specific object as the target of predetermined image processing among the objects placed in the virtual space 10, and a control unit (21,31) that displays an image on the display unit 34 in which image processing has been applied to the specific object in the virtual space 10 based on the user operation. With such an information processing device, it is possible to easily obtain an image in the virtual space 10 that matches the user's intentions.

[0107] In the information processing device of this embodiment, a predetermined image processing operation causes a specific object to differ from an image captured in the real world from an image after the image processing operation has been performed, in response to user operation. For example, the predetermined image processing operation is a process to delete or blur a specific object set by user operation on the image after the image processing operation has been performed. This makes it possible to capture images that would be difficult to capture in the real world, for example, and makes it easier to obtain images in the virtual space 10 that match the user's intentions.

[0108] Furthermore, the virtual space server 2 and personal terminal 3 in this embodiment are examples of information processing devices that output audio from the virtual space 10, where an avatar representing the user moves, through a speaker 37, which is an example of an output unit for the user. The information processing device includes an input unit that receives user operations to set a specific object as the target of predetermined audio processing among the objects placed in the virtual space 10, and a control unit that outputs audio processed audio from the specific object in the virtual space 10 based on the user operation. With such an information processing device, it is possible to easily obtain audio in the virtual space 10 that matches the user's intentions.

[0109] In the information processing device of this embodiment, the predetermined audio processing may suppress audio from a specific object set by user operation. Furthermore, in the information processing device of this embodiment, the predetermined audio processing may change at least one of the volume and sound quality of audio from a specific object in response to user operation. This makes it possible to capture sound that would be difficult in the real world, making it easier to obtain audio in the virtual space 10 that aligns with the user's intentions. The predetermined audio processing may also differentiate the audio from the audio captured in the real world from the audio processed by the specific object in response to user operation.

[0110] In the information processing device of this embodiment, the predetermined voice processing may change the voice emitted from the avatar according to the appearance of the avatar. This allows the user to use voices appropriate to the avatar in the virtual space 10, thereby improving the usability of the virtual space 10.

[0111] In the information processing device of this embodiment, user operations include setting up a group that includes the user and other members. Specific objects include objects other than the group members. This allows predetermined image and audio processing to be performed on objects other than the user's group members, making it easier to create the intended viewing experience for the user's group in the virtual space 10.

[0112] In the information processing device of this embodiment, user operations further include selecting objects other than group members that are not subject to image processing or audio processing. This allows users to exclude objects they intend to process from image processing and audio processing, even if they are not group members, making it easier for users to obtain the experience they intended.

[0113] In this information processing device, the control unit manages, with respect to the avatar representing the user, an example of a first name, the real name, which is displayed on the display unit 34 for group members, and an example of a second name, a nickname, which is different from the first name, which is displayed on the display unit 34 for non-group members. This makes it possible to improve the user experience within the group while ensuring the user's privacy outside the group.

[0114] In the information processing device of this embodiment, the avatar representing the user has a first appearance, which is an example of a real face, displayed on the display unit 34 for group members, and a second appearance, which is different from the first appearance, displayed on the display unit 34 for non-group members. This makes it possible to improve the user experience within the group while ensuring the user's privacy outside the group.

[0115] In the information processing device of this embodiment, user operation includes selecting objects in the virtual space 10 to be subjected to image processing and audio processing. This makes it easier for the user to individually specify the objects they want to be subjected to image processing and audio processing, thereby obtaining viewing in the virtual space 10 that aligns with the user's intentions.

[0116] In the information processing device of this embodiment, user operations include operations to request a predetermined service in a metaverse trip, which is an example of an avatar traveling around locations in the virtual space 10. According to the information processing device of this embodiment, the user experience in a metaverse trip can be improved.

[0117] In this information processing device, user operation includes requesting a guide to lead the avatar on a metaverse journey through locations in the virtual space 10. Audio processing outputs the guide's voice from the output unit during the journey. This allows the user to listen to the guide's voice while traveling through the metaverse, improving the user experience.

[0118] In this information processing device, the control unit identifies location information indicating a predetermined location in the virtual space 10, and displays an image of the avatar after it has moved to that location on the display unit 34 based on the location information. By using this warp movement with location information, it becomes easier to arrange meetings according to a schedule, for example, during a metaverse trip, thereby improving the user experience in the virtual space 10.

[0119] In this embodiment, the virtual space server 2, which acts as an information processing device, has a control unit 21 that performs predetermined image processing to generate an image with image processing applied to a specific object, and / or performs predetermined audio processing to generate audio with audio processing applied to audio from a specific object. This virtual space server 2 makes it easier for users to have an experience in the virtual space 10 that aligns with their intentions.

[0120] In this embodiment, the personal terminal 3, as an information processing device, further includes a communication unit 35 that communicates data with a virtual space server 2, which is an example of an external device that performs predetermined image processing and / or predetermined audio processing. The control unit 31 obtains an image of a specific object with image processing applied, and / or audio from a specific object with audio processing applied, from the external device via the communication unit 35. Even with such a personal terminal 3, it is possible to easily obtain an experience in the virtual space 10 that aligns with the user's intentions.

[0121] In this embodiment, an information processing method is provided for displaying an image in a virtual space 10, where an avatar representing the user moves, on a display unit 34 for the user. This method includes the steps of: receiving a user operation to set a specific object as the target of predetermined image processing among the objects placed in the virtual space 10; and displaying an image on the display unit 34 of the specific object in the virtual space 10 that has been image processed based on the user operation. This makes it easier to obtain an image in the virtual space 10 that matches the user's intentions.

[0122] In this embodiment, an information processing method is provided that causes the audio of a virtual space 10 in which an avatar representing the user moves to be output from an output unit for the user. This method includes the steps of: receiving a user operation to set an object to be subjected to predetermined audio processing among the objects placed in the virtual space 10; and, based on the user operation, causing the output unit to output audio that has been processed from the audio of the object set in the virtual space 10. This makes it easier to obtain audio in the virtual space 10 that matches the user's intentions.

[0123] In this embodiment, a program is provided to cause the control unit to execute each of the above-described information processing methods. This makes it easier for users to have an experience in the virtual space 10 that aligns with their intentions.

[0124] (Embodiment 2) Embodiment 2 of this disclosure will be described below with reference to Figures 12-13. Embodiment 1 described an example of operation using location information for the warp travel service. Embodiment 2 describes an example of operation using eye-tracking input for the warp travel service.

[0125] Hereinafter, descriptions of the configuration and operation similar to those of the virtual space system 1 according to Embodiment 1 will be omitted as appropriate, and the virtual space system 1 according to this embodiment will be described.

[0126] In the virtual space system 1 of this embodiment, the user's gaze is detected by capturing an image including the user's eyes using, for example, the camera 38 of the personal terminal 3. The camera 38 is an example of a detection unit in the personal terminal 3 of this embodiment.

[0127] For example, an infrared camera is used as camera 38, and the control unit 31 detects the user's gaze by performing corneal reflection image recognition. With corneal reflection, the gaze can be detected by image recognition of the positional relationship between the reflected light from the infrared light source on the cornea and the pupil. Camera 38 may also be a visible light camera, and for example, the gaze may be detected by image recognition of the positional relationship between the inner corner of the eye and the iris instead of corneal reflection. Alternatively, an external camera 38, such as a camera 38 in an external HMD attached to the personal terminal 3, may be used. In this case, the communication unit of the personal terminal 3 is an example of a detection unit that acquires the image captured by camera 38.

[0128] Figure 12 is a diagram illustrating a modified example of the warp travel service of this embodiment. Figure 13 is a flowchart illustrating the operation of such a warp travel service.

[0129] Figure 12(A) illustrates the live view when the eye-tracking warp movement service is ON. In this example, the display unit 34 of the personal terminal 3 displays a near-range button 65 and a far-range button 66 for eye-tracking warp movement.

[0130] For example, in the flow shown in Figure 13, the control unit 31 of the personal terminal 3 detects eye-gaze input when the user looks at the near-range button 65 (S51) and when the user looks at the far-range button 66 (S52). The control unit 31 detects eye-gaze input when it detects that the user is looking at the same spot on the screen of the display unit 34 for a predetermined period of time (e.g., several seconds) or longer. If no eye-gaze input is detected for either button 65 or 66 (NO in S51 and S52), the control unit 31 repeats the processing in steps S51 and S52.

[0131] When eye-tracking input from the short-range button 65 is detected (YES in S51), the control unit 31 switches the live view of the display unit 34 to a screen for short-range warp movement (S53). For short-range warp movement, for example, an overhead view of the virtual space 10 is used. An example of the display in step S53 is shown in Figure 12(B).

[0132] In step S53, the display unit 34 of the personal terminal 3 displays, for example, an overhead view image 70 of the virtual space 10, a current location marker 71 indicating the current position of the user's own avatar 15, and a grid frame 72 that divides the overhead view image 70 into regions, as shown in Figure 12(B).

[0133] For example, in step S53, the control unit 31 of the personal terminal 3 communicates with the virtual space server 2 to acquire image data of the overhead view image 70. During this warp travel service, the control unit 21 of the virtual space server 2 acquires various information from the personal terminal 3 and prompts the image processing unit 21a to generate the requested image data in a timely manner and transmits it to the personal terminal 3.

[0134] For example, while the screen shown in Figure 12 is displayed, the control unit 31 detects eye-tracking input to specify the warp destination (S54) and detects various eye-tracking inputs to adjust the screen display (S55).

[0135] For example, if the user continues to gaze at the desired location on the overhead image 70, the control unit 31 detects this gaze input (YES in S54). As shown in the screen display in Figure 12(B), the overhead image 70 is divided into regions by the grid frame 72, making it easier for the user to direct their gaze to the desired region within the frame using the grid frame 72 as a reference, and to specify the warp destination location by gaze input.

[0136] Furthermore, during the detection in step S54, the control unit 31 displays a warp marker 73 indicating the warp destination on the overhead image 70 at a position corresponding to the user's line of sight, as shown in Figure 12(C), for example. If the user continues to stare at the displayed warp marker 73, the control unit 31 confirms the line of sight input for specifying the warp destination and proceeds to YES in step S54.

[0137] When eye-tracking input for specifying the warp destination is detected (YES in S54), for example, the control unit 31 of the personal terminal 3 sends an instruction to the virtual space server 2 to move the user's own avatar 15 to the position of the warp marker 73 (S56). The control unit 31 acquires image data, etc., from the virtual space server 2 showing the state in which the user's own avatar 15 has moved to the position of the warp marker 73, and displays the live view after the warp on the display unit 34 (S56).

[0138] Furthermore, the eye-tracking input (S55) for adjusting the screen display for short-distance warp movement, as shown in Figure 12(B), is, for example, scrolling the overhead image 70 in the up, down, left, and right directions, and zooming in and out. For example, eye-tracking input for scrolling is performed by directing the gaze from the center of the display unit 34 screen to each end in the up, down, left, and right directions. Eye-tracking input for zooming in and out is performed by directing the gaze to a predetermined location on the screen, such as the upper right corner or the lower right corner.

[0139] When the control unit 31 detects various eye-tracking inputs for screen adjustment (YES in S55), it requests the adjusted overhead view image 70 based on the detected eye-tracking input from the virtual space server 2 and updates the screen display on the display unit 34 (S57). Subsequently, the control unit 31 repeats the processing from step S54 onwards on the updated screen display.

[0140] Furthermore, when eye-tracking input from the long-distance button 66 is detected (YES in S52), the control unit 31 switches the live view of the display unit 34 to display a screen for long-distance warp travel (S58). For long-distance warp travel, for example, in a display configuration similar to that of the short-distance warp travel screen described above, an image showing a map of the virtual space 10 is used instead of the overhead view image 70.

[0141] For example, during the display of the screen for such long-distance warp movement, the control unit 31 detects eye-tracking input for specifying the warp destination position, similar to step S54 (S59), and detects various eye-tracking inputs for screen adjustment, similar to step S55 (S60). In this case, by using a map image of the virtual space 10, the user can specify a distant position in the virtual space 10 with greater accuracy and speed than when performing short-distance warp movement (Figure 12(B)).

[0142] When the control unit 31 detects various gaze inputs for screen adjustment for long-distance warp movement (YES in S60), it requests the adjusted map image from the virtual space server 2 in place of the overhead image 70 in step S57, and updates the screen display on the display unit 34 (S61). When the control unit 31 detects gaze input for position specification for long-distance warp movement (YES in S59), it warps the user's own avatar 15 and displays its live view on the display unit 34, similar to the case of short-distance warp movement (YES in S54) (S56).

[0143] The control unit 31 of the personal terminal 3 terminates the process exemplified in this flow by displaying the live view after warp movement (S56). Thereafter, the control unit 31 repeats the process of this flow, for example, at a predetermined period.

[0144] Through the above process, the system 1 can, for example, detect the user's gaze input on the personal terminal 3, making it easier for the user to perform warp movements desired by the user in the virtual space 10.

[0145] As described above, as an information processing device in this embodiment, for example, the personal terminal 3 further includes a camera 38 as an example of a detection unit for detecting the user's gaze. Based on the gaze detected by the detection unit, the control unit 31 moves the avatar in the virtual space 10 to a position separated from the current position (S56). This allows the user to warp in the virtual space 10 by gaze input, improving user convenience in this system 1.

[0146] The various processes described above may be performed not only by the control unit 31 of the personal terminal 3, but also by the control unit 21 of the virtual space server 2. For example, the control unit 21 may perform eye-tracking image recognition by acquiring the captured image from the camera 38 from the personal terminal 3 via the communication unit 23. The communication unit 23 of the virtual space server 2 is an example of a detection unit in the virtual space server 2 as an information processing device in this embodiment.

[0147] (Other embodiments) As described above, Embodiments 1 and 2 have been explained as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited thereto and can be applied to embodiments that are modified, replaced, added to, or omitted as appropriate. Furthermore, it is possible to create new embodiments by combining the components described in each of the above embodiments.

[0148] In the above-described Embodiment 1, an example of image processing to remove a user-desired subject from an image in the virtual space system 1 was explained, but this embodiment is not limited to this. For example, the virtual space server 2 of this embodiment may perform a process to blur a user-desired subject on the image of the virtual space 10 as an alternative image processing to the subject removal in step S37 of Figure 9. Such image processing can be realized, for example, by utilizing the technology disclosed in Patent Document 1. Such image processing also makes it easier to obtain an image in the virtual space 10 that conforms to the user's intentions, similar to the above-described embodiment.

[0149] Furthermore, while the above embodiments describe examples of predetermined image and audio processing during video recording in the virtual space system 1, this embodiment is not limited thereto. For example, the system 1 may perform the same image processing as in the above embodiments when taking still images. For example, the virtual space server 2 may, in response to an operation by the user to specify subjects they do not want to be shown in a commemorative photo taken during a metaverse trip, perform image processing to delete or blur those subjects. In addition, the system 1 may apply predetermined image processing, etc., at times other than when shooting videos or still images.

[0150] Furthermore, while deletion and blurring were exemplified as predetermined image processing in the above embodiments, this embodiment is not limited to these, and various image processing may be performed on the object as the subject. For example, when taking an image at a tourist spot 11 (Figure 1) in the virtual space 10, image processing such as changing the relative scale between the various objects of the tourist spot 11 and the avatar 15 may be performed. This makes it easy to take images similar to trick art photography in the real world using the virtual space 10, and makes it easier to obtain images in the virtual space 10 that match the user's intentions.

[0151] Furthermore, while the above embodiments describe examples of metaverse travel applications in the virtual space system 1, these embodiments are not limited to these. The system 1 can be applied to obtaining at least one of images and sounds for various user experiences in the virtual space.

[0152] For example, System 1 may perform audio processing to suppress audience noise during live viewing of a performance or lecture by a famous person in a virtual space 10. In this case, the audience may remain displayed on the image, or image processing may be performed to remove the display of the audience. Alternatively, audio processing may be performed to suppress other noises while retaining only the sound of a specific instrument or a specific person. Furthermore, System 1 may only record audio and not images, or it may output audio only without displaying images.

[0153] As described above, embodiments have been explained as examples of the technology in this disclosure. For this purpose, accompanying drawings and a detailed description have been provided. Therefore, among the components described in the accompanying drawings and detailed description, there may be not only components that are essential for solving the problem, but also components that are not essential for solving the problem, in order to illustrate the above technology.

[0154] (Summary of characteristics) The various aspects of this disclosure are described below.

[0155] A first aspect of the present disclosure is an information processing device that displays an image in a virtual space in which an avatar representing a user moves on a display unit for the user, comprising: an input unit that receives a user operation to set a specific object as the target of predetermined image processing among the objects arranged in the virtual space; and a control unit that displays an image on the display unit in which image processing has been applied to the specific object in the virtual space based on the user operation.

[0156] The second aspect is an information processing device described in the first aspect, wherein a predetermined image processing method, in response to user operation, causes an image to be different from an image of a specific object taken in the real world from an image to an image to which the image processing method has been applied.

[0157] The third aspect is an information processing apparatus described in the first or second aspect, wherein the predetermined image processing is a process of deleting or blurring a specific object set by user operation on an image to which the image processing has been applied.

[0158] A fourth aspect is an information processing device that outputs audio from an output unit for a user in a virtual space in which an avatar representing a user moves, and comprises an input unit that receives a user operation to set a specific object as the target of predetermined audio processing in the objects arranged in the virtual space, and a control unit that outputs audio from an output unit in which audio processing has been applied to the audio from the specific object in the virtual space based on the user operation.

[0159] The fifth aspect is the information processing device described in the fourth aspect, wherein the predetermined audio processing suppresses audio from a specific object set by user operation.

[0160] The sixth aspect is an information processing apparatus according to the fourth or fifth aspect, wherein a predetermined audio processing method changes at least one of the volume and sound quality of audio from a specific object in response to user operation.

[0161] The seventh aspect is an information processing device described in any of the fourth to sixth aspects, wherein the predetermined voice processing modifies the voice emitted from the avatar according to the appearance of the avatar.

[0162] The eighth aspect is an information processing device described in any of the first to seventh aspects, wherein the user operation includes an operation to set up a group that includes the user and other members, and the specific object includes objects other than members of the group.

[0163] The ninth aspect is an information processing apparatus described in the eighth aspect, wherein the user operation further includes an operation to select objects other than group members that are not subject to image processing and / or audio processing.

[0164] The tenth embodiment is an information processing device described in any of the first to ninth embodiments, wherein the control unit manages a first name displayed on a display unit for group members and a second name, which is different from the first name, displayed on a display unit for non-group members, with respect to an avatar representing a user.

[0165] The eleventh embodiment is an information processing device according to any of the first to tenth embodiments, wherein the avatar representing the user has a first appearance displayed on a display unit for group members and a second appearance different from the first appearance displayed on a display unit for non-group members.

[0166] The twelfth aspect is an information processing device described in any of the first to eleventh aspects, wherein the user operation includes an operation to select an object in a virtual space to be the subject of image and / or sound processing.

[0167] The thirteenth aspect is an information processing device described in any of the first to twelfth aspects, wherein the user operation includes an operation in which an avatar requests a predetermined service during a journey around locations in a virtual space.

[0168] The 14th aspect is an information processing device described in any of the 1st to 13th aspects, wherein the user operation includes an operation to request a guide that will lead the avatar on a journey around places in a virtual space, and the voice processing causes the voice of the guide to be output from the output unit during the journey.

[0169] The 15th embodiment is an information processing device described in any of the 1st to 14th embodiments, wherein the control unit identifies location information indicating a predetermined location in a virtual space, and based on the location information, causes the display unit to display an image of the avatar after it has moved to the location.

[0170] The sixteenth embodiment is an information processing device described in any of the first to fifteenth embodiments, further comprising a detection unit for detecting the user's gaze, wherein the control unit moves the avatar in the virtual space to a position separated from its current position based on the gaze detected by the detection unit.

[0171] The 17th aspect is an information processing apparatus according to any of the 1 to 16 aspects, wherein the control unit performs predetermined image processing to generate an image in which a specific object has been processed, and / or performs predetermined audio processing to generate audio in which audio has been processed from audio from a specific object.

[0172] The 18th embodiment is an information processing device according to any of the 1st to 16th embodiments, further comprising a communication unit that communicates data with an external device that performs predetermined image processing and / or predetermined audio processing, wherein the control unit obtains an image of a specific object that has undergone image processing and / or audio from a specific object via the communication unit.

[0173] The 19th aspect is an information processing method for displaying an image in a virtual space in which an avatar representing a user moves on a display unit for the user, comprising the steps of: receiving a user operation to set a specific object as the target of predetermined image processing among the objects arranged in the virtual space; and displaying an image on the display unit in which image processing has been applied to the specific object in the virtual space based on the user operation.

[0174] The 20th aspect is an information processing method for outputting audio from an output unit for a user in a virtual space in which an avatar representing a user moves, and includes the steps of: receiving a user operation to set an object to be subjected to predetermined audio processing in an object placed in the virtual space; and, based on the user operation, outputting audio from an output unit in which audio processing has been applied to the audio from the object set in the virtual space.

[0175] The 21st aspect is a program for causing a control unit to execute the information processing method described in the 19th or 20th aspect. [Industrial applicability]

[0176] This disclosure is applicable to various information technologies that provide users with experiences in virtual space. [Explanation of symbols]

[0177] 1. Virtual Space System 10 Virtual Space 15 Avatars 2. Virtual Space Server 21 Control Unit 22 Memory section 23 Communications Department 3. Personal devices 31 Control Unit 32 Storage section 33 Operation section 34 Display section 35 Communications Department 36 Mike 37 speakers 38 Cameras

Claims

1. An information processing device that outputs audio from an output unit for the user in a virtual space where an avatar representing the user moves, An input unit that accepts user input to set a specific object, different from the avatar representing the user, among the objects placed in the virtual space, as the target of predetermined voice processing, A control unit that, based on the user operation, outputs from the output unit an audio that has been processed based on the audio from the specific object in the virtual space. An information processing device equipped with the following features.

2. An information processing device that outputs audio from an output unit for a user in a virtual space in which an avatar representing a user moves, An input unit that accepts user input to set a specific object as the target of predetermined audio processing in the object placed in the virtual space, A control unit that, based on the user operation, outputs from the output unit an audio that has been processed based on the audio from the specific object in the virtual space. Equipped with, The output unit includes a first output unit and a second output unit different from the first output unit. The control unit, Based on the user operation, the audio processed from the audio of the specific object is output from the first output unit. For a user different from the user who performed the aforementioned user operation, the audio output from the second output unit for the user corresponding to the specific object will not be subjected to the aforementioned audio processing for the audio from the specific object. Information processing device.

3. The aforementioned predetermined audio processing suppresses audio from specific objects set by the user operation. The information processing apparatus according to claim 1 or 2.

4. The predetermined audio processing modifies at least one of the volume and sound quality of the audio from the specific object in response to the user operation. The information processing apparatus according to claim 1 or 2.

5. The aforementioned predetermined audio processing modifies the audio emitted from the avatar according to the appearance of the avatar. The information processing apparatus according to claim 1 or 2.

6. The user operation includes setting up a group that includes the user and other members. The aforementioned specific object includes objects other than members of the group. The information processing apparatus according to claim 1 or 2.

7. The user operation described above further includes selecting objects other than members of the group that are not subject to the audio processing. The information processing apparatus according to claim 6.

8. The user operation includes selecting an object in the virtual space to be used for the audio processing. The information processing apparatus according to claim 1 or 2.

9. The user operation includes an operation in which the avatar requests a predetermined service during a journey around a location in the virtual space. The information processing apparatus according to claim 1 or 2.

10. The user operation includes an operation in which the avatar requests a guide to take it on a journey around the virtual space, The aforementioned audio processing causes the guide's voice to be output from the output unit during the trip. The information processing apparatus according to claim 1 or 2.

11. The device further includes a detection unit for detecting the user's gaze, Based on the line of sight detected by the detection unit, the control unit moves the avatar in the virtual space to a position separated from its current position. The information processing apparatus according to claim 1 or 2.

12. An information processing method for outputting audio from an output unit for a user in a virtual space where an avatar representing a user moves, The steps include receiving a user operation to set an object different from the avatar representing the user, among the objects placed in the virtual space, as the target of predetermined voice processing, Based on the user operation, the output unit outputs audio from the virtual space that has been processed using the audio processing applied to the audio from the configured object. Information processing methods including

13. An information processing method for outputting audio from an output unit for a user in a virtual space where an avatar representing a user moves, The steps include: accepting user input to set a specific object to be subjected to predetermined audio processing in the virtual space; Based on the user operation, the output unit outputs audio from the virtual space that has been processed using the audio processing applied to the audio from the configured object. Includes, The output unit includes a first output unit and a second output unit different from the first output unit. In the step of outputting from the output unit, Based on the user operation, the audio processed from the audio of the specific object is output from the first output unit. For a user different from the user who performed the aforementioned user operation, the audio output from the second output unit for the user corresponding to the specific object will not be subjected to the aforementioned audio processing for the audio from the specific object. Information processing methods.

14. A program for causing a control unit to execute the information processing method described in claim 12 or 13.

Images