Method for generating sound and electronic device performing same

Abstract

This electronic device may comprise at least one processor including processing circuitry, and a memory including one or more storage media for storing instructions, wherein, when the instructions are executed individually or collectively by the at least one processor, the instructions cause the electronic device to: display extended-reality content through a display of the electronic device; identify an environment object on the basis of a user input while displaying the extended-reality content; generate sound data by using the environment object and physical environment information; and reproduce the sound data while displaying the extended-reality content through the display.

Description

Method for generating sound and electronic device for doing so

[0001] One embodiment disclosed in this document relates to an electronic device for generating sound, a method thereof, and a storage medium, and hereinafter, a technology for generating sound for extended reality content is disclosed.

[0002] Extended reality technologies, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), are developing by applying computer graphics technology. VR technology can create a computer-generated virtual space that doesn't exist in the real world, making it feel like reality. AR or MR technologies integrate the real and virtual worlds by overlaying computer-generated information onto the real world, enabling users to interact with both the real and virtual worlds.

[0003] With the VST (video see-through) method, an image of the physical environment is captured using a camera and then displayed on a display screen. Digital information can be superimposed on the captured image. Meanwhile, with the OST (optical see-through) method, digital information can be superimposed on the physical environment while the user directly views it through a transparent display or lens.

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0005] According to one embodiment, an electronic device may include at least one processor including processing circuitry. The electronic device may include a memory including one or more storage media storing instructions. When the instructions are individually or collectively executed by the at least one processor, the electronic device may: display augmented reality content through a display of the electronic device. When the instructions are individually or collectively executed by the at least one processor, the electronic device may: identify an environmental object based on a user input while displaying the augmented reality content. When the instructions are individually or collectively executed by the at least one processor, the electronic device may: generate acoustic data using the environmental object and physical environment information. When the instructions are individually or collectively executed by the at least one processor, the electronic device may: reproduce the acoustic data while displaying the augmented reality content through the display.

[0006] According to one embodiment, a method performed by an electronic device may include an operation of displaying extended reality content through a display of the electronic device. The method performed by the electronic device may include an operation of identifying an environmental object based on a user input while displaying the extended reality content. The method performed by the electronic device may include an operation of generating acoustic data using the environmental object and physical environment information. The method performed by the electronic device may include an operation of playing the acoustic data while displaying the extended reality content through the display.

[0007] According to one embodiment, a non-transitory computer-readable recording medium may store one or more programs including commands. When the commands are individually or collectively executed by at least one processor of an electronic device, the commands may cause the electronic device to: display extended reality content through a display of the electronic device. When the commands are individually or collectively executed by at least one processor of the electronic device, the commands may cause the electronic device to: identify an environmental object based on a user input while displaying the extended reality content. When the commands are individually or collectively executed by at least one processor of the electronic device, the commands may cause the electronic device to: generate acoustic data using the environmental object and physical environment information. When the commands are individually or collectively executed by at least one processor of the electronic device, the commands may cause the electronic device to: reproduce the acoustic data while displaying the extended reality content through the display.

[0008] According to one embodiment, an electronic device may include at least one processor comprising processing circuitry. The electronic device may include a memory comprising one or more storage media storing instructions. When the instructions are individually or collectively executed by the at least one processor, the instructions may cause the electronic device to: display extended reality content through a display of the electronic device. When the instructions are individually or collectively executed by the at least one processor, the instructions may cause the electronic device to: display, through the display, a first user interface object that allows the electronic device to identify an environmental object based on a user input. When the instructions are individually or collectively executed by the at least one processor, the instructions may cause the electronic device to: display, through the display, a second user interface object that allows the electronic device to control reproduction of sound data generated using the environmental object and physical environment information based on a user input.

[0009] In one embodiment, a method performed by an electronic device may include an operation of displaying extended reality content through a display of the electronic device. The method performed by the electronic device may include an operation of displaying, through the display, a first user interface object that causes the electronic device to identify an environmental object based on a user input. The method performed by the electronic device may include an operation of displaying, through the display, a second user interface object that causes the electronic device to control reproduction of sound data generated using the environmental object and physical environment information based on the user input.

[0010] According to one embodiment, a non-transitory computer-readable recording medium may store one or more programs including commands. When the commands are individually or collectively executed by at least one processor of an electronic device, the commands may cause the electronic device to: display extended reality content through a display of the electronic device. When the commands are individually or collectively executed by at least one processor of the electronic device, the commands may cause the electronic device to: display, through the display, a first user interface object that allows the electronic device to identify an environmental object based on a user input. When the commands are individually or collectively executed by at least one processor of the electronic device, the commands may cause the electronic device to display, through the display, a second user interface object that allows the electronic device to control reproduction of sound data generated using the environmental object and physical environment information based on a user input.

[0011] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0012] FIG. 2 is a perspective view illustrating the internal configuration of a wearable electronic device according to one embodiment.

[0013] FIGS. 3A and 3B are drawings showing the front and back of a wearable electronic device according to one embodiment.

[0014] FIG. 4 is a diagram illustrating an artificial intelligence system according to one embodiment.

[0015] Figure 5 is a flowchart of a sound generation method according to one embodiment.

[0016] Fig. 6 is a flowchart of a method for generating sound using characteristic information of an environmental object and physical environment information, according to an example.

[0017] Figure 7a is a diagram illustrating identification of environmental objects according to various examples.

[0018] Figure 7b is a drawing illustrating a display screen according to an example.

[0019] Figure 8 is a drawing explaining sound generation results according to various examples.

[0020] Figure 9 is a drawing illustrating the musical properties of sound according to an example.

[0021] Figure 10 is a flowchart of a sound generation method according to one embodiment.

[0022] Fig. 11 is a drawing illustrating a method for generating sound using a virtual environment object according to an example.

[0023] Figure 12 is a drawing explaining the sound generation results according to various examples.

[0024] Hereinafter, an embodiment of the present document may be described with reference to the attached drawings.

[0025] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0026] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0027] The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0028] The auxiliary processor (123) may control at least a part of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where artificial intelligence is performed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0029] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0030] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0031] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0032] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0033] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0034] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0035] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0036] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0037] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0038] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0039] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0040] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0041] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0042] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, Wi-Fi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0043] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0044] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0045] In one embodiment, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0046] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0047] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service by itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0048] FIG. 2 is a perspective view illustrating the internal configuration of a wearable electronic device according to one embodiment.

[0049] Referring to FIG. 2, a wearable electronic device (200) according to an embodiment may include at least one of a light output module (211), a display member (201), and a camera module (250). An electronic device (e.g., an electronic device (101) of FIG. 1) may be implemented in the form of a wearable electronic device (200). A wearable electronic device (200) (e.g., an electronic device (102) of FIG. 1) may be implemented separately from an electronic device (e.g., an electronic device (101) of FIG. 1). A wearable electronic device (200) (e.g., an electronic device (102) of FIG. 1) may be connected to an electronic device (e.g., an electronic device (101) of FIG. 1).

[0050] According to one embodiment, the light output module (211) may include a light source capable of outputting an image and a lens that guides the image to the display member (201). The output module (211) may include at least one of a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0051] According to one embodiment, the display member (201) may include an optical waveguide (e.g., a waveguide). An output image of an optical output module (211) incident on one end of the optical waveguide may be propagated inside the optical waveguide and provided to a user. The optical waveguide may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or at least one reflective element (e.g., a reflective mirror). For example, the optical waveguide may guide an output image of the optical output module (211) to a user's eye by using at least one diffractive element or reflective element.

[0052] According to one embodiment, the camera module (250) can capture still images and / or videos. The camera module (250) is disposed within the lens frame and may be disposed around the display member (201). The camera module (250) may include a first camera module (251), a second camera module (253), and a third camera module (255).

[0053] According to one embodiment, the first camera module (251) can capture and / or recognize the trajectory of the user's eye (e.g., pupil, iris) or gaze. The first camera module (251) can periodically or aperiodically transmit information related to the trajectory of the user's eye or gaze (e.g., trajectory information) to a processor (e.g., processor (120) of FIG. 1).

[0054] According to one embodiment, the second camera module (253) can capture an external image.

[0055] According to one embodiment, the third camera module (255) can be used for hand detection and tracking, and user gesture (e.g., hand movement) recognition. The third camera module (255) can be used for 3 degrees of freedom (3DoF), 6DoF head tracking, position (spatial, environmental) recognition, and / or movement recognition. The second camera module (253) can also be used for hand detection and tracking, and user gesture recognition. At least one of the first camera module (551) to the third camera module (255) can be replaced with a sensor module (e.g., a LiDAR sensor). For example, the sensor module can include at least one of a vertical cavity surface emitting laser (VCSEL), an infrared sensor, and / or a photodiode.

[0056] FIGS. 3A and 3B are drawings showing the front and back of a wearable electronic device according to one embodiment.

[0057] Referring to FIGS. 3A and 3B, according to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1) may be implemented in the form of a wearable electronic device (300). Camera modules (311, 312, 313, 314, 315, 316) (e.g., camera module (180) of FIG. 1) and / or depth sensor (317) (e.g., sensor module (176) of FIG. 1) for obtaining information related to the surrounding environment of the wearable electronic device (300) may be arranged on a first surface (310) of the housing.

[0058] According to one embodiment, the camera modules (311, 312) can acquire images related to the environment surrounding the wearable electronic device.

[0059] According to one embodiment, the camera modules (313, 314, 315, 316) can acquire images while the wearable electronic device is worn by a user. The camera modules (313, 314, 315, 316) can be used for hand detection, tracking, and recognition of user gestures (e.g., hand movements). The camera modules (313, 314, 315, 316) can be used for 3DoF, 6DoF head tracking, position (spatial, environmental) recognition, and / or movement recognition. The camera modules (311, 312) can also be used for hand detection and tracking, and recognition of user gestures.

[0060] According to one embodiment, the depth sensor (317) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for purposes such as time of flight (TOF) to determine the distance to an object. Instead of or in addition to the depth sensor (317), the camera modules (313, 314, 315, 316) may determine the distance to an object.

[0061] According to one embodiment, a camera module (325, 326) for facial recognition (e.g., camera module (180) of FIG. 1) and / or a display (321) (e.g., display module (160) of FIG. 1) (and / or a lens) may be disposed on the second side (320) of the housing.

[0062] According to one embodiment, a face recognition camera module (325, 326) adjacent to the display may be used to recognize a user's face, or may recognize and / or track both eyes of the user.

[0063] According to one embodiment, the display (321) (and / or lens) may be disposed on the second side (320) of the wearable electronic device (300). The wearable electronic device (300) may not include camera modules (315, 316) among the plurality of camera modules (313, 314, 315, 316). Although not illustrated in FIGS. 3A and 3B , the wearable electronic device (300) may further include at least one of the configurations illustrated in FIG. 2 .

[0064] As described above, according to one embodiment, the wearable electronic device (300) may have a form factor for being worn on a user's head. The wearable electronic device (300) may further include a strap and / or a wearing member for being secured to a body part of the user. The wearable electronic device (300) may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the user's head.

[0065] FIG. 4 is a diagram illustrating an artificial intelligence system according to one embodiment.

[0066] The artificial intelligence system (400) may include a user query / response interface (410), an AI framework (420), a knowledge component (430), an application / service component (440), and / or a generative model (450).

[0067] In an artificial intelligence system (hereinafter, “system”) (400), a user query / response interface (410) can receive input. The input can include user input and / or data obtained or generated by an electronic device (e.g., an electronic device (101) of FIG. 1, a wearable electronic device (200) of FIG. 2, or a wearable electronic device (300) of FIG. 3). The data may include images, videos, and / or sensor data (e.g., illumination data around the electronic device obtained from a sensor or sensor hub (e.g., a coprocessor (123)), posture data (or orientation data) of the electronic device, temperature inside the electronic device (e.g., temperature of the display module (160) or temperature of the at least one processor (120)), size information of a display area of ​​the display module (160), and / or images obtained through an image sensor (e.g., included in a camera module (180)) of the electronic device) generated by at least one processor (e.g., at least one processor (120)) of the electronic device. For example, the user input may be an input of a type such as a natural language, touch data obtained through a touch circuit included in the display module (160) (e.g., used to identify input from a finger and / or a stylus), an image, audio, and / or a video. Additionally, when user input is transmitted, context information may also be transmitted. Context information may include various side information related to the time at which the user input was input into the system (400). For example, this may include information about the application currently being used by the user or information about the user's location.Additionally, user input may be a mixed type of input, including natural language, images, audio, video, and / or contextual information, as described above. Furthermore, user input may include non-natural language input, such as selecting a menu.

[0068] The user query / response interface (410) may provide the user with output from the generative artificial intelligence system. The output may include results (or result information) generated or obtained by the system (400) based at least in part on the input. The output may include a natural language-based response and / or specific content. The output may also include an action requested by the user. For example, the output may have a format based on user settings of the electronic device.

[0069] The AI ​​framework (420) can receive user input. Based on the user input (e.g., the user's query), the AI ​​framework (420) can coordinate and control one or more components necessary to perform an action corresponding to the user's intent.

[0070] User input received from the user query / response interface (410) can be transmitted to a prompt design component (421). The prompt design component (421) can be used to generate a prompt suitable as input to a generative model (450) based on the user input.

[0071] The prompt design component (421) may be an AI component that uses a machine learning algorithm or a neural network. The prompt design component (421) may generate improved prompts through learning over time. The prompt design component (421) may access a knowledge repository (430) to generate prompts based on user input. The knowledge repository (430) may include user preference data, a prompt library, and / or prompt examples. The prompt design component (423) may provide the generated prompts to a generative model (e.g., LLM, LVM, and / or LMM).

[0072] The APIs / Plugins management component (423) can communicate with an external information source based on a request for additional information when user input is transmitted to the generative model (450).

[0073] The APIs / Plugins management component (423) can establish a communication channel for communication with the outside of the system (400) via APIs. The APIs / Plugins management component (423) can enable access to various data sources via the communication channel. For example, the APIs / Plugins management component (423) can be used to request another component (e.g., an application / service component (440)) to perform feedback (or response) according to the prompt. The acquired information can be used to generate a prompt by the prompt design component (421) together with user input, or can be used as input to the generative model (450).

[0074] The APIs / Plugins management component (423) can request a final action via an API when the final action in response to user input, rather than an intermediate action, must be performed by an application or service.

[0075] The refiner component (425) can at least partially tune (or adjust or change) the results (e.g., content) obtained (or output) from the generative model (450). For example, the refiner component (425) can determine the relevance (e.g., score) between the output (e.g., content) of the generative model and a user input. For example, the refiner component (425) can determine whether the output contains biased information (e.g., selective information). For example, the refiner component (425) can determine whether the output contains harmful information (e.g., violent content or profanity).

[0076] The refinement component (425) can determine the degree of matching (e.g., a score) between the output of the generative model (450) and the user input (e.g., the intent of the user input). If the refinement component (425) determines that the output of the generative model (450) does not correspond to the user input, the refinement component (425) can modify the output to correspond to the user input.

[0077] The refinement component (425) can provide hints (e.g., hints for prompt generation) to the user so that the user can obtain information that matches the user's intention from the generative model (450).

[0078] In one embodiment, the generative model (450) may form at least a portion of an artificial intelligence neural network. The generative model (450) may include a model that generates images or a model that generates language. The image generation model may include, for example, a generative adversarial network (GAN), a variational autoencoder (VAE), or a diffusion-based model using a VAE and a transformer. The language generation model may include, for example, a large language model (LLM), a large multimodal model (LMM), a large vision model (LVM), a large vision language model (LVLM), or a large action model (LAM). The LAM may automatically generate actions for an environment (e.g., a robot, a car, an electronic device (101), or a program (140)). Additionally, for at least some AI models (e.g. LLM), there may be a LoRA (low-rank adaptation) adapter that is fine-tuned for a specific task or specific situation, for example.

[0079] Figure 5 is a flowchart of a sound generation method according to one embodiment.

[0080] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0081] According to one embodiment, operations 510 to 540 below may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1 , the wearable electronic device (200) of FIG. 2 , or the wearable electronic device (300) of FIG. 3 ). The electronic device may include at least some of the components of the electronic device (101) described in FIG. 1 . For example, the electronic device may include at least one processor (e.g., the processor (120) of FIG. 1 ) including a processing circuit. The electronic device may include a memory (e.g., the memory (130) of FIG. 1 ) including one or more storage media for storing instructions.

[0082] According to one embodiment, the electronic device may be a device such as a mobile terminal (e.g., a smartphone, tablet, or laptop) or a stationary terminal (e.g., a personal computer (PC)).

[0083] According to one embodiment, the electronic device may include at least a part of the configuration of the wearable electronic device (200) of FIG. 2 and / or the wearable electronic device (300) of FIGS. 3A and 3B. The electronic device may be implemented in the form of a wearable electronic device (e.g., the wearable electronic device (200) of FIG. 2) (e.g., smart glasses including virtual reality glasses). The electronic device may also be implemented in the form of a wearable electronic device (e.g., the wearable electronic device (300) of FIG. 3) (e.g., a head-mounted display (HMD) including an augmented reality (AR) device, a virtual reality (VR) device, and / or a mixed reality (MR) device). The electronic device may be configured to easily control a user interface (UI) component provided in an AR environment, a VR environment, and / or an MR environment.

[0084] According to one embodiment, the electronic device may be of the VST type, which is configured to block external light so that, when worn, light emitted from the display reaches the user's eyes, but external light does not reach the user's eyes. According to one embodiment, the electronic device may include of the OST type, which is configured to allow external light to reach the user's eyes through the glasses when worn.

[0085] The electronic device may include a sensor (e.g., a sensor module (176) of FIG. 1, a camera module (180), a camera module (250) of FIG. 2, a first camera module (251), a second camera module (253), a depth sensor (317) of FIG. 3A, or camera modules (311, 312, 313, 314, 315, 316, 325, 326) of FIGS. 3A and 3B). According to one embodiment, the sensor may measure or sense a physical quantity, thereby converting the measured or sensed information into an electrical signal (or sensed data). For example, the sensor may include at least one camera or image sensor for capturing at least one frame of a still image or a moving image of a real space (or a physical environment). For example, the sensor may include at least one of a button for touch input, a gesture sensor, a gyroscope, a gyro sensor, a barometric pressure sensor, a magnetic sensor, a magnetometer, an acceleration sensor, an accelerometer, a grip sensor, a proximity sensor, an RGB sensor, a biophysical sensor, a temperature sensor, a humidity sensor, an ambient light sensor, an ultraviolet sensor, an electromyography sensor, an electroencephalography sensor, an electrocardiogram sensor, an infrared sensor, an ultrasonic sensor, an iris sensor, or a fingerprint sensor, but the present disclosure is not limited thereto.

[0086] In one embodiment, the sensor may capture a physical environment including an object. For example, the sensor may include at least one of an image sensor, a LiDAR sensor, a red-green-blue depth (RGB-D) sensor, a depth sensor, a time of flight (ToF) sensor, an ultrasonic sensor, a radar sensor, and a stereo camera, but the present disclosure is not limited thereto.

[0087] According to one embodiment, a sensor can generate sensing data. The sensing data can be at least one still image or video of a physical environment. The sensing data can be an image (or real-space image) of one or more objects included in the physical environment. The sensing data can include depth information. For example, the sensing data can be a color image that includes depth information, such as an RGB-D image.

[0088] According to one embodiment, an electronic device can obtain sensing data from a sensor. The electronic device can provide extended reality content using the sensing data obtained from the sensor. The electronic device can generate extended reality content (or an image of the extended reality content) by blending a physical environment and a virtual environment based on the sensing data. The extended reality content can include one or more physical environment objects included in a physical environment captured in real time by the electronic device and / or virtual environment objects such as user interface elements (e.g., an avatar, a control element, an interactive element, or any graphic element).

[0089] In operation 510, the electronic device may display extended reality content through a display of the electronic device.

[0090] According to one embodiment, the extended reality content may include an image of a physical environment including one or more physical environment objects. The electronic device may display an image of a physical environment including one or more physical environment objects based on sensing data acquired from a sensor.

[0091] According to one embodiment, the augmented reality content may include various user interface elements, such as icons or windows of applications (e.g., web browsers, galleries, notes, games, or social networks), notifications, previews, or dialog boxes. The electronic device may display the augmented reality content with the user interface elements overlaid on an image of the physical environment.

[0092] Action 510 may be performed in parallel with actions 520 to 540.

[0093] In operation 520, the electronic device may identify environmental objects based on user input while displaying extended reality content.

[0094] In one embodiment, the environmental object may be a physical environmental object included in an image of a physical environment (e.g., a real space). The electronic device may identify a physical environmental object included in the augmented reality content based on user input.

[0095] In one embodiment, the environment object may be a virtual environment object. The electronic device may identify a virtual environment object included in the augmented reality content based on user input. The virtual environment object may be a user interface element as described above or may be navigated through a user interface element. For example, the virtual environment object may be an object (or image) on a web page displayed through a web browser. The virtual environment object may be an object (or image) displayed through a gallery application. The virtual environment object may be an image of a virtual avatar.

[0096] According to one embodiment, an electronic device can identify an environmental object based on a user's gesture input indicated by the sensing data. For example, the electronic device can identify an environmental object contained in an area based on a user's gesture input designating a specific area, such as drawing a closed path with a finger within a view (or viewport) of an augmented reality content. For example, the electronic device can identify an environmental object corresponding to a specific point based on a user's gesture input designating a specific point, such as pointing at the specific point on the augmented reality content for a specified period of time.

[0097] According to one embodiment, an electronic device can identify an environmental object based on a user's touch input. The electronic device can receive the user's touch input through a display. For example, the electronic device can identify an environmental object included in an area based on a user's touch input that designates a specific area, such as drawing a closed path or designating a closed area through a touch move on the augmented reality content. For example, the electronic device can identify an environmental object corresponding to a specific point based on a user's touch input that designates a specific point, such as long pressing (or holding for a specified period of time after touching down) on the specific point on the augmented reality content.

[0098] In operation 530, the electronic device may generate acoustic data using environmental objects and physical environmental information. The physical environmental information may include at least one of acoustic information, spatial information, or temporal information of an actual space (or physical environment).

[0099] In one embodiment, the electronic device can separate the environmental object identified in operation 520 from the background or other objects. The electronic device can generate acoustic data using the separated environmental object (or an image of the environmental object).

[0100] According to one embodiment, when the electronic device identifies an environmental object based on a user input (e.g., a gesture input or a touch input) that designates a specific area of ​​the extended reality content in operation 520, the electronic device may generate acoustic data using an area containing the identified environmental object (or an image of an area containing the environmental object).

[0101] The electronic device can generate acoustic data using an artificial intelligence model (e.g., the generative model (450) of FIG. 4). The artificial intelligence model can be a multimodal model (e.g., LMM) pre-trained to generate acoustic data based on prompts containing natural language, images, or a combination of natural language and images.

[0102] According to one embodiment, the AI ​​model (or AI neural network) may include various foundation models, such as a language model, a code model, an image model, and / or other AI neural network models. The AI ​​model may include an LLM, an LVM, and / or an LVLM (e.g., a generative model (450) of FIG. 4). For convenience of explanation, the present disclosure will describe LLM and / or LVM as examples.

[0103] The AI ​​model that can be used in this disclosure may include an LLM, an AI neural network-based language model that has learned a large amount of text data through pre-training. The LLM may contain a relatively larger number of parameters (e.g., approximately 10 billion or more) than existing general language models. The LLM may utilize a transformer AI neural network structure based on an attention mechanism.

[0104] In one embodiment, the training of the LLM may include pre-training and / or fine-tuning. Pre-training may involve training the LLM to acquire general language knowledge using a large amount of text data. For example, pre-training may involve self-supervised learning, which predicts the next word in a text string using a previous word string. Fine-tuning may involve training the LLM to be suitable for a specific domain (e.g., chatbot, AI assistant, translation, summary generation, question answering) and / or task. Fine-tuning may involve further training (e.g., supervised learning, adaptive learning) the LLM using a dataset corresponding to the specific domain and / or task based on the pre-trained model. The LLM may perform a task based on text input containing natural language, referred to as a prompt.

[0105] In one embodiment, fine-tuning can be omitted in LLM learning. Users can control the prompts provided to the LLM to improve performance on a desired task. For example, users can control whether the prompts provide additional examples of tasks and / or guidance for performing the task, such as in-context learning, zero-shot learning, and / or few-shot learning. Publicly available LLMs include Bidirectional Encoder Representations from Transformer (BERT) and generative pre-trained transformer (GPT).

[0106] The term "LLM" can refer to the language neural network model itself, but it can also refer to the model of an LLM-based application (e.g., chatbot, AI assistant, translation, summary generation, text classification, sentence generation). For example, an LLM-based chatbot like ChatGPT or an LLM-based translator can also be referred to as "LLM."

[0107] "LLM" may include an inference engine utilizing the LLM neural network model. For example, "inputting an input prompt to the LLM" may mean "inputting the input prompt to an inference engine based on the LLM." For example, "the output of the LLM for the input prompt" may mean the output information of the last neural network layer of the LLM obtained when the input prompt is input to the LLM-based inference engine, and / or the output information modified through additional processing.

[0108] The attention mechanism is a technique that allows an AI model to focus (attention) on important parts of input data. The attention mechanism can be used to predict output data by predicting the extent to which a portion of time-series input data (e.g., time-series input data such as voice or video, or input data of some layers of a neural network) contributes to the output of the intermediate layers and / or the final output of the neural network. While a recurrent neural network (RNN) structure, which sequentially processes each element of a sequence, may exhibit poor prediction performance when there is information dependence between long time-series distances, the attention mechanism can account for information dependence between long time-series distances by controlling the level of weight concentration (attention) within the entire and / or partial context of the input data. A transformer can be configured as an encoder-decoder structure. The encoder can process the input data and output compressed information (e.g., a contextual representation). The decoder can process compressed information and output data in token units. Each encoder and decoder can include an independent attention network, and may further include a cross-attention network connecting the encoder and decoder.

[0109] According to one embodiment, an electronic device may generate a prompt to be provided to an artificial intelligence model using environmental objects and physical environment information. The electronic device may generate the prompt including an instruction or command that causes the artificial intelligence model to generate acoustic data based on the environmental objects and physical environment information. The prompt may include at least the environmental objects and the physical environment information. For example, the prompt may include environmental objects, physical environment information, and natural language requesting the model to generate acoustic data appropriate for the environmental objects and the physical environment information.

[0110] A method for generating acoustic data using environmental objects and physical environmental information is described in detail with reference to FIG. 6.

[0111] In one embodiment, an electronic device may include (or store) an artificial intelligence model. The artificial intelligence model may be embedded (or installed or deployed) in the electronic device. The electronic device may generate acoustic data corresponding to a prompt through on-device artificial intelligence computing using the artificial intelligence model included in the electronic device.

[0112] According to one embodiment, the electronic device may be connected to an external electronic device (e.g., the electronic device (102) of FIG. 1) or server (e.g., the server (108) of FIG. 1) that includes (or stores) an artificial intelligence model, either through direct (e.g., wired) communication or wireless communication (e.g., Bluetooth, Wi-Fi direct, or NFC). The electronic device may offload the generation of acoustic data to the external electronic device or server. The electronic device may transmit environmental objects and physical environmental information (or prompts) to the external electronic device or server. The electronic device may receive acoustic data generated in response to the prompts from the external electronic device or server.

[0113] In operation 540, the electronic device can play the generated audio data while displaying the extended reality content through the display.

[0114] According to one embodiment, the electronic device can generate at least one of an image, a title, a lighting graphic effect, a widget, or an icon corresponding to the sound data using environmental objects and physical environmental information.

[0115] Fig. 6 is a flowchart of a sound generation method using environmental objects and physical environmental information according to an example.

[0116] According to one embodiment, the operations 610 and 620 below may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3). The electronic device may include at least some of the components of the electronic device (101) described in FIG. 1. For example, the electronic device may include at least one processor including a processing circuit (e.g., the processor (120) of FIG. 1). The electronic device may include a memory including one or more storage media for storing instructions (e.g., the memory (130) of FIG. 1).

[0117] According to one embodiment, operation 530 of generating acoustic data of FIG. 5 may include operations 610 and 620.

[0118] In operation 610, the electronic device can generate feature information of an environmental object.

[0119] In one embodiment, the feature information of an environmental object may include classification information of the environmental object. The electronic device may generate classification information of the environmental object using an artificial intelligence model (e.g., a convolutional neural network) pre-trained to output labels by analyzing images.

[0120] According to one embodiment, the feature information of an environmental object may include at least one keyword associated with the environmental object. The electronic device may generate feature information including at least one keyword associated with the environmental object using an artificial intelligence model (e.g., a generative model (450) of FIG. 4) pre-trained to output natural language based on a prompt including an image. The electronic device may generate at least one keyword associated with the environmental object based on the color, pattern, shape, structure, texture, context, and / or classification information of the environmental object.

[0121] In one embodiment, the characteristic information of an environment object may include meta information of the environment object. For example, if a virtual environment object is recognized based on user input to a window of a web browser, the meta information of the environment object may include information such as other images on a web page containing the virtual environment object, the background of the image containing the virtual environment object, text, or the uniform resource locator (URL) of the web page.

[0122] In operation 620, the electronic device can generate acoustic data using environmental objects, feature information of the environmental objects, and physical environmental information.

[0123] For example, the physical environment information may include at least one of acoustic information, spatial information, or temporal information of an actual space (or physical environment).

[0124] For example, acoustic information of a physical environment may include information such as an ambient acoustic signal obtained from an input device of an electronic device (e.g., an input module (150) of FIG. 1), the size of an ambient acoustic signal, or the level of noise.

[0125] For example, spatial information of a physical environment may include information about whether the physical environment is indoors or outdoors, information about the location of the physical environment, or text describing the physical environment, such as illuminance, lighting, or atmosphere. Spatial information of a physical environment may be determined based on an image of the physical environment. An electronic device may acquire spatial information of the physical environment, for example, using an artificial intelligence model pre-trained to output natural language based on an image or a combination of natural language and images.

[0126] Time information in the physical environment may include, for example, a date, a season, an anniversary, a holiday, the current time, or a time of day such as morning, noon, late afternoon, or evening.

[0127] An electronic device can generate acoustic data using an artificial intelligence model. For example, the artificial intelligence model can be a multimodal model (e.g., an LMM) pre-trained to generate acoustic data based on a prompt containing natural language, an image, or a combination of natural language and an image. The electronic device can generate a prompt that includes instructions for the artificial intelligence model to generate acoustic data based on at least environmental objects and physical environment information. The electronic device can then use the artificial intelligence model to generate acoustic data corresponding to the prompt.

[0128] According to one embodiment, an electronic device may generate a prompt to be provided to an artificial intelligence model using environmental objects, feature information of the environmental objects, and physical environmental information. For example, the electronic device may generate a prompt including an instruction or command that causes the artificial intelligence model to generate acoustic data based on the environmental objects, feature information of the environmental objects, and physical environmental information. The prompt may include at least the environmental objects, feature information of the environmental objects, and physical environmental information. For example, the prompt may include natural language requesting the generation of acoustic data suitable for the environmental objects, feature information of the environmental objects, and physical environmental information.

[0129] In one embodiment, an electronic device may include (or store) an artificial intelligence model. The artificial intelligence model may be embedded (or installed or distributed) in the electronic device. The electronic device may generate acoustic data corresponding to a prompt through on-device artificial intelligence computing using the artificial intelligence model included in the electronic device.

[0130] According to one embodiment, the electronic device may be connected to an external electronic device (e.g., the electronic device (102) of FIG. 1) or server (e.g., the server (108) of FIG. 1) that includes (or stores) an artificial intelligence model, either through direct communication or wireless communication. The electronic device may offload the generation of acoustic data to the external electronic device or server. The electronic device may transmit environmental objects, characteristic information of the environmental objects, and physical environmental information (or prompts) to the external electronic device or server. The electronic device may receive acoustic data generated in response to a prompt from the external electronic device or server.

[0131] According to one embodiment, an electronic device can determine musical attributes of audio data based on user input. The electronic device can determine musical attributes based on user input specifying the mood and / or music type (or genre) of the audio data to be generated. The electronic device can generate audio data using environmental objects, physical environmental information, and musical attributes of the audio data. The musical attributes are described in detail with reference to FIG. 9.

[0132] According to one embodiment, an electronic device can generate audio data using environmental objects, physical environmental information, and stored user interest information. The stored user interest information may include information about genres of music frequently listened to by the user, music properties previously designated by the user, or audio data selected from previously generated audio data that the user has favorited or audio data selected from multiple audio data generated simultaneously.

[0133] Figure 7a is a diagram illustrating the identification of environmental objects according to various examples. Figure 7b is a diagram illustrating a display screen according to an example.

[0134] Screens (710, 720, 730, 740) of FIG. 7A and screens (711, 713) of FIG. 7B are examples of screens displayed through a display of an electronic device (e.g., the electronic device (101) of FIG. 1 , the wearable electronic device (200) of FIG. 2 , or the wearable electronic device (300) of FIG. 3 ). The electronic device may include at least a portion of the configuration of the electronic device (101) of FIG. 1 . For example, the electronic device may include at least one processor including a processing circuit (e.g., the processor (120) of FIG. 1 ). The electronic device may include a memory including one or more storage media for storing instructions (e.g., the memory (130) of FIG. 1 ).

[0135] Referring to FIG. 7a, on the screen (710), the electronic device can display extended reality content through the display of the electronic device.

[0136] According to one embodiment, the extended reality content may include an image of a physical environment including one or more physical environment objects. The electronic device may display an image of a physical environment including one or more physical environment objects based on sensing data. For example, referring to screen (710), the extended reality content may include physical environment objects such as a Christmas tree, a sofa, a table, and a window included in the physical environment.

[0137] While displaying augmented reality content, the electronic device can identify environmental objects based on user input. In one embodiment, the environmental objects may be physical environmental objects included in an image of a physical environment (e.g., an actual space). The electronic device can identify physical environmental objects included in the augmented reality content based on user input.

[0138] Referring to FIG. 7B, on the screen (711), the electronic device may display user interface elements providing access to options for functions or applications, such as Wi-Fi, Bluetooth, Do Not Disturb mode, recording, or screen brightness adjustment. The user interface elements may change according to the user's settings. For example, the electronic device may display a user interface element (75) for generating audio data. In response to receiving user input selecting the user interface element (75), the electronic device may initiate an operation for generating audio data.

[0139] In response to receiving user input selecting a user interface element (75), the electronic device may provide, on the screen (713), a guide guiding input for generating acoustic data. For example, the electronic device may output, in text and / or voice form, a guide guiding user input for identifying environmental objects that form the basis for generating acoustic data.

[0140] Referring to FIG. 7A, on the screen (720), the electronic device may identify a physical environment object included in a specific area of ​​the extended reality content based on a user input designating the area. For example, the electronic device may identify a physical environment object included in the area based on a user's gesture input of drawing a closed path with a finger within a view (or viewport) of the extended reality content. For example, the electronic device may identify a physical environment object included in the area based on a user's touch input of drawing a closed path through a touch move on the extended reality content.

[0141] On the screen (730), the electronic device can identify physical environment objects included in a specific area of ​​the augmented reality content based on a user input designating that area. For example, the electronic device can identify physical environment objects included in that area based on a user's touch input designating a closed area through a touch move on the augmented reality content.

[0142] On the screen (740), the electronic device can identify a physical environment object corresponding to a specific point on the extended reality content based on a user input designating the specific point. For example, the electronic device can identify a physical environment object corresponding to the specific point based on a user's gesture input pointing to the specific point on the extended reality content for a specified period of time. For example, the electronic device can identify a physical environment object corresponding to the specific point based on a user's touch input long-pressing (or holding for a specified period of time after touching down) the specific point on the extended reality content.

[0143] Referring to FIG. 7a, in screens (720, 730, 740), the electronic device can identify a physical environment object corresponding to a Christmas tree.

[0144] According to one embodiment, the electronic device may display, via a display, a user interface object (71, 73) that enables the electronic device to identify an environmental object based on a user input.

[0145] On the screen (730), after receiving a user input designating a specific area on the extended reality content, the electronic device can identify a physical environment object included in the area in response to receiving a user input for a user interface object (71). For example, after receiving a user input designating a specific area, the electronic device can identify a physical environment object included in the area in response to receiving a user input selecting 'generate' of the user interface object (71). Then, the electronic device can generate sound data using the identified physical environment object and physical environment information.

[0146] On the screen (740), the electronic device may, after receiving a user input designating a specific point on the extended reality content, identify a physical environment object corresponding to the point in response to receiving the user input for the user interface object (73). For example, the electronic device may, after receiving the user input designating a specific point, identify a physical environment object corresponding to the point in response to receiving a user input selecting 'Create' of the user interface object (73). The electronic device may generate sound data using the identified physical environment object and physical environment information.

[0147] According to one embodiment, the electronic device may use an artificial intelligence model to identify a physical environment object corresponding to a user input from an image of the physical environment obtained from an extended reality content or a sensor of the electronic device (e.g., the sensor module (176) of FIG. 1, the camera module (180) of FIG. 1, the camera module (250) of FIG. 2, the first camera module (251), the second camera module (253), the depth sensor (317) of FIG. 3A, or the camera modules (311, 312, 313, 314, 315, 316, 325, 326) of FIGS. 3A and 3B). The artificial intelligence model may be a model trained to identify objects in the image.

[0148] According to one embodiment, in screen (720), the electronic device may further display a user interface object (not shown) through the display that allows the electronic device to identify an environmental object based on a user input. After receiving a user input designating a specific area on the augmented reality content, the electronic device may identify a physical environmental object included in the area in response to receiving a user input for the user interface object (not shown). Similarly to what was described with reference to screens (730, 740), after receiving a user input designating a specific area, the electronic device may identify a physical environmental object included in the area in response to receiving a user input selecting 'Create' of the user interface object (not shown). Then, the electronic device may generate acoustic data using the identified physical environmental object and physical environmental information.

[0149] Figure 8 is a drawing explaining sound generation results according to various examples.

[0150] Screens (810, 820, 830, 840, 850, 860) are examples of screens displayed through a display of an electronic device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3). The electronic device may include at least a portion of the configuration of the electronic device (101) of FIG. 1. For example, the electronic device may include at least one processor including a processing circuit (e.g., the processor (120) of FIG. 1). The electronic device may include a memory including one or more storage media for storing instructions (e.g., the memory (130) of FIG. 1).

[0151] As described above with reference to FIGS. 5 to 7B, an electronic device can display augmented reality content through its display. While displaying the augmented reality content, the electronic device can identify environmental objects based on user input. In one embodiment, the electronic device can identify physical environmental objects included in a specific area based on user input designating the area. In one embodiment, the electronic device can identify physical environmental objects corresponding to a specific point based on user input designating the point.

[0152] On the screen (810), the electronic device can generate acoustic data using identified physical environment objects and physical environment information.

[0153] On the screen (810), the electronic device can display a user interface object (80). According to one embodiment, the user interface object (80) can indicate a generation status of sound data.

[0154] As described above with reference to FIG. 6, according to one embodiment, the characteristic information of an environmental object may include classification information of the environmental object. According to one embodiment, the characteristic information of the environmental object may include at least one keyword associated with the environmental object. For example, the characteristic information of a physical environmental object corresponding to a Christmas tree identified in FIG. 7A may include classification information such as 'tree' or 'Christmas tree' and keywords such as 'green', 'year-end', 'Christmas', 'winter', 'snow', or 'party'.

[0155] As described above with reference to FIG. 6, the physical environment information may include at least one of acoustic information, spatial information, or temporal information of an actual space (or physical environment).

[0156] According to one embodiment, the electronic device can generate acoustic data using physical environment objects, feature information of the physical environment objects, and physical environment information.

[0157] According to one embodiment, the electronic device can generate sound data using physical environment objects, feature information of the physical environment objects, physical environment information, and music properties of the sound data.

[0158] According to one embodiment, the electronic device can generate acoustic data using physical environment objects, feature information of the physical environment objects, physical environment information, and previously stored user interest information.

[0159] According to one embodiment, the electronic device can generate audio data using any combination of physical environment objects, feature information of the physical environment objects, physical environment information, music properties of audio data, and stored user interest information, including at least the physical environment objects and the physical environment information.

[0160] In screens (820, 830, 840, 850, 860), the electronic device can play generated audio data while displaying extended reality content through the display.

[0161] In one embodiment, the electronic device can adjust the playback of audio data based on physical environment information. For example, if the magnitude of the ambient audio signal obtained from the input device of the electronic device (e.g., the input module (150) of FIG. 1) is small, the electronic device can play the audio data at a low volume.

[0162] On the screen (820, 830, 840, 850, 860), the electronic device can display user interface objects (81, 83, 85, 87, 89).

[0163] According to one embodiment, the user interface objects (81, 83, 85, 89) may indicate the playback status of the audio data. For example, the electronic device may display the user interface objects (81, 83) with animation applied while playing the generated audio data. For example, the electronic device may display the user interface objects (85, 89) that control the playback of the generated audio data.

[0164] According to one embodiment, the user interface object (87) may include user interface elements that provide access to options for functions or applications, such as calls, messages, Wi-Fi, and the gallery. The user interface object (87) may include a user interface element (871) for generating audio data. For example, the electronic device may regenerate audio data based on user input to the user interface element (871). For example, the electronic device may further display an object for controlling playback of the audio data, such as the user interface object (85) on the screen (840), based on user input to the user interface element (871).

[0165] In one embodiment, the electronic device may regenerate acoustic data based on user input to a user interface object (81). For example, the user interface object (81) may have acoustic data generated based on an environmental object overlapping or adjacent to the user interface object (81), and may be intended to reproduce the same.

[0166] According to one embodiment, the electronic device can generate at least one of an image, a title, a lighting graphic effect, a widget, or an icon corresponding to the sound data using physical environment objects and physical environment information.

[0167] According to one embodiment, the electronic device may generate at least one of an image, a title, a lighting graphic effect, a widget, or an icon corresponding to the sound data using any combination of physical environment objects, feature information of the physical environment objects, physical environment information, music properties of the sound data, and stored user interest information, as well as generating sound data.

[0168] According to one embodiment, the electronic device may display at least one of an image, a title, a lighting graphic effect, a widget, or an icon corresponding to the audio data. For example, on the screen (840, 860), the electronic device may display a user interface object (85, 89) including an image and a title corresponding to the audio data.

[0169] Figure 9 is a drawing illustrating the musical properties of sound according to an example.

[0170] The screen (910) is an example of a screen displayed through a display of an electronic device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3). The electronic device may include at least a portion of the configuration of the electronic device (101) of FIG. 1. For example, the electronic device may include at least one processor including a processing circuit (e.g., the processor (120) of FIG. 1). The electronic device may include a memory including one or more storage media for storing instructions (e.g., the memory (130) of FIG. 1).

[0171] As described above with reference to FIGS. 5 to 8, an electronic device can display extended reality content through a display of the electronic device. While displaying the extended reality content, the electronic device can identify environmental objects based on user input. According to one embodiment, the electronic device can identify physical environmental objects included in an area based on user input designating a specific area. According to one embodiment, the electronic device can identify physical environmental objects corresponding to a specific point based on user input designating the point. The electronic device can generate acoustic data using the identified physical environmental objects and physical environmental information.

[0172] According to one embodiment, the electronic device can determine musical properties of audio data based on user input. The acoustic properties of the audio data can include information about the mood (e.g., upbeat, exciting, mysterious, escalating, calm, fast, or grand) and / or the type (or genre) of music (e.g., hip-hop, classical, jazz, house music, EDM, pop, or electronica) of the audio data to be generated. The electronic device can determine musical properties based on user input specifying the mood and / or the type of music of the audio data to be generated.

[0173] In one embodiment, on the screen (910), the electronic device may display a user interface object (91) that causes the electronic device to determine musical properties of the sound data based on user input.

[0174] For example, the electronic device may receive, through the user interface object (91), a user input for selecting at least one of various options regarding the musical properties of the audio data. For example, the electronic device may receive, through the user interface object (91), a user input (e.g., typing or voice) for directly inputting the musical properties of the audio data. The electronic device may determine the musical properties of the audio data based on the user input received through the user interface object (91).

[0175] An electronic device can generate audio data using physical environment objects, physical environment information, and musical properties of audio data.

[0176] According to one embodiment, depending on the type of electronic device (e.g., VST or OST) and / or the type of extended reality content (e.g., MR, VR, or AR), the operations described above with reference to FIGS. 5 to 9 may be performed as is or with at least some modifications.

[0177] According to one embodiment, a VST type electronic device can display extended reality content in which virtual environment objects, such as user interface elements, are overlaid on an image of a physical environment through a display. For example, the extended reality content can be a mixed reality (MR) image in which an image of a physical environment and virtual environment objects are synthesized. In the mixed reality image, virtual environment objects can interact with physical environment objects in a way that, for example, an avatar sits on a real desk or is hidden behind a real object. For example, the extended reality content can be an augmented reality (AR) image in which virtual environment objects are simply overlaid on an image of a physical environment. While displaying the extended reality content, the electronic device can perform the operations described above, such as operations 510 to 540 of FIG. 5 and operations 610 and 620 of FIG. 6.

[0178] According to one embodiment, an electronic device of the VST type can display, through a display, extended reality content corresponding to an image of a virtual environment including virtual environment objects such as user interface elements. The extended reality content can be a virtual reality (VR) image corresponding to an image of a virtual environment including virtual environment objects such as an avatar, a control element, an interactive element, or any graphical element (e.g., a background (or, a map, a game element, or a weather effect). The electronic device can display the virtual reality image as extended reality content through the display of the electronic device. While displaying the extended reality content, the electronic device can identify virtual environment objects based on a user input.

[0179] A VST type electronic device can generate audio data using virtual environment objects and virtual environment information. The virtual environment information can include at least one of audio information, spatial information, or temporal information of the virtual environment. The audio information of the virtual environment can include information such as an audio signal played in the virtual environment, the size of the audio signal, or the level of noise. The spatial information of the virtual environment can include information such as whether the virtual environment is indoors or outdoors, information about the location of the virtual environment, or text describing the virtual environment, such as illuminance, lighting, or atmosphere. The temporal information of the virtual environment can include, for example, a date, a season, an anniversary, a holiday (or a holiday), the current time, or a time zone such as morning, noon, late afternoon, or evening. The electronic device can generate audio data using an artificial intelligence model (e.g., a generative model (450) of FIG. 4). The electronic device can generate a prompt to be provided to the artificial intelligence model using the virtual environment objects and virtual environment information. An electronic device may generate a prompt that includes at least one of a virtual environment object, virtual environment information, a physical environment object, physical environment information, or user-personalized data (e.g., user interest information). The electronic device may then pass the prompt as input to an artificial intelligence model to generate data such as an image, a title, or a lighting graphic effect, in addition to audio data. The artificial intelligence model may include one or more models each corresponding to an output data type, or may be a multi-modal model that generates multiple data types.

[0180] The electronic device may generate audio data corresponding to the prompt, or receive audio data generated in response to the prompt from an external electronic device (e.g., the electronic device (102) of FIG. 1) or server (e.g., the server (108) of FIG. 1) that includes (or stores) an artificial intelligence model. The electronic device may play the generated audio data while displaying the extended reality content through the display.

[0181] According to one embodiment, an OST type electronic device may include a transparent or translucent display (or a holographic display). While a user views a physical environment through the display, the OST type electronic device may display virtual environment objects, such as user interface elements, through the display. Augmented reality content may collectively refer to the physical environment that the user can directly view and virtual environment objects generated and displayed by the electronic device. For example, the augmented reality content may be mixed reality (MR) content or augmented reality (AR) content in which virtual environment objects are overlaid on a physical environment. The electronic device may provide augmented reality content by displaying virtual environment objects through the display of the electronic device.

[0182] An electronic device can obtain sensing data using a camera (e.g., a camera module (180) of FIG. 1, a camera module (250) of FIG. 2, or camera modules (311, 312, 313, 314, 315, 316) of FIG. 3A) and / or a sensor (e.g., a sensor module (176) of FIG. 1 or a depth sensor (317) of FIG. 3A). When providing mixed reality (MR) content, the electronic device can recognize a physical environment based on the sensing data and perform 3D mapping. The electronic device can implement interaction between virtual environment objects and the physical environment based on the results of recognizing and mapping the physical environment. When providing augmented reality (AR) content, the electronic device can recognize a physical environment and a field of view based on the sensing data, thereby placing virtual environment objects at appropriate locations corresponding to the physical environment.

[0183] An OST type electronic device can identify an environmental object based on user input while providing augmented reality content. According to one embodiment, the environmental object may be a physical environmental object included in a physical environment. The electronic device can identify a physical environmental object included in the physical environment based on the aforementioned sensing data. According to one embodiment, the environmental object may be a virtual environmental object. The electronic device can generate audio data using the environmental object and physical environment information. The electronic device can play the generated audio data while providing augmented reality content.

[0184] Figure 10 is a flowchart of a sound generation method according to one embodiment.

[0185] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0186] According to one embodiment, the operations 1010 to 1040 below may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1 , the wearable electronic device (200) of FIG. 2 , or the wearable electronic device (300) of FIG. 3 ). The electronic device may include at least some of the components of the electronic device (101) described in FIG. 1 . For example, the electronic device may include at least one processor (e.g., the processor (120) of FIG. 1 ) including a processing circuit. The electronic device may include a memory (e.g., the memory (130) of FIG. 1 ) including one or more storage media for storing instructions.

[0187] In operation 1010, the electronic device may display extended reality content through a display of the electronic device.

[0188] According to one embodiment, the extended reality content may include an image of a physical environment including one or more physical environment objects. The electronic device may display an image of a physical environment including one or more physical environment objects based on sensing data acquired from a sensor.

[0189] According to one embodiment, the augmented reality content may include various user interface elements, such as icons or windows of applications (e.g., web browsers, galleries, notes, games, or social networks), notifications, previews, or dialog boxes. The electronic device may display the augmented reality content with the user interface elements overlaid on an image of the physical environment.

[0190] In operation 1020, the electronic device may display, through the display, a first user interface object (e.g., user interface object (71, 73) of FIG. 7A) that causes the electronic device to identify an environmental object based on a user input.

[0191] In one embodiment, the environmental object may be a physical environmental object included in an image of a physical environment (e.g., a real space). In one embodiment, the environmental object may be a virtual environmental object. The virtual environmental object may be a user interface element as described above or may be navigated through a user interface element.

[0192] In one embodiment, after receiving a user input (e.g., a gesture input or a touch input) designating a specific area on the extended reality content, the electronic device can, in response to receiving the user input for a first user interface object, identify an environmental object included in the area.

[0193] In one embodiment, after receiving a user input (e.g., a gesture input or a touch input) designating a particular point on the extended reality content, the electronic device can, in response to receiving the user input for a first user interface object, identify an environmental object corresponding to the point.

[0194] The electronic device can generate acoustic data using identified environmental objects and physical environmental information.

[0195] According to one embodiment, the electronic device can generate acoustic data using environmental objects, feature information of the environmental objects, and physical environmental information.

[0196] The electronic device can play the generated audio data while displaying the extended reality content through the display.

[0197] In operation 1030, the electronic device may display, through the display, a second user interface object (e.g., the user interface object (85) of FIG. 8) that causes the electronic device to control playback of sound data generated using environmental objects and physical environmental information based on user input.

[0198] According to one embodiment, operation 1040 below may be optionally performed.

[0199] In operation 1040, the electronic device may display, through the display, a third user interface object (e.g., the user interface object (91) of FIG. 9) that causes the electronic device to determine a musical property of the sound data based on user input.

[0200] For example, the electronic device may receive, through a third user interface object, a user input for selecting at least one of various options regarding the musical properties of the audio data. For example, the electronic device may receive, through the third user interface object, a user input (e.g., typing or voice) for directly inputting the musical properties of the audio data. The electronic device may determine the musical properties of the audio data based on the user input received through the third user interface object.

[0201] According to one embodiment, the electronic device can generate sound data using environmental objects, characteristic information of the environmental objects, physical environmental information, and musical properties of the sound data.

[0202] According to one embodiment, the electronic device can generate acoustic data using environmental objects, feature information of the environmental objects, physical environmental information, and stored user interest information.

[0203] According to one embodiment, the electronic device can generate sound data using any combination of at least the environmental objects and the physical environmental information, among environmental objects, characteristic information of the environmental objects, physical environmental information, musical properties of the acoustic data, and stored user interest information.

[0204] The electronic device can play the generated audio data while displaying the extended reality content through the display.

[0205] Fig. 11 is a drawing illustrating a method for generating sound using a virtual environment object according to an example.

[0206] Screens (1110, 1120, 1130, 1140) are examples of screens displayed through a display of an electronic device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3). The electronic device may include at least a portion of the configuration of the electronic device (101) of FIG. 1. For example, the electronic device may include at least one processor including a processing circuit (e.g., the processor (120) of FIG. 1). The electronic device may include a memory including one or more storage media for storing instructions (e.g., the memory (130) of FIG. 1).

[0207] On the screen (1110), the electronic device can display extended reality content through the display of the electronic device.

[0208] According to one embodiment, the extended reality content may include an image of a physical environment including one or more physical environment objects. The electronic device may display an image of a physical environment including one or more physical environment objects based on sensing data. For example, referring to screen (1110), the extended reality content may include physical environment objects such as a sofa, a table, and a window included in the physical environment.

[0209] According to one embodiment, the augmented reality content may include various user interface elements, such as icons or windows of applications (e.g., web browsers, galleries, notes, games, or social networks), notifications, previews, or dialog boxes. The electronic device may display the augmented reality content with the user interface elements overlaid on an image of a physical environment. For example, referring to screen (1110), the augmented reality content may include a window (1111) of a web browser overlaid on an image of a physical environment.

[0210] An electronic device can identify an environment object based on user input while displaying augmented reality content. In one embodiment, the environment object may be a virtual environment object. The electronic device can identify a virtual environment object included in the augmented reality content based on user input.

[0211] A virtual environment object may be a user interface element as described above, or may be navigated through a user interface element. For example, a virtual environment object may be an object (or image) on a web page displayed through a web browser. A virtual environment object may be an object (or image) displayed through a gallery application. A virtual environment object may be an image of a virtual avatar.

[0212] According to one embodiment, an electronic device may display, through a display, a user interface object (1101) that causes the electronic device to identify an environmental object based on a user input. The user interface object (1101) may be understood to identify, based on the user input, any environmental object as an environmental object to be used for generating acoustic data. Identification of an environmental object by a user input to the user interface object (1101) may trigger generation of acoustic data using the identified environmental object. In response to identifying an environmental object based on the user input to the user interface object (1101), the electronic device may generate acoustic data using the identified environmental object and physical environmental information.

[0213] In one embodiment, on the screen (1120), the electronic device can recognize a virtual environment object (1125) based on a user's gesture input (1121) indicated by the sensing data. For example, the electronic device can recognize a virtual environment object (1125) included in an area based on a user's gesture input (1121) designating a specific area, such as drawing a closed path with a finger on a window (1111) of a web browser within a view (or viewport) of the extended reality content. For example, the electronic device can recognize a virtual environment object (1125) corresponding to a specific point based on a user's gesture input (1121) designating a specific point, such as pointing at the specific point with a finger for a specified period of time on a window (1111) of a web browser within the extended reality content.

[0214] In one embodiment, on the screen (1120), the electronic device can recognize a virtual environment object (1125) based on a user's touch input. For example, the electronic device can identify a virtual environment object (1125) included in an area based on a user's touch input that designates a specific area, such as drawing a closed path or designating a closed area through a touch move on a window (1111) of a web browser on the extended reality content. For example, the electronic device can recognize a virtual environment object (1125) corresponding to a specific point based on a user's touch input that long presses (or holds for a specified period of time after touching down) a specific point on a window (1111) of a web browser on the extended reality content.

[0215] On the screen (1120), the electronic device can recognize a virtual environment object (or image) (1125) on a web page based on a user input to a window (1111) of a web browser. According to one embodiment, the electronic device can separate the recognized virtual environment object (1125) from the background or other objects. The electronic device can display the separated virtual environment object (or image of the virtual environment object) (1125) by overlaying it on a point designated by a user input (e.g., a gesture input or a touch input). The electronic device can move the separated virtual environment object (1125), for example, according to a user's gesture input (1123) of moving a finger within a view of the augmented reality content, or a user's touch input of moving a touch on the augmented reality content.

[0216] In one embodiment, the electronic device may, in response to receiving a user gesture input (1123) for moving a separated virtual environment object (1125) onto a user interface object (1101), identify the virtual environment object (1125) as an environment object to be used for generating acoustic data. The electronic device may generate acoustic data using the identified virtual environment object (1125) and physical environment information.

[0217] According to one embodiment, the electronic device may, in response to receiving a user touch input of dragging and dropping a separated virtual environment object (1125) onto a user interface object (1101), identify the virtual environment object (1125) as an environment object to be used for generating acoustic data. The electronic device may generate acoustic data using the identified virtual environment object (1125) and physical environment information.

[0218] On screen (1130), the electronic device can generate acoustic data using identified virtual environment objects (1125) and physical environment information.

[0219] On the screen (1130), the electronic device may display a user interface object (1103). According to one embodiment, the user interface object (1103) may indicate a generation status of sound data.

[0220] According to one embodiment, the electronic device can generate sound data using a virtual environment object (1125), feature information of the virtual environment object (1125), and physical environment information.

[0221] According to one embodiment, the electronic device can generate sound data using a virtual environment object (1125), feature information of the virtual environment object (1125), physical environment information, and music properties of the sound data.

[0222] According to one embodiment, the electronic device can generate sound data using a virtual environment object (1125), feature information of the virtual environment object (1125), physical environment information, and previously stored user interest information.

[0223] According to one embodiment, the electronic device may generate sound data using any combination of a virtual environment object (1125), feature information of the virtual environment object (1125), physical environment information, music properties of sound data, and stored user interest information, including at least the virtual environment object (1125) and physical environment information.

[0224] On the screen (1140), the electronic device can play the generated audio data while displaying the extended reality content through the display.

[0225] On the screen (1140), the electronic device may display a user interface object (1104). According to one embodiment, the user interface object (1104) may indicate a playback status of audio data. For example, the electronic device may display an animated user interface object (1104) while playing the generated audio data. According to one embodiment, the electronic device may regenerate the audio data based on a user input to the user interface object (1104).

[0226] According to one embodiment, the electronic device may generate at least one of an image, title, lighting graphic effect, widget, or icon corresponding to the sound data using a virtual environment object (1125) and physical environment information.

[0227] The electronic device may generate at least one of an image, title, lighting graphic effect, widget, or icon corresponding to the sound data using any combination of the virtual environment object (1125), feature information of the virtual environment object (1125), physical environment information, music properties of the sound data, and stored user interest information, as well as the virtual environment object (1125) and physical environment information.

[0228] The electronic device may display at least one of an image, title, lighting graphic effect, widget, or icon corresponding to the audio data. For example, on the screen (1140), the electronic device may display a lighting graphic effect corresponding to the audio data.

[0229] According to one embodiment, when displaying a lighting graphic effect corresponding to sound data, the electronic device may display the lighting graphic effect in a part other than a virtual object (e.g., a part corresponding to a physical environment), such as a window (1111) of a web browser.

[0230] The above-described method for generating acoustic data using a virtual environment object (1125) can be substantially equally applied to a physical environment object. For example, an electronic device can recognize a physical environment object based on a user's gesture input (1121) or a user's touch input indicated by sensing data. The electronic device can separate the recognized physical environment object from the background or other objects of the physical environment image. In response to receiving a user's gesture input (1123) for moving the separated physical environment object (or an image of the physical environment object) onto a user interface object (1101) or a user's touch input for dragging and dropping onto the user interface object (1101), the electronic device can identify the physical environment object as an environment object to be used for generating acoustic data. The electronic device can generate acoustic data using the identified physical environment object and physical environment information.

[0231] Figure 12 is a drawing explaining the sound generation results according to various examples.

[0232] Screens (1210, 1220, 1230, 1240) are examples of screens displayed through a display of an electronic device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3). The electronic device may include at least a portion of the configuration of the electronic device (101) of FIG. 1. For example, the electronic device may include at least one processor including a processing circuit (e.g., the processor (120) of FIG. 1). The electronic device may include a memory including one or more storage media for storing instructions (e.g., the memory (130) of FIG. 1).

[0233] As described above with reference to FIGS. 5 to 11, an electronic device can display extended reality content through a display of the electronic device. While displaying the extended reality content, the electronic device can identify an environment object based on a user input. According to one embodiment, the electronic device can recognize a virtual environment object included in a specific area based on a user input designating the area. According to one embodiment, the electronic device can recognize a virtual environment object corresponding to a specific point based on a user input designating the point. Based on the user input, the electronic device can identify the recognized virtual object as an environment object to be used for generating acoustic data.

[0234] On screen (1210), the electronic device can generate acoustic data using identified virtual environment objects and physical environment information.

[0235] According to one embodiment, the electronic device can generate audio data using any combination of virtual environment objects, feature information of the virtual environment objects, physical environment information, music properties of audio data, and previously stored user interest information, including at least the virtual environment objects and the physical environment information.

[0236] On the screen (1210), the electronic device may display a user interface object (1201). According to one embodiment, the user interface object (1201) may indicate a generation status of sound data.

[0237] On screen (1220, 1230, 1240), the electronic device can play generated audio data while displaying extended reality content through the display.

[0238] According to one embodiment, the electronic device can generate at least one of an image, a title, a lighting graphic effect, a widget, or an icon corresponding to the sound data using virtual environment objects and physical environment information.

[0239] The electronic device may generate at least one of an image, title, lighting graphic effect, widget, or icon corresponding to the sound data using any combination of virtual environment objects, feature information of the virtual environment objects, physical environment information, music properties of the sound data, and stored user interest information, as well as generating sound data.

[0240] The electronic device can display at least one of an image, title, lighting graphic effect, widget, or icon corresponding to the audio data.

[0241] On the screen (1220, 1230, 1240), the electronic device can display user interface objects (1202, 1203, 1204).

[0242] On screen (1220), the electronic device can display a user interface object (1202) containing images and titles corresponding to the sound data generated on screen (1210).

[0243] According to one embodiment, on the screen (1230), the electronic device may display a user interface object (1203) including images of previously generated acoustic data together with images corresponding to the acoustic data generated on the screen (1210). The user interface object (1203) may be a widget including a plurality of images each corresponding to at least a portion of the acoustic data generated to date. The electronic device may display images and titles corresponding to each piece of acoustic data based on a user input of swiping the user interface object (1203) left or right. The electronic device may play acoustic data corresponding to the selected image based on a user input of selecting an image.

[0244] For example, in response to receiving a user input, such as a hover input that points to, touches, or places an indicator over a user interface object (1202) on a screen (1220), the electronic device may display a user interface object (1203) on a screen (1230) that includes images relating to previously generated acoustic data.

[0245] According to one embodiment, in the screen (1230), the electronic device may display a user interface object (1203) including images corresponding to each sound data when a plurality of sound data are generated in the screen (1210). The user interface object (1203) may be a widget including images corresponding to each sound data when a plurality of sound data are generated. The electronic device may play sound data corresponding to the selected image based on a user input selecting an arbitrary image.

[0246] For example, on the screen (1240), the electronic device may display a user interface object (1204) indicating the playback status of audio data. While playing the generated audio data, the electronic device may display an animated user interface object (1204). The user interface object (1204) may be an icon corresponding to the generated audio data.

[0247] In one embodiment, the electronic device can store the generated audio data. The electronic device can store the audio data in a storage of the electronic device (e.g., a separate storage of a music application, or a shared media file storage).

[0248] The electronic device may store the audio data in association with at least one of an image, title, lighting graphic effect, widget, or icon corresponding to the audio data.

[0249] For example, an electronic device can store and manage audio data and corresponding images and titles through a music application. The electronic device can display corresponding images and titles in a list of stored audio data. The electronic device can play back audio data based on user input for selecting or playing back the stored audio data through the music application. When playing back audio data stored through the music application, the electronic device can display lighting graphic effects associated with the audio data through the display.

[0250] For example, in response to receiving user input selecting a particular image via a user interface object (1203), the electronic device may play audio data corresponding to the image.

[0251] Electronic devices can store audio data in association with metadata of the audio data. The metadata of the audio data can include any combination of at least the environmental objects and physical environment information among the environmental objects used to generate the audio data, characteristic information of the environmental objects, physical environment information, musical properties of the audio data, and stored user interest information. For example, the electronic device can display the environmental objects (or images of the environmental objects) used to generate the audio data in a list of stored audio data.

[0252] As described with reference to FIG. 6, the characteristic information of an environment object may include meta information of the environment object. When a virtual environment object is recognized based on a user input on a window of a web browser, the meta information of the environment object may include information such as other images on a web page containing the virtual environment object, the background of the image containing the virtual environment object, text, and the URL of the web page. For example, in response to receiving a user input for selecting sound data from a list of stored sound data, the electronic device may display a web page indicated by the URL stored in association with the selected sound data and play the sound data.

[0253] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure pertains.

[0254] According to one embodiment, an electronic device (101, 200, 300) includes at least one processor (120) including processing circuitry; and a memory (130) including one or more storage media storing instructions, wherein when the instructions are individually or collectively executed by the at least one processor (120), the electronic device (101, 200, 300) may cause: displaying an extended reality content through a display (160) of the electronic device (101, 200, 300); identifying an environmental object based on a user input while displaying the extended reality content; generating acoustic data using the environmental object and physical environment information; and playing the acoustic data while displaying the extended reality content through the display (160).

[0255] According to one embodiment, the instructions, when individually or collectively executed by at least one processor (120), may cause the electronic device (101, 200, 300) to: identify a physical environment object included in extended reality content based on user input.

[0256] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101, 200, 300) may: identify a virtual environment object included in extended reality content based on user input.

[0257] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101, 200, 300) may: generate feature information including classification information of an environmental object, and generate acoustic data using the environmental object, the feature information of the environmental object, and the physical environment information.

[0258] According to one embodiment, the physical environment information may include at least one of acoustic information, spatial information, or temporal information of the physical environment.

[0259] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101, 200, 300) may: determine a musical attribute of sound data based on a user input, and generate sound data using environmental objects, physical environmental information, and the musical attribute of the sound data.

[0260] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101, 200, 300) may be caused to: generate acoustic data using environmental objects, physical environmental information, and previously stored user interest information.

[0261] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101, 200, 300) may: generate at least one of an image, a title, a lighting graphic effect, a widget, or an icon corresponding to sound data using environmental objects and physical environmental information.

[0262] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101, 200, 300) may: generate a prompt including instructions to cause an artificial intelligence model (450) to generate acoustic data based on at least environmental objects and physical environmental information, and generate the acoustic data corresponding to the prompt using the artificial intelligence model (450).

[0263] According to one embodiment, an electronic device (101, 200, 300) comprises at least one processor (120) comprising processing circuitry; and a memory (130) including one or more storage media storing instructions, wherein when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101, 200, 300) can: display extended reality content through a display (160) of the electronic device (101, 200, 300), and display a first user interface object (71, 73, 1101) through the display (160) that allows the electronic device (101, 200, 300) to identify an environmental object based on a user input, and display a second user interface object (85) through the display (160) that allows the electronic device (101, 200, 300) to control reproduction of sound data generated using environmental objects and physical environment information based on the user input.

[0264] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101, 200, 300) may: display, via the display (160), a third user interface object (91) that causes the electronic device (101, 200, 300) to determine a musical attribute of sound data based on a user input.

[0265] According to one embodiment, the instructions, when individually or collectively executed by at least one processor (120), may cause the electronic device (101, 200, 300) to: identify an environmental object based on a user input to a first user interface object (71, 73, 1101).

[0266] According to one embodiment, the environmental object may be a physical environmental object included in a physical environmental image of the extended reality content.

[0267] According to one embodiment, the environment object may be a virtual environment object included in a virtual environment image of the extended reality content.

[0268] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101, 200, 300) may: generate acoustic data using environmental objects and physical environmental information.

[0269] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101, 200, 300) may: generate feature information including classification information of an environmental object, and generate acoustic data using the environmental object, the feature information of the environmental object, and the physical environment information.

[0270] According to one embodiment, the physical environment information may include at least one of acoustic information, spatial information, or temporal information of the physical environment.

[0271] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101, 200, 300) may: determine a music attribute based on a user input to a third user interface object (91), and generate sound data using the music attribute of an environmental object, physical environmental information, and sound data.

[0272] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101, 200, 300) may be caused to: generate acoustic data using environmental objects, physical environmental information, and previously stored user interest information.

[0273] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101, 200, 300) may: cause the electronic device (101, 200, 300) to: reproduce audio data while displaying extended reality content via the display (160).

[0274] According to one embodiment, a method performed by an electronic device (101, 200, 300) may include: displaying extended reality content through a display of the electronic device (101, 200, 300); identifying an environmental object based on a user input while displaying the extended reality content; generating sound data using the environmental object and physical environment information; and playing the sound data while displaying the extended reality content through the display.

[0275] According to one embodiment, the act of identifying an environmental object may include an act of identifying a physical environmental object included in the extended reality content based on user input.

[0276] According to one embodiment, the act of identifying an environmental object may include an act of identifying a virtual environmental object (1125) included in the extended reality content based on a user input.

[0277] According to one embodiment, the operation of generating sound data using environmental objects and physical environmental information may include an operation of generating feature information including classification information of the environmental objects; and an operation of generating sound data using environmental objects, feature information of the environmental objects, and physical environmental information.

[0278] According to one embodiment, the physical environment information may include at least one of acoustic information, spatial information, or temporal information of the physical environment.

[0279] According to one embodiment, the operation of generating sound data using environmental objects and physical environmental information may include the operation of determining a musical attribute of the sound data based on a user input; and the operation of generating sound data using the environmental objects, the physical environmental information, and the musical attribute of the sound data.

[0280] According to one embodiment, the operation of generating sound data using environmental objects and physical environmental information may include an operation of generating sound data using environmental objects, physical environmental information, and previously stored user interest information.

[0281] According to one embodiment, the method performed by the electronic device (101, 200, 300) may further include an operation of generating at least one of an image, a title, a lighting graphic effect, a widget, or an icon corresponding to the sound data using environmental objects and physical environmental information.

[0282] According to one embodiment, a method performed by an electronic device (101, 200, 300) may include: displaying extended reality content through a display of the electronic device (101, 200, 300); displaying, through the display, a first user interface object (71, 73, 1101) that causes the electronic device (101, 200, 300) to identify an environmental object based on a user input; and displaying, through the display, a second user interface object (85, 89) that causes the electronic device (101, 200, 300) to control reproduction of sound data generated using environmental objects and physical environment information based on the user input.

[0283] According to one embodiment, the method performed by the electronic device (101, 200, 300) may further include an action of displaying, via the display, a third user interface object (91) that causes the electronic device (101, 200, 300) to determine a musical attribute of the acoustic data based on a user input.

[0284] According to one embodiment, the method performed by the electronic device (101, 200, 300) may further include an action of identifying an environmental object based on a user input to the first user interface object (71, 73, 1101).

[0285] According to one embodiment, the environmental object may be a physical environmental object included in a physical environmental image of the extended reality content.

[0286] According to one embodiment, the environment object may be a virtual environment object included in a virtual environment image of the extended reality content.

[0287] According to one embodiment, the method performed by the electronic device (101, 200, 300) may further include an operation of generating acoustic data using environmental objects and physical environmental information.

[0288] According to one embodiment, the operation of generating sound data using environmental objects and physical environmental information may include an operation of generating feature information including classification information of the environmental objects; and an operation of generating sound data using environmental objects, feature information of the environmental objects, and physical environmental information.

[0289] According to one embodiment, the physical environment information may include at least one of acoustic information, spatial information, or temporal information of the physical environment.

[0290] According to one embodiment, the operation of generating sound data using environmental objects and physical environmental information may include an operation of determining a music property based on a user input to a third user interface object (91); and an operation of generating sound data using the environmental objects, physical environmental information, and the music property of the acoustic data.

[0291] According to one embodiment, the operation of generating sound data using environmental objects and physical environmental information may include an operation of generating sound data using environmental objects, physical environmental information, and previously stored user interest information.

[0292] According to one embodiment, the method performed by the electronic device (101, 200, 300) may further include an operation of playing audio data while displaying extended reality content through the display.

[0293] In one embodiment, a non-transitory computer-readable recording medium stores one or more programs including commands, which, when individually or collectively executed by at least one processor of an electronic device (101, 200, 300), cause the electronic device (101, 200, 300) to: display extended reality content through a display of the electronic device (101, 200, 300); identify an environmental object based on a user input while displaying the extended reality content; generate acoustic data using the environmental object and physical environment information; and reproduce the acoustic data while displaying the extended reality content through the display.

[0294] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains.

[0295] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0296] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0297] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0298] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0299] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0300] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separately arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0301] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0302] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0303] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0304] The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0305] Although the embodiments have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the described embodiments. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0306] Therefore, other implementations, embodiments and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In electronic devices (101, 200, 300), At least one processor (120) comprising processing circuitry; and A memory (130) comprising one or more storage media for storing instructions, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101, 200, 300) causes: Displaying extended reality content through the display (160) of the above electronic device (101, 200, 300), While displaying the above extended reality content, identify environmental objects based on user input, Generate acoustic data using the above environmental objects and physical environment information, While displaying extended reality content through the above display (160), the above sound data is played. To do, Electronic devices (101, 200, 300).

2. In paragraph 1, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101, 200, 300) causes: Identifying physical environment objects included in the extended reality content based on the user input To do, Electronic devices (101, 200, 300).

3. In paragraph 1 or 2, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101, 200, 300) causes: Identifying a virtual environment object (1125) included in the extended reality content based on the user input To do, Electronic devices (101, 200, 300).

4. In any one of paragraphs 1 to 3, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101, 200, 300) causes: Generate feature information including classification information of the above environmental object, Generating the sound data using the above environmental object, characteristic information of the above environmental object, and the physical environmental information. To do, Electronic devices (101, 200, 300).

5. In any one of paragraphs 1 to 4, The above physical environment information Containing at least one of acoustic information, spatial information, or temporal information of the physical environment, Electronic devices (101, 200, 300).

6. In any one of paragraphs 1 to 5, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101, 200, 300) causes: Determine musical attributes of the above sound data based on user input, Generate the sound data using the above environmental object, the physical environmental information, and the music property of the sound data. To do, Electronic devices (101, 200, 300).

7. In any one of paragraphs 1 to 6, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101, 200, 300) causes: Generate the sound data using the above environmental objects, the physical environmental information, and the previously stored user interest information. To do, Electronic devices (101, 200, 300).

8. In paragraphs 1 to 7, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101, 200, 300) causes: Generate a prompt including instructions to cause the artificial intelligence model (450) to generate acoustic data based on at least the environmental objects and the physical environmental information, Generate the above sound data corresponding to the above prompt using the above artificial intelligence model (450). To do, Electronic devices (101, 200, 300).

9. In a method performed by an electronic device (101, 200, 300), An operation of displaying extended reality content through a display (160) of the above electronic device (101, 200, 300); An action of identifying environmental objects based on user input while displaying the above extended reality content; An operation of generating sound data using the above environmental objects and physical environmental information; and An operation of playing the sound data while displaying the extended reality content through the above display (160) including, method.

10. A non-transitory computer-readable recording medium storing one or more programs including commands, which, when individually or collectively executed by at least one processor (120) of an electronic device (101, 200, 300), cause the electronic device (101, 200, 300) to perform the method according to claim 9.

11. In electronic devices (101, 200, 300), At least one processor (120) comprising processing circuitry; and A memory (130) comprising one or more storage media for storing instructions, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101, 200, 300) causes: Displaying extended reality content through the display (160) of the above electronic device (101, 200, 300), Through the display (160), a first user interface object (71, 73, 1101) is displayed that allows the electronic device (101, 200, 300) to identify an environmental object based on a user input, Through the display (160), a second user interface object (85, 89) is displayed to allow the electronic device (101, 200, 300) to control the reproduction of sound data generated using the environmental object and physical environmental information based on user input. To do, Electronic devices (101, 200, 300).

12. In paragraph 11, The above environment object is, A physical environment object included in the physical environment image of the above extended reality content; or A virtual environment object (1125) included in the virtual environment image of the above extended reality content, Electronic devices (101, 200, 300).

13. In any one of paragraphs 11 and 12, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101, 200, 300) causes: Generating sound data using the above environmental objects and physical environmental information - the physical environmental information includes at least one of acoustic information, spatial information, or time information of the physical environment - To do, Electronic devices (101, 200, 300).

14. In any one of paragraphs 11 to 13, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101, 200, 300) causes: Generate feature information including classification information of the above environmental object, Generating the sound data using the above environmental object, characteristic information of the above environmental object, and the physical environmental information. To do, Electronic devices (101, 200, 300).

15. In any one of paragraphs 11 to 14, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101, 200, 300) causes: Through the display (160), a third user interface object (91) is displayed that allows the electronic device (101, 200, 300) to determine a musical attribute of sound data based on a user input, Determine the music property based on user input to the third user interface object (91), Generate the sound data using the above environmental object, the physical environmental information, and the music property of the sound data. To do, Electronic devices (101, 200, 300).

Images