Sound rendering method and electronic equipment

Abstract

The invention provides a sound rendering method, which is applied to first electronic equipment and comprises the following steps: displaying a first virtual scene; first voice input of a user is detected, first sound is obtained, and the first sound comprises first direct sound of the user; first rendering sound is generated according to the first direct sound and the first virtual scene, the first rendering sound comprises first virtual reflected sound, and the first virtual reflected sound is sound formed by reflecting the first direct sound in the first virtual scene; and playing the first rendering sound. In the embodiment of the invention, the electronic equipment can obtain the sound of a user, then render the sound of the user according to a first virtual scene to generate a first rendered sound, and play the first rendered sound, the first rendered sound comprises a first virtual reflected sound brought by the first virtual scene, and after the user hears the first rendered sound, the first rendered sound is reflected by the first virtual scene. The user can better feel that the user is in the first virtual scene, the immersion of the user is improved, and the user experience is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of electronic devices, and more specifically, to a method for rendering sound and an electronic device. Background Art

[0002] At present, when electronic devices are running VR, AR, MR and other services, they use spatial audio and other technologies to play sounds, rendering the sounds in the service to a spatial effect that matches the screen the user is watching or the virtual scene selected by the user. However, in some services, users need to speak and interact with objects in the virtual scene. At this time, the user will hear the direct sound and reflected sound of their speech. The reflected sound is related to the real scene where the user is, and the real scene may be different from the virtual scene, which will cause the user's immersive experience to decrease and reduce the user experience. Therefore, how to render sound has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The present application provides a method for rendering sound and an electronic device, which can obtain the voice of a user, then render the user's voice according to a virtual scene to generate a rendered sound, and play the rendered sound. The rendered sound includes a virtual reflected sound brought by the virtual scene. After hearing the rendered sound, the user can better feel that he is in the virtual scene, which enhances the user's sense of immersion and helps to improve the user experience.

[0004] In a first aspect, a method for rendering sound is provided, which is applied to a first electronic device, and the method includes: displaying a first virtual scene; detecting a first voice input of a user, and obtaining a first sound, the first sound including a first direct sound of the user; generating a first rendered sound according to the first direct sound and the first virtual scene, wherein the first rendered sound includes a first virtual reflected sound, and the first virtual reflected sound is a sound formed by the first direct sound being reflected in the first virtual scene; and playing the first rendered sound.

[0005] In an embodiment of the present application, the electronic device can obtain the user's voice, then render the user's voice according to the first virtual scene to generate a first rendered sound, and play the first rendered sound. Since the first rendered sound is rendered according to the first virtual scene, the first rendered sound includes a first virtual reflected sound brought by the first virtual scene. After hearing the first rendered sound, the user can better feel that he is in the first virtual scene, which enhances the user's sense of immersion and helps to improve the user experience.

[0006] In combination with the first aspect, in some implementations of the first aspect, generating a first rendered sound based on the first direct sound and the first virtual scene includes: determining a first acoustic parameter based on the first virtual scene; and rendering the first direct sound using the first acoustic parameter to generate the first rendered sound.

[0007] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: determining that a preset condition is met, switching the first acoustic parameter to a second acoustic parameter; detecting a second voice input from the user, obtaining a second sound, the second sound including a second direct sound; using the second acoustic parameter to render the second direct sound to generate a second rendered sound; and playing the second rendered sound.

[0008] In the embodiment of the present application, the positions of the virtual scene and the sound source are not static. When the virtual scene, the position of the sound source, and the radiation direction of the sound source change, the electronic device can render the user's voice based on new acoustic parameters to play a second rendered sound. After hearing the second rendered sound, the user can better feel the changes in the virtual scene, thereby enhancing the user's sense of immersion and helping to improve the user experience.

[0009] In combination with the first aspect, in certain implementations of the first aspect, the preset condition is: switching from the first virtual scene to the second virtual scene, or the position and / or radiation direction of the sound source in the first virtual scene changes, wherein the sound source is the direct sound of the user.

[0010] In combination with the first aspect, in certain implementations of the first aspect, before generating the first rendered sound based on the first direct sound and the first virtual scene, the method also includes: obtaining the sound intensity level of the first direct sound; generating the first rendered sound based on the first direct sound and the first virtual scene includes: generating the first rendered sound based on the first direct sound, the sound intensity level of the first direct sound and the first virtual scene.

[0011] It is understandable that the sound intensity level of the first rendering sound is associated with the sound intensity level of the first direct sound. For example, if the sound intensity level of the first direct sound is 60 dB, the sound intensity level of the first rendering sound is 60 dB, and if the sound intensity level of the first direct sound is 40 dB, the sound intensity level of the first rendering sound is 40 dB.

[0012] In an embodiment of the present application, the sound intensity level of the first rendered sound rendered by the electronic device is associated with the first direct sound of the user. When the sound intensity level of the user's direct sound increases, the first rendered sound also increases. When the sound intensity level of the user's direct sound decreases, the first rendered sound also decreases. This can bring the user a better sense of immersion and help improve the user experience.

[0013] In combination with the first aspect, in certain implementations of the first aspect, the first virtual scene includes a first virtual object, which is associated with the user, and the method also includes: obtaining the timbre of the first virtual object; generating a first rendered sound based on the first direct sound and the first virtual scene, including: generating a first rendered sound based on the first direct sound, the timbre of the first virtual object and the first virtual scene, the first rendered sound also including a first virtual direct sound, and the first virtual direct sound is generated based on the first direct sound and the timbre of the first virtual object.

[0014] For example, taking a stage scene as an example, the user corresponds to a virtual singer #1 in the stage scene. When the user sings, the electronic device can obtain the user's direct sound, render the user's direct sound into a virtual direct sound with the timbre of the virtual singer #1, and use the acoustic parameters of the stage scene to render the direct sound to obtain a virtual reflected sound, and then the electronic device plays the virtual direct sound and the virtual reflected sound.

[0015] In an embodiment of the present application, when a user corresponds to a virtual object in a virtual scene, the electronic device can render the user's direct sound with the timbre of the virtual object and the acoustic parameters of the virtual scene, thereby obtaining a virtual direct sound with the timbre of the virtual object and a virtual reflected sound based on the virtual scene, which can enhance the user's sense of immersion in playing the virtual object and help improve the user's experience.

[0016] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: generating a first noise reduction, the first noise reduction being used to offset a first reflected sound and / or noise of a real scene in which the user is located, the first reflected sound being formed by the first direct sound being reflected by the real scene; and playing the first noise reduction.

[0017] In an embodiment of the present application, the electronic device can generate a first noise reduction that cancels out reflected sound and noise, thereby avoiding interference of reflected sound and noise of a real scene on the user, which can bring the user a better sense of immersion and help improve the user experience.

[0018] In combination with the first aspect, in some implementations of the first aspect, generating the first noise reduction includes: acquiring a third acoustic parameter, the third acoustic parameter including an acoustic parameter of the real scene; and rendering the first direct sound according to the third acoustic parameter to generate the first noise reduction.

[0019] In combination with the first aspect, in certain implementations of the first aspect, obtaining the third acoustic parameter includes: obtaining the third acoustic parameter using one or more sensors, and the one or more sensors include one or more of the following: an inertial measurement unit IMU, a speaker, a microphone, an ultrasonic radar, an infrared radar, a millimeter wave radar, and a camera.

[0020] In combination with the first aspect, in some implementations of the first aspect, the first acoustic parameter and the third acoustic parameter further include a head-related transfer function HRTF parameter of the user.

[0021] In combination with the first aspect, in certain implementations of the first aspect, the first electronic device is connected to headphones, and the first electronic device plays the first rendered sound through the headphones. The method also includes: detecting the type of the headphones; when it is determined that the headphones are in-ear headphones or semi-in-ear headphones, adding a first virtual direct sound to the first rendered sound, and the first virtual direct sound is generated based on the first direct sound.

[0022] In an embodiment of the present application, after the electronic device is connected to the earphone, it can detect the type of the earphone and can play the first virtual direct sound according to the type of the earphone to make up for the lack of direct sound in the air conduction path caused by the earphone, which can bring a better sense of immersion to the user and help improve the user experience.

[0023] In combination with the first aspect, in certain implementations of the first aspect, when the earphone is an in-ear earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a first ratio, and when the earphone is a semi-in-ear earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a second ratio, and the first ratio is greater than the second ratio.

[0024] In an embodiment of the present application, after the electronic device is connected to the earphone, it can detect the type of the earphone, and can play the first virtual direct sound and adjust the ratio of the first virtual direct sound to the first virtual reflected sound according to the type of the earphone, which can make up for the lack of direct sound in the air conduction path caused by the earphone, can bring the user a better sense of immersion, and help to improve the user experience.

[0025] In combination with the first aspect, in certain implementations of the first aspect, the first electronic device is connected to headphones, and the first electronic device plays the first rendered sound through the headphones. The first rendered sound also includes a first virtual direct sound, and the first virtual direct sound is generated based on the first direct sound. The method also includes: detecting the type of the headphones; and determining the ratio of the first virtual direct sound to the first virtual reflected sound based on the type of the headphones.

[0026] For example, when the earphone is an in-ear earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a first ratio, and when the earphone is a semi-in-ear earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a second ratio. Since the effect of in-ear isolated air-conducted direct sound is better than that of semi-in-ear earphones, when the earphone is an in-ear earphone, the ratio of the first virtual direct sound added by the first electronic device should be greater than the ratio of the first virtual direct sound added by the first electronic device when the earphone is a semi-in-ear earphone, and thus the first ratio is greater than the second ratio.

[0027] In an embodiment of the present application, after the electronic device is connected to the earphone, it can detect the type of the earphone, and can play the first virtual direct sound and adjust the ratio of the first virtual direct sound to the first virtual reflected sound according to the type of the earphone, which can make up for the lack of direct sound in the air conduction path caused by the earphone, can bring the user a better sense of immersion, and help to improve the user experience.

[0028] In combination with the first aspect, in certain implementations of the first aspect, when the earphone is an in-ear earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a first ratio, when the earphone is a semi-in-ear earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a second ratio, and when the earphone is an open-ear earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a third ratio, and the first ratio is greater than the second ratio, and the second ratio is greater than the third ratio.

[0029] In combination with the first aspect, in certain implementations of the first aspect, the first electronic device is connected to the second electronic device, and the second electronic device displays the first virtual scene. The method also includes: receiving a third rendering sound sent by the second electronic device; and playing the third rendering sound.

[0030] For example, a first electronic device is connected to a second electronic device, and both the first electronic device and the second electronic device display a first virtual scene, user #1 of the first electronic device corresponds to virtual character #1, user #2 of the second electronic device corresponds to virtual character #2, and virtual character #2 is in front of the left of virtual character #1. After the first electronic device receives a third rendered sound associated with user #2 sent by the second electronic device, the third rendered sound can be played through spatial audio technology, so that user #1 feels that user #2 is in front of his left.

[0031] In the embodiment of the present application, the electronic device can also receive and play third-party rendered sounds of other users sent by other electronic devices, so that users can experience multi-person interaction, which can bring users a better sense of immersion and help improve the user experience.

[0032] In combination with the first aspect, in some implementations of the first aspect, the first virtual scene includes virtual scene sound, and the method further includes: playing the virtual scene sound.

[0033] For example, taking the virtual scene as a jungle scene, the jungle scene includes the sound of a lion and the sound of a tiger, and the first electronic device can also play the sound of the lion and the tiger.

[0034] In an embodiment of the present application, when the virtual scene includes virtual scene sounds, the electronic device can also play the sounds of the virtual scene, which can bring a better sense of immersion to the user and help enhance the user experience.

[0035] In combination with the first aspect, in some implementations of the first aspect, the method further includes: synchronously recording or live broadcasting the first virtual scene and the first rendered sound.

[0036] For example, taking the virtual scene as a live selling scene, the user is the selling anchor, the real scene of the selling anchor is a smaller live broadcast room, and the virtual live selling scene is a shopping mall. The first electronic device can broadcast the first virtual scene and the first rendered sound it displays, so that other users watching the live broadcast can feel that the selling anchor is in the shopping mall, and there can be a better live broadcast atmosphere.

[0037] In a second aspect, an electronic device is provided, comprising: one or more processors; one or more memories; the one or more memories storing one or more computer programs, the one or more computer programs comprising instructions, which, when executed by the one or more processors, enable the electronic device to execute the first aspect or any possible implementation of the first aspect.

[0038] According to a third aspect, a computer-readable storage medium is provided, which includes a computer program or instructions. When the computer program or instructions are executed on a computer, the first aspect and any possible implementation method of the first aspect are executed.

[0039] According to a fourth aspect, a computer program product is provided. The computer program product includes a computer program or instructions. When the computer program or instructions are run on a computer, the first aspect and any possible implementation method of the first aspect are executed.

[0040] According to a fifth aspect, a computer program is provided, which, when executed on a computer, enables the method in the first aspect and any possible implementation thereof to be executed.

[0041] In the sixth aspect, an electronic device is provided, which includes modules / units for executing the above aspects or any possible design method of the above aspects; these modules / units can be implemented by hardware, or the corresponding software can be implemented by hardware.

[0042] Among them, for the beneficial effects of the second to sixth aspects, please refer to the beneficial effects of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0044] Figure 2 It is a software structure block diagram of the electronic device provided in the embodiment of the present application.

[0045] Figure 3 It is a schematic diagram of direct sound and reflected sound provided in an embodiment of the present application.

[0046] Figure 4 It is a schematic diagram of the time and loudness of direct sound and reflected sound reaching the user's ears provided by an embodiment of the present application.

[0047] Figure 5 It is a schematic diagram of a VR business scenario provided in an embodiment of the present application.

[0048] Figure 6 It is a schematic flowchart of a method for rendering sound provided in an embodiment of the present application.

[0049] Figure 7 It is a schematic diagram of an application scenario provided by an embodiment of the present application.

[0050] Figure 8 It is a schematic diagram of another application scenario provided by an embodiment of the present application.

[0051] Fig. 9 It is a schematic diagram of another application scenario provided by an embodiment of the present application.

[0052] Fig.10 It is a schematic flowchart of a method for rendering sound provided in an embodiment of the present application.

[0053] Fig.11 It is a schematic flowchart of a method for rendering sound provided in an embodiment of the present application.

[0054] Fig.12 It is a schematic flowchart of a method for rendering sound provided in an embodiment of the present application.

[0055] Fig.13 It is a set of GUIs provided by the embodiments of the present application.

[0056] Fig.14 It is a schematic flowchart of a method for rendering sound provided in an embodiment of the present application.

[0057] Fig.15 It is a set of GUIs provided by the embodiments of the present application.

[0058] Fig.16 It is a schematic flowchart of a method for rendering sound provided in an embodiment of the present application.

[0059] Fig.17 It is a schematic flowchart of a method for rendering sound provided in an embodiment of the present application.

[0060] Fig.18 This is a schematic diagram of the composition of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0062] The terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one, two or more. The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0063] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0064] The following describes electronic devices and embodiments for using such electronic devices. In some embodiments, the electronic device may be a portable electronic device that also includes other functions such as a personal digital assistant and / or a music player function, such as a mobile phone, a tablet computer, a head-mounted display device (e.g., a virtual reality (VR) head-mounted display device, an augmented reality (AR) head-mounted display device, and a mixed reality (MR) head-mounted display device). Exemplary embodiments of portable electronic devices include but are not limited to carrying Or a portable electronic device with other operating systems. The portable electronic device may also be other portable electronic devices, such as a laptop computer, etc. It should also be understood that in some other embodiments, the electronic device may not be a portable electronic device, but a desktop computer.

[0065] For example, Figure 1 1 shows a schematic diagram of the structure of the electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0066] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0067] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0068] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0069] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0070] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0071] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0072] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0073] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0074] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0075] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0076] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-CDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0077] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0078] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0079] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0080] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0081] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0082] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0083] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0084] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0085] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0086] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0087] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0088] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0089] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0090] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0091] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0092] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the alarm application icon, an instruction to create a new alarm is executed.

[0093] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to realize fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering calls, etc. For example, when the mobile phone detects a user's touch operation on the lock screen interface, the mobile phone can collect the user's fingerprint information through the fingerprint sensor 180H, and match the collected fingerprint information with the fingerprint information preset in the mobile phone. If the match is successful, the mobile phone can enter the non-lock screen interface from the lock screen interface.

[0094] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0095] Figure 2 1 is a software structure diagram of the electronic device 100 of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer. The application layer can include a series of application packages.

[0096] like Figure 2 As shown, the application layer may include camera, settings, third-party applications, etc. Among them, the third-party applications may include gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0097] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer may include some predefined functions.

[0098] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0099] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make the data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0100] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc., such as the indication information for prompting a virtual shutter key in the embodiment of the present application. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon can include a view for displaying text and a view for displaying pictures.

[0101] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0102] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0103] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0104] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0105] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0106] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0107] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0108] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0109] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0110] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0111] A 2D graphics engine is a drawing engine for 2D drawings.

[0112] In addition, the system library may also include status monitoring service modules, such as a physical status recognition module for analyzing and identifying user gestures; a sensor service module for monitoring sensor data uploaded by various sensors at the hardware layer to determine the physical status of the electronic device 100.

[0113] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0114] The hardware layer can include various types of sensors, such as Figure 1 The various types of sensors introduced in the present application include acceleration sensors, gyroscope sensors, touch sensors, etc.

[0115] Before introducing the embodiments of the present application, several concepts involved in the embodiments of the present application are first introduced.

[0116] Direct sound: refers to the sound that travels directly from the sound source to the receiver without any reflection.

[0117] Reflected sound: refers to the sound that is reflected by other objects and then propagates to the receiver. It usually arrives 50ms-80ms after the direct sound. The reflected sound can include early reflection and late reverberation.

[0118] Reverberation and echo: Both reverberation and echo refer to the sound that reaches the user after being reflected, but there is a clear difference between reverberation and echo. Reverberation refers to the reflection phenomenon in a relatively small space. The reverberation multipath effect is obvious, the sound transmission path is complex, and the delay is small. Echo refers to the reflection of sound in a relatively open space, with a longer delay, and the echo can be clearly separated from the direct sound.

[0119] Combine the following Figure 3 and Figure 4 A detailed introduction to direct sound and reflected sound is given. Figure 3 A schematic diagram showing direct sound and reflected sound, Figure 4 Schematic diagram showing the time and loudness of direct and reflected sounds reaching the user's ears.

[0120] like Figure 3 As shown, assuming that the user is in a room, the user can hear the sound of his own speech after speaking. The sound of his own speech heard by the user includes direct sound and reflected sound. The direct sound may include direct sound through the bone conduction path, that is, the sound conducted to the user's auditory system through the head bones, and direct sound through the air conduction path, that is, the sound directly transmitted from the user's mouth in the air to the ear. The reflected sound includes the sound emitted from the user's mouth and reflected by the wall and obstacles in the room and then transmitted to the user's ear.

[0121] like Figure 4 As shown, combined with Figure 3 From the description, it is not difficult to conclude that since the reflected sound needs to be reflected by the wall, the reflected sound will reach the user's ears later than the direct sound, and the loudness of the reflected sound will also be lower than the direct sound.

[0122] Reflected sound is related to the environment in which the user is located. For example, when the user is in a small and enclosed environment, the user can hear more reflected sound, and the time interval between the reflected sound and the direct sound is short. For another example, when the user is in a large venue, the user hears more reflected sound, and the time interval between the reflected sound and the direct sound is large. For another example, when the user is in a wide outdoor environment, the user hears less reflected sound. Therefore, reflected sound can help users judge the environment in which they are located.

[0123] At present, when electronic devices are running VR, AR, MR and other services, they will use spatial audio and other technologies to play sounds in the service to render the sounds in the service to a spatial effect that matches the screen the user is watching or the virtual scene selected by the user. However, in some services, users need to speak and interact with objects in the virtual scene. At this time, the user will hear the direct sound and reflected sound of their speech. The reflected sound is related to the real scene where the user is, and the real scene may be different from the virtual scene, which will cause the user's immersive experience to decrease and reduce the user experience. The following will combine Figure 5 Provide explanation.

[0124] Figure 5 A schematic diagram of a VR business scenario is shown.

[0125] like Figure 5 As shown, the user uses an electronic device (e.g., a VR head-mounted display device) in the living room. The electronic device displays a virtual scene 501, which is a virtual outdoor scene, and the outdoor scene also includes an object 502. The electronic device can render the sound of object 502 in combination with the outdoor scene, so that the user can feel that he is in an outdoor scene through the virtual scene 501 and the sound of object 502 displayed by the VR head-mounted display device. However, when the user interacts with the object 502 by speaking, the user will hear the direct sound of his speech and the reflected sound reflected by the living room. The reflected sound reflected by the living room makes the user feel that he is in the living room and cannot be combined with the outdoor scene, which makes the user feel a sense of separation, resulting in a decrease in the sense of immersion and reducing the user's experience. Based on this, a method for rendering sound provided in an embodiment of the present application can render the user's sound in combination with a virtual scene, so that the user's own sound heard by the user is consistent with the virtual scene, bringing a strong sense of immersion and helping to improve the user's experience.

[0126] Figure 6 A schematic flow chart of a method for rendering sound 600 provided in an embodiment of the present application is shown. Figure 6 As shown, the method 600 includes:

[0127] S601, displaying a first virtual scene.

[0128] When the first electronic device runs VR, AR, MR and other services, it can display the first virtual scene corresponding to the above services.

[0129] Exemplarily, the first virtual scene includes but is not limited to a jungle, an indoor scene, a stage, a conference room, etc.

[0130] S602, detecting a first voice input of a user, and acquiring a first sound, where the first sound includes a first direct sound of the user.

[0131] Exemplarily, when the first electronic device detects a first voice input from a user, it may acquire a first sound through a microphone, where the first sound includes a first direct sound from the user.

[0132] Exemplarily, when the first electronic device is connected to an external device with a microphone and detects a first voice input of a user, the first electronic device can obtain a first sound through the external device. For example, the external device can be a headset.

[0133] Exemplarily, when the first electronic device detects a first voice input from a user, it can acquire a first sound through a bone conduction pickup device, and the first sound is the first direct sound of the user.

[0134] Exemplarily, when the first electronic device detects the user's first voice input, it can obtain a first sound through a microphone and a bone conduction pickup device. The first sound includes the user's first direct sound, and the first direct sound includes the bone conduction path direct sound (i.e., the direct sound obtained by the bone conduction pickup device) and the air conduction path direct sound (i.e., the direct sound obtained by the microphone).

[0135] Optionally, in some embodiments, when the first sound is acquired through a microphone, the first sound may also include a first reflected sound, where the first reflected sound is formed when the first direct sound is reflected by a real scene where the user is located.

[0136] S603: Generate a first rendered sound according to the user's first direct sound and the first virtual scene.

[0137] The first electronic device can determine the first acoustic parameter according to the first virtual scene, and then render the first direct sound of the user according to the first acoustic parameter to generate a first rendered sound, wherein the first rendered sound includes a first virtual reflected sound, and the first virtual reflected sound is a sound formed by the first direct sound reflecting in the first virtual scene. The sound intensity level of the first rendered sound can be preset, or can be set by the user, or can be determined according to the sound intensity level of the first direct sound. For a description of the sound intensity level of the first rendered sound being determined according to the sound intensity level of the first direct sound, please refer to the following description of Fig.11 The description of is not elaborated here.

[0138] Optionally, in some embodiments, the first rendered sound further includes a first virtual direct sound. The virtual direct sound in the embodiment of the present application can be understood as the electronic device obtaining the user's direct sound and then playing the user's direct sound according to the user's timbre or the timbre of the virtual object.

[0139] When the first sound includes both the first direct sound and the first reflected sound, since there is a correlation between the first direct sound and the first reflected sound, the first direct sound can be separated by the correlation, or the first direct sound can be separated according to the acoustic parameters of the actual scene where the user is located.

[0140] The first acoustic parameter may be used to characterize the first virtual scene. The first acoustic parameter includes but is not limited to reverberation parameter, direct-to-reverberant ratio (DRR), echo parameter, reflection density, envelopment, speech intelligibility (SI), and impulse response (IR).

[0141] The first acoustic parameter may be preset or may be calculated in real time. The following description will be made by taking the real-time calculation of the reverberation parameter as an example.

[0142] When calculating reverberation parameters in real time, the reflection path can be simulated by the position of the sound source, the radiation direction of the sound source, and the three-dimensional geometric information and material information of the first virtual scene, or the reverberation parameters can be calculated by estimating the geometric parameters and material parameters of the virtual scene.

[0143] Radiation direction

[0144] In the above description, it is necessary to determine the position of the sound source and the radiation direction of the sound source. In view of this, the embodiments of the present application provide the following two possible implementation methods.

[0145] In one possible implementation, the position of the sound source and the radiation direction of the sound source are preset, that is, after the first electronic device obtains the user's first sound, it can simulate that the user's first sound is emitted from a preset position of the first virtual scene and propagates in a preset radiation direction.

[0146] In a possible implementation, the first virtual scene includes a virtual character corresponding to the user, and the position of the sound source and the radiation direction of the sound source are determined according to the position and mouth orientation of the virtual character.

[0147] It should be noted that the above method for calculating reverberation parameters is only an example and should not be understood as a specific limitation on the embodiments of the present application. The embodiments of the present application may also adopt other methods to calculate reverberation parameters.

[0148] After determining the first acoustic parameter and the user's voice, the first electronic device can render the user's voice based on the first acoustic parameter. The rendering method is not specifically limited in the embodiment of the present application. For example, the electronic device can separate the angle and distance and use the head related transfer function (HRTF) and the room impulse response (RIR) for rendering. For another example, the first electronic device can use binaural impulse response (BIR) for rendering.

[0149] S604, playing the first rendering sound.

[0150] Exemplarily, the first electronic device may play the first rendered sound through a speaker.

[0151] Exemplarily, when the first electronic device is connected to an external device with a speaker, the first electronic device can play the first rendered sound through the speaker.

[0152] Exemplarily, when the first electronic device has a bone conduction function, the first rendered sound can be played by bone conduction.

[0153] Exemplarily, when the first electronic device is connected to an external device with a bone conduction function, the first electronic device can play the first rendered sound through the external device in a bone conduction manner, and the external device can be a bone conduction headset.

[0154] In an embodiment of the present application, the electronic device can obtain the user's voice, then render the user's voice according to the acoustic parameters of the virtual scene to generate a first rendered sound, and play the first rendered sound. Since the first rendered sound is rendered according to the first virtual scene, the first rendered sound includes a first virtual reflected sound brought by the first virtual scene. After hearing the first rendered sound, the user can better feel that he is in the first virtual scene, which enhances the user's sense of immersion and helps to improve the user experience.

[0155] In order to more clearly introduce the method for rendering sound provided by the embodiment of the present application, the following will be combined with Figure 7-9 Taking a specific application scenario as an example, the method for rendering sound provided by an embodiment of the present application is introduced.

[0156] Figure 7 A schematic diagram of an application scenario provided by an embodiment of the present application is shown.

[0157] like Figure 7As shown, the application scenario is a game scenario, and the virtual scenario corresponding to the game scenario is a jungle, but the real scenario the user is in is a car. When the game setting requires the user to speak, the first electronic device can obtain the user's voice, and then render the user's voice according to the acoustic parameters of the jungle, and play the rendered voice, so that the user can hear his own speech matching the jungle.

[0158] Figure 8 A schematic diagram showing another application scenario provided by an embodiment of the present application is shown.

[0159] like Figure 8 As shown, the application scenario is a singing scenario, and the virtual scenario corresponding to the singing scenario is a stage, but the real scenario the user is in is a bedroom. When the user starts singing, the first electronic device can obtain the user's singing voice, and then render the user's singing voice according to the acoustic parameters of the stage, and play the rendered singing voice, so that the user can feel that he is on the stage.

[0160] It should be noted that the purpose of this rendered singing is to supplement the reflected sound of the stage, rather than to reproduce it through the speakers to increase the intensity of the direct sound.

[0161] Fig. 9 A schematic diagram showing another application scenario provided by an embodiment of the present application is shown.

[0162] like Fig. 9 As shown, the application scenario is a meeting scenario, and the virtual scenario corresponding to the meeting scenario is a meeting room, but the real scenario the user is in is a grassland. When the user needs to speak during the meeting, the first electronic device can obtain the user's voice, and then render the user's voice according to the acoustic parameters of the meeting room, and play the rendered voice. When the user hears the rendered voice, he can feel that he is in the meeting room.

[0163] When electronic devices are performing VR, AR, MR and other services, the virtual scene may change, or in the same virtual scene, the user's voice is used as the sound source, and the position and radiation direction of the sound source may also change.

[0164] For example, the virtual scene changes from a jungle scene to an indoor scene. For another example, taking the virtual scene as a stage scene, the sound source is located at position #1 of the stage at time #1, and is located at position #2 of the stage at time #2.

[0165] It is understandable that when the above changes occur, when the user's voice is acquired again, it is necessary to render the user's voice using the changed acoustic parameters. Fig.10 Provide explanation.

[0166] like Fig.10As shown, S603, generating a first rendered sound according to the first direct sound of the user and the first virtual scene, includes:

[0167] S6031: Determine a first acoustic parameter according to the first virtual scene.

[0168] S6032: Render the first direct sound using the first acoustic parameter to generate a first rendered sound.

[0169] The method 600 further includes:

[0170] S605: Determine whether a preset condition is met, and switch the first acoustic parameter to a second acoustic parameter.

[0171] Exemplarily, the preset condition is that the first virtual scene changes to the second virtual scene.

[0172] For example, taking a game scene as an example, as the game progresses, the virtual scene in the game changes from a jungle scene to an indoor scene, and the first electronic device can obtain the second acoustic parameters corresponding to the indoor scene.

[0173] Exemplarily, the preset condition is that the position of the sound source and / or the radiation direction of the sound source changes.

[0174] For example, Figure 8 As shown, taking the virtual scene as a stage scene as an example, the stage scene includes a virtual character #1, and the virtual character #1 corresponds to the user. The virtual character #1 is located at position #1 of the stage at time #1, and is located at position #2 of the stage at time #2, that is, the position of the sound source has changed.

[0175] For another example, taking the virtual scene as a conference scene, the conference scene includes a virtual character #1, which corresponds to the user. The virtual character #1 sits on a chair and speaks at time #1, and stands up from the chair and speaks at time #2, that is, the position of the sound source has changed.

[0176] For another example, taking the virtual scene as a conference scene, the conference scene includes virtual character #1, virtual character #2 and virtual character #3. The virtual character #1 corresponds to the user. The virtual character #1 speaks to the virtual character #2 at time #1 and speaks to the virtual character #3 at time #2, that is, the radiation direction of the sound source has changed.

[0177] It can be understood that the second acoustic parameter can be preset or calculated in real time. The description of the second acoustic parameter can refer to the description of the first acoustic parameter, and for the sake of brevity, it will not be repeated here.

[0178] S606: Detect a second voice input of the user, and obtain a second sound, where the second sound includes a second direct sound of the user.

[0179] Optionally, in some embodiments, when the second sound is acquired through a microphone, the second sound may also include a second reflected sound, where the second reflected sound is formed when the second direct sound is reflected by a real scene where the user is located.

[0180] S607: Generate a second rendered sound according to the second direct sound of the user and the second acoustic parameter.

[0181] S608, playing the second rendering sound.

[0182] It should be understood that the description of S606-S608 can refer to the description of S602-S604, and for the sake of brevity, it will not be repeated here.

[0183] In the embodiment of the present application, the positions of the virtual scene and the sound source are not static. When the virtual scene, the position of the sound source, and the radiation direction of the sound source change, the electronic device can render the user's voice based on new acoustic parameters to play a second rendered sound. After hearing the second rendered sound, the user can better feel the changes in the virtual scene, thereby enhancing the user's sense of immersion and helping to improve the user experience.

[0184] The ratio of direct sound and reflected sound heard by users in real scenes is certain, for example, the ratio of direct sound to reflected sound is 5:1. Therefore, in the same real scene, when the sound intensity of the direct sound increases, the sound intensity of the reflected sound will also increase. For example, when a user is speaking in a room, when the user increases the volume of his or her speech, the direct sound of his or her own speech heard by the user will become louder, and similarly, the reflected sound will also become louder in the same proportion. Based on this, in order to ensure the user's sense of immersion in the virtual scene and the authenticity of the rendered sound, the electronic device can obtain the sound intensity level of the direct sound, and then when rendering the sound, the rendered sound can be made to match the sound intensity level of the direct sound. The following text combines Fig.11 Provide explanation.

[0185] Optionally, in some embodiments, Fig.11 As shown, the method 600 further includes:

[0186] S609: Obtain the sound intensity level of the first direct sound.

[0187] Exemplarily, when the first electronic device acquires the first sound through the microphone, it can determine the sound intensity level of the first sound, and then determine the sound intensity level of the first direct sound according to the sound intensity level of the first sound.

[0188] Exemplarily, when the first electronic device acquires the first sound through the bone conduction pickup device, the first sound is the first direct sound, and the sound intensity level of the first sound is the sound intensity level of the first direct sound.

[0189] Exemplarily, when the first electronic device obtains the first sound through a microphone and a bone conduction pickup device, the first direct sound includes the direct sound of the bone conduction path and the direct sound of the air conduction path. The first electronic device can respectively determine the sound intensity levels of the direct sound of the bone conduction path and the direct sound of the air conduction path, and then determine the sound intensity level of the first direct sound according to the sound intensity levels of the direct sound of the bone conduction path and the direct sound of the air conduction path.

[0190] S603, generating a first rendered sound according to the first direct sound of the user and the first virtual scene, including:

[0191] S6033: Generate a first rendering sound according to the first direct sound of the user, the sound intensity level of the first direct sound, and the first virtual scene.

[0192] After the first electronic device determines the sound intensity level of the first direct sound, it can use the first acoustic parameters of the first virtual scene to render the first direct sound to generate a first rendered sound. Then, when determining the sound intensity level of the first rendering, the sound intensity level of the first rendered sound can be determined based on the sound intensity level of the first direct sound.

[0193] It is understandable that the sound intensity level of the first rendering sound is associated with the sound intensity level of the first direct sound. For example, if the sound intensity level of the first direct sound is 60 dB, the sound intensity level of the first rendering sound is 60 dB, and if the sound intensity level of the first direct sound is 40 dB, the sound intensity level of the first rendering sound is 40 dB.

[0194] In an embodiment of the present application, the sound intensity level of the first rendered sound rendered by the electronic device is associated with the first direct sound of the user. When the sound intensity level of the user's direct sound increases, the first rendered sound also increases. When the sound intensity level of the user's direct sound decreases, the first rendered sound also decreases. This can bring the user a better sense of immersion and help improve the user experience.

[0195] As mentioned above, in some scenarios, a user may have a virtual object in the virtual scene, including but not limited to a virtual character or a virtual animal. The timbre of the virtual object may be different from that of the user. If the user's timbre is still used during rendering, it may still lead to a decrease in the user's sense of immersion. Therefore, when rendering, the electronic device may render with the timbre of the virtual object to ensure the user's sense of immersion. Fig.12 Provide explanation.

[0196] Optionally, in some embodiments, Fig.12 As shown, the method 600 further includes:

[0197] S610: Acquire a timbre of a first virtual object, where the first virtual object is a virtual object corresponding to the user.

[0198] S603, generating a first rendered sound according to the first direct sound of the user and the first virtual scene, including:

[0199] S6034: Generate a first rendered sound according to the first direct sound of the user, the timbre of the first virtual object, and the first virtual scene.

[0200] The first electronic device can render the first direct sound using the first acoustic parameter of the first virtual scene and the timbre of the first virtual object, thereby obtaining a first rendered sound. The first rendered sound also includes a first virtual direct sound, which is generated based on the first direct sound and the timbre of the first virtual object. In other words, after determining the timbre of the first virtual object, the timbre of the user in the first direct sound can be replaced with the timbre of the first virtual object to generate the first virtual direct sound.

[0201] For example, taking a stage scene as an example, the user corresponds to a virtual singer #1 in the stage scene. When the user sings, the electronic device can obtain the user's direct sound, render the user's direct sound into a virtual direct sound with the timbre of the virtual singer #1, and use the acoustic parameters of the stage scene to render the direct sound to obtain a virtual reflected sound, and then the electronic device plays the virtual direct sound and the virtual reflected sound.

[0202] It should be noted that there is no actual execution order between S602 and S610. In other words, S602 may be executed first and then S610, or S610 may be executed first and then S602, or S602 and S610 may be executed simultaneously.

[0203] In an embodiment of the present application, when a user corresponds to a virtual object in a virtual scene, the electronic device can render the user's direct sound with the timbre of the virtual object and the acoustic parameters of the virtual scene, thereby obtaining a virtual direct sound with the timbre of the virtual object and a virtual reflected sound based on the virtual scene, which can enhance the user's sense of immersion in playing the virtual object and help improve the user's experience.

[0204] In some embodiments, when a user corresponds to a virtual object in a virtual scene, the first electronic device can automatically obtain the timbre of the virtual object, and then render the user's direct sound with the timbre of the virtual object and the acoustic parameters of the virtual scene.

[0205] In other embodiments, the first electronic device may also respond to the user's operation of selecting the timbre of the virtual object, and then render the user's direct sound with the timbre of the virtual object and the acoustic parameters of the virtual scene.

[0206] For example, Fig.13In the GUI shown, the electronic device displays a virtual scene 1301, which is a stage scene and includes a virtual object 1302. When displaying the virtual scene 1302, the electronic device may also display an option box 1303, which includes a user timbre option 1304 and a virtual object timbre option 1305. The electronic device detects that the user clicks on the virtual object timbre option 1305, and may render the user's direct sound with the timbre of the virtual object 1302 and the virtual scene 1301 when rendering the user's direct sound.

[0207] Optionally, in some other embodiments, the timbre of the first virtual direct sound is the timbre of the user.

[0208] It should be noted that, when the timbre of the first virtual direct sound is the timbre of the virtual object, the ratio of the first virtual direct sound to the first virtual reflected sound is a first ratio; when the timbre of the first virtual direct sound is the timbre of the user, the ratio of the first virtual direct sound to the first virtual reflected sound is a second ratio; the first ratio may be greater than the second ratio.

[0209] The user will hear the first reflected sound and / or the noise of the real scene (for example, the sound of other people talking in the real scene) in the actual real scene. The first reflected sound and noise will affect the user's sense of immersion. Therefore, in some embodiments of the present application, the electronic device can perform noise reduction processing, which will be described below for details.

[0210] like Fig.14 As shown, the method 600 further includes:

[0211] S611, generating a first noise reduction.

[0212] S612, playing the first noise reduction.

[0213] The electronic device may play the first noise reduction sound while playing the first rendering sound.

[0214] It should be noted that there is no actual execution order for S611 and S601 - S603 .

[0215] The first electronic device generating the first noise reduction may include the following possible implementations:

[0216] In one possible implementation, the first sound acquired by the first electronic device includes a first reflected sound, and the first electronic device generates a sound wave in the opposite direction of the first reflected sound according to a preset noise reduction algorithm and the first reflected sound, i.e., a first noise reduction, and the first noise reduction is used to offset the first reflected sound.

[0217] In one possible implementation, the first sound acquired by the first electronic device includes a first reflected sound and noise. The first electronic device generates a sound wave in the opposite direction to the first reflected sound and noise according to a preset noise reduction algorithm, the first reflected sound and noise, i.e., a first noise reduction. The first noise reduction is used to offset the first reflected sound and noise.

[0218] In one possible implementation, the first sound acquired by the first electronic device includes a first reflected sound. The first electronic device acquires acoustic parameters of a real scene, and then renders the first direct sound according to the acoustic parameters of the real scene to generate a first noise reduction, which is used to offset the first reflected sound.

[0219] The acoustic parameters of the real scene include but are not limited to: reverberation parameters, DRR, echo parameters, reflection density, envelopment, SI and IR.

[0220] The embodiment of the present application does not limit the manner in which the electronic device obtains the acoustic parameters of the real scene, and several possible implementation methods are given as examples below.

[0221] In a possible implementation manner, the first electronic device is preset with acoustic parameters of multiple real scenes, and the electronic device can obtain corresponding acoustic parameters according to the user's selection.

[0222] For example, Fig.15 As shown, the first electronic device is preset with acoustic parameters of a noisy car, acoustic parameters of a bedroom, acoustic parameters of a living room, and acoustic parameters of the outdoors. When displaying a virtual scene 1501, the electronic device may display an option box 1502, which includes a noisy car option 1503, a bedroom option 1504, an outdoors option 1505, and a living room option 1506. The electronic device may determine the acoustic parameters corresponding to the option according to the user's operation of selecting the option.

[0223] In a possible implementation, the first electronic device obtains acoustic parameters of a real scene through one or more sensors, including but not limited to: an inertial measurement unit (IMU), a speaker, a microphone, an ultrasonic radar, an infrared radar, a millimeter wave radar, and a camera.

[0224] Taking the example that the one or more sensors include a speaker and a microphone, the first electronic device can play audio of a specific frequency through the speaker and receive the audio of the specific frequency through the microphone, and then determine the acoustic parameters based on the audio of the specific frequency played by the speaker and the audio of the specific frequency received by the microphone.

[0225] Taking the example that the one or more sensors include a camera, the first electronic device can provide the camera to obtain three-dimensional size information of the real scene, and then the electronic device can determine the acoustic parameters according to the three-dimensional size information.

[0226] In a possible implementation manner, the first electronic device may obtain acoustic parameters of a real scene based on big data statistics.

[0227] For example, when the first electronic device executes VR service #1, it can communicate with the server, which stores the acoustic parameters of the real scene corresponding to VR service #1, and the first electronic device can receive the parameters of the real scene corresponding to VR service #1 sent by the server. The acoustic parameters of the real scene corresponding to VR service #1 stored by the server can be determined based on the acoustic parameters uploaded by many users.

[0228] In a possible implementation manner, the first electronic device obtains acoustic parameters of a real scene according to system information.

[0229] For example, if the first electronic device is preset with outdoor acoustic parameters, the first electronic device can determine that the user is outdoors through positioning information, and the first electronic device can obtain the outdoor acoustic parameters.

[0230] Optionally, in some embodiments, the first electronic device may also obtain HRTF parameters of the user, and then generate the first noise reduction according to the acoustic parameters of the real scene and the HRTF parameters of the user.

[0231] In an embodiment of the present application, the electronic device can generate a first noise reduction that cancels out reflected sound and noise, thereby avoiding interference of reflected sound and noise of a real scene on the user, which can bring the user a better sense of immersion and help improve the user experience.

[0232] As shown above, the first electronic device can be connected to an external device with a speaker and a microphone to obtain and play corresponding sounds, and the external device can be a headset. The headset can include in-ear headphones, semi-in-ear headphones, and open headphones. It is not difficult to understand that when the headset is an in-ear headset, the direct sound of the air conduction path will be isolated, and when the headset is a semi-in-ear headset, part of the direct sound of the air conduction path will be isolated. Due to the isolation or partial isolation of the direct sound of the air conduction path, the proportion of virtual reflected sound in the sound heard by the user will increase, which will lead to a decrease in immersion. Therefore, the first electronic device can play virtual direct sound, or increase the proportion of virtual direct sound in the rendered sound when it is determined that the connected external device is a headset, and the headset is an in-ear headset and a semi-in-ear headset. The following will be combined with Fig.16 Provide detailed explanation.

[0233] Optionally, in some embodiments, Fig.16As shown, before executing S604, the method 600 further includes:

[0234] S613, detecting the type of the earphone.

[0235] The first electronic device can detect the type of headphones connected to it. When it is determined that the type of headphones is in-ear headphones or semi-in-ear headphones and the first rendered sound does not include virtual direct sound, S614 can be executed. When the first rendered sound includes virtual direct sound, S615 can be executed.

[0236] S614: Add a first virtual direct sound to the first rendered sound, where the first virtual direct sound is generated based on the first direct sound.

[0237] When the first electronic device determines that the type of the earphone is an in-ear earphone or a semi-in-ear earphone, a first virtual direct sound can be generated, and the first virtual direct sound is generated based on the first direct sound, that is, after obtaining the first direct sound, the first direct sound can be played. It should be noted that the timbre of the first virtual direct sound can be the timbre of the user, or can also be the timbre of the virtual object.

[0238] In some embodiments, when the first electronic device adds the first virtual direct sound to the first rendered sound, it may also determine the ratio of the first virtual direct sound to the first virtual reflected sound according to the type of headphones.

[0239] For example, when the earphone is an in-ear earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a first ratio, and when the earphone is a semi-in-ear earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a second ratio. Since the effect of in-ear isolated air-conducted direct sound is better than that of semi-in-ear earphones, when the earphone is an in-ear earphone, the ratio of the first virtual direct sound added by the first electronic device should be greater than the ratio of the first virtual direct sound added by the first electronic device when the earphone is a semi-in-ear earphone, and thus the first ratio is greater than the second ratio.

[0240] In other embodiments, when the first electronic device adds the first virtual direct sound to the first rendered sound, the ratio of the first virtual direct sound to the first virtual reflected sound is preset. In other words, in these embodiments, the ratio of the first virtual direct sound to the first virtual reflected sound does not change with the change of the headphone type.

[0241] S615: Determine a ratio of the first virtual direct sound to the first virtual reflected sound according to the type of the earphone.

[0242] When the first rendered sound includes the first virtual direct sound, the first electronic device may determine a ratio of the first virtual direct sound to the first virtual reflected sound according to the type of the earphone.

[0243] In some embodiments, when the earphone is an in-ear earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a first ratio.

[0244] In some embodiments, when the earphone is a semi-in-ear earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a second ratio.

[0245] In some embodiments, when the earphone is an open-ear earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a third ratio.

[0246] It can be understood that the first ratio is greater than the second ratio, and the second ratio is greater than the third ratio.

[0247] In an embodiment of the present application, after the electronic device is connected to the earphone, it can detect the type of the earphone, and can play the first virtual direct sound and adjust the ratio of the first virtual direct sound to the first virtual reflected sound according to the type of the earphone, which can make up for the lack of direct sound in the air conduction path caused by the earphone, can bring the user a better sense of immersion, and help to improve the user experience.

[0248] Some VR, AR, MR and other services can support connections between electronic devices, for example, electronic devices are connected through a wireless local area network or through a server. When the first electronic device executes such a service, it can receive direct sounds from other users sent by the electronic device connected to it, and then play the direct sounds. Fig.17 Make an introduction.

[0249] Optionally, in some embodiments, Fig.17 As shown, the method 600 further includes:

[0250] S616: Receive a third rendered sound sent by the second electronic device.

[0251] S617, play the third rendering sound.

[0252] The first electronic device can receive and play a third rendered sound sent by a second electronic device, where the second electronic device is connected to the first electronic device and the second electronic device also displays the first virtual scene. The third rendered sound includes a virtual direct sound and a virtual reflected sound of a user corresponding to the second electronic device.

[0253] It should be understood that the method for determining the third rendering sound for the second electronic device can be found above, and for the sake of brevity, it will not be repeated here.

[0254] It is understandable that the first electronic device can use technology such as spatial audio to play the third rendered sound, so that the user can perceive the direction of the third rendered sound.

[0255] For example, a first electronic device is connected to a second electronic device, and both the first electronic device and the second electronic device display a first virtual scene, user #1 of the first electronic device corresponds to virtual character #1, user #2 of the second electronic device corresponds to virtual character #2, and virtual character #2 is in front of the left of virtual character #1. After the first electronic device receives a third rendered sound associated with user #2 sent by the second electronic device, the third rendered sound can be played through spatial audio technology, so that user #1 feels that user #2 is in front of his left.

[0256] In the embodiment of the present application, the electronic device can also receive and play third-party rendered sounds of other users sent by other electronic devices, so that users can experience multi-person interaction, which can bring users a better sense of immersion and help improve the user experience.

[0257] Optionally, in some embodiments, the first virtual scene includes virtual scene sound, and the first electronic device can also play the virtual scene sound.

[0258] For example, taking the virtual scene as a jungle scene, the jungle scene includes the sound of a lion and the sound of a tiger, and the first electronic device can also play the sound of the lion and the tiger.

[0259] In an embodiment of the present application, when the virtual scene includes virtual scene sounds, the electronic device can also play the sounds of the virtual scene, which can bring a better sense of immersion to the user and help enhance the user experience.

[0260] Optionally, in some embodiments, the first electronic device may also synchronously record and broadcast the first virtual scene and the first rendered sound. In these embodiments, the first rendered sound includes a first virtual direct sound and a first virtual reflected sound.

[0261] For example, taking the virtual scene as a live selling scene, the user is the selling anchor, the real scene of the selling anchor is a smaller live broadcast room, and the virtual live selling scene is a shopping mall. The first electronic device can broadcast the first virtual scene and the first rendered sound it displays, so that other users watching the live broadcast can feel that the selling anchor is in the shopping mall, and there can be a better live broadcast atmosphere.

[0262] The above mainly introduces a method for rendering sound provided by an embodiment of the present application from the perspective of an electronic device. It is understandable that, in order to realize the above functions, the electronic device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0263] The embodiment of the present application can divide the processor in the electronic device into functional modules (or units) according to the above method example. For example, each functional module (or unit) can be divided corresponding to each function, or two or more functions can be integrated into one processing module (or unit). The above-mentioned integrated module (or unit) can be implemented in the form of hardware or in the form of software functional modules (or units). It should be noted that the division of modules (or units) in the embodiment of the present application is schematic, which is only a logical function division, and there may be other division methods in actual implementation.

[0264] In the case where each functional module (or unit) is divided according to each function, Fig.18 A schematic diagram of the composition of a first electronic device 1800 provided in an embodiment of the present application is shown. As shown in FIG1800 , the first electronic device 1800 includes: a display module 1810 , an acquisition module 1820 , a processing module 1830 and a playback module 1840 .

[0265] The display module 1810 is used to display the first virtual scene.

[0266] The acquisition module 1820 is used to detect a first voice input of a user and acquire a first sound, where the first sound includes a first direct sound of the user.

[0267] The processing module 1830 is used to generate a first rendered sound according to the user's first direct sound and the first virtual scene, wherein the first rendered sound includes a first virtual reflected sound, which is a sound formed by the first direct sound reflecting in the first virtual scene.

[0268] The playing module 1840 is configured to play the first rendered sound.

[0269] Optionally, in some embodiments, the processing module 1830 is specifically configured to:

[0270] determining a first acoustic parameter according to the first virtual scene;

[0271] The first direct sound is rendered using the first acoustic parameter to generate a first rendered sound.

[0272] Optionally, in some embodiments, the processing module 1830 is further configured to determine whether a preset condition is satisfied and switch the first acoustic parameter to a second acoustic parameter.

[0273] The acquisition module 1820 is further configured to detect a second voice input of the user and acquire a second sound, wherein the second sound includes a second direct sound.

[0274] The processing module 1830 is further configured to render the second direct sound using the second acoustic parameter to generate a second rendered sound.

[0275] The playing module 1840 is further configured to play the second rendered sound.

[0276] Optionally, in some embodiments, the preset condition is: switching from the first virtual scene to the second virtual scene, or the position and / or radiation direction of the sound source in the first virtual scene changes, wherein the sound source is the direct sound of the user.

[0277] Optionally, in some embodiments, before the processing module 1830 generates the first rendering sound according to the first direct sound and the first virtual scene, the acquiring module 1820 is further configured to acquire the sound intensity level of the first direct sound.

[0278] The processing module 1830 is specifically configured to generate a first rendering sound according to the first direct sound, the sound intensity level of the first direct sound, and the first virtual scene.

[0279] Optionally, in some embodiments, the first virtual scene includes a first virtual object, the first virtual object is associated with the user, and the acquisition module 1820 is further used to acquire the timbre of the first virtual object.

[0280] Processing module 1830 is specifically used to generate a first rendered sound based on the user's first direct sound, the timbre of the first virtual object and the first virtual scene. The first rendered sound also includes a first virtual direct sound, which is generated based on the first direct sound and the timbre of the first virtual object.

[0281] Optionally, in some embodiments, the processing module 1830 is further used to generate a first noise reduction, which is used to offset the first reflected sound and / or the noise of the real scene in which the user is located. The first reflected sound is formed by the first direct sound being reflected by the real scene.

[0282] The playing module 1840 is also used to play the first noise reduction.

[0283] Optionally, in some embodiments, the acquisition module 1820 is further used to acquire a third acoustic parameter, where the third acoustic parameter includes an acoustic parameter of a real scene.

[0284] The processing module 1830 is specifically configured to generate a first noise reduction according to a third acoustic parameter.

[0285] Optionally, in some embodiments, the acquisition module 1820 is specifically used to acquire a third acoustic parameter using one or more sensors, and the one or more sensors include one or more of the following: an inertial measurement unit IMU, a speaker, a microphone, an ultrasonic radar, an infrared radar, a millimeter wave radar, and a camera.

[0286] Optionally, in some embodiments, the first acoustic parameter and the third acoustic parameter further include head-related transfer function HRTF parameters of the user.

[0287] Optionally, in some embodiments, the first electronic device is connected to an earphone, and the first electronic device plays the first rendered sound through the earphone, and the processing module 1830 is further configured to:

[0288] Detect the type of headphones;

[0289] When it is determined that the earphone is an in-ear earphone or a semi-in-ear earphone, a first virtual direct sound is added to the first rendered sound, where the first virtual direct sound is generated based on the first direct sound.

[0290] Optionally, in some embodiments, when the earphone is an in-ear earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a first ratio, and when the earphone is a semi-in-ear earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a second ratio, and the first ratio is greater than the second ratio.

[0291] Optionally, in some embodiments, the first electronic device is connected to a headset, and the first electronic device plays the first rendered sound through the headset, the first rendered sound also includes a first virtual direct sound, and the first virtual direct sound is generated according to the first direct sound, and the processing module 1830 is further used to:

[0292] Detect the type of headphones;

[0293] The ratio of the first virtual direct sound to the first virtual reflected sound is determined according to the type of headphones.

[0294] Optionally, in some embodiments, when the earphone is an in-ear earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a first ratio, when the earphone is a semi-in-ear earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a second ratio, and when the earphone is an open earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a third ratio, and the first ratio is greater than the second ratio, and the second ratio is greater than the third ratio.

[0295] Optionally, in some embodiments, the first electronic device is connected to the second electronic device, the second electronic device displays the first virtual scene, and the acquisition module 1810 is further used to receive a third rendered sound sent by the second electronic device.

[0296] The playing module 1840 is further configured to play the third rendering sound.

[0297] Optionally, in some embodiments, the first virtual scene includes virtual scene sound, and the method further includes:

[0298] The playing module 1840 is also used to play the virtual scene sound.

[0299] Optionally, in some embodiments, the playback module 1840 is further configured to synchronously record or broadcast the first virtual scene and the first rendering sound.

[0300] The embodiment of the present application provides a computer program product, when the computer program product is run on an electronic device, the electronic device executes the technical solution in the above embodiment. Its implementation principle and technical effect are similar to those of the above method-related embodiments, and will not be repeated here.

[0301] The embodiment of the present application provides a readable storage medium, the readable storage medium contains instructions, when the instructions are executed in an electronic device, the electronic device executes the technical solution of the above embodiment. The implementation principle and technical effect are similar, and will not be repeated here.

[0302] The embodiment of the present application provides a chip, the chip is used to execute instructions, when the chip is running, the technical solution in the above embodiment is executed. The implementation principle and technical effect are similar, and will not be repeated here.

[0303] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0304] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0305] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0306] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0307] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0308] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the part of the technical solution of the embodiment of the present application that essentially contributes or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0309] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiment of the present application, which should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A method for rendering sound, characterized in that: The method is applied to a first electronic device, and the method includes: displaying a first virtual scene; Detecting a first voice input of a user and acquiring a first sound, wherein the first sound includes a first direct sound of the user; generating a first rendered sound according to the first direct sound and the first virtual scene, wherein the first rendered sound includes a first virtual reflected sound, and the first virtual reflected sound is a sound formed by the first direct sound being reflected in the first virtual scene; The first rendered sound is played.

2. The method according to claim 1, characterized in that The generating a first rendered sound according to the first direct sound and the first virtual scene includes: determining a first acoustic parameter according to the first virtual scene; The first direct sound is rendered using the first acoustic parameter to generate the first rendered sound.

3. The method according to claim 2, characterized in that The method further comprises: Determining that a preset condition is met, switching the first acoustic parameter to a second acoustic parameter; Detecting a second voice input of a user and acquiring a second sound, wherein the second sound includes a second direct sound; Rendering the second direct sound using the second acoustic parameter to generate a second rendered sound; Play the second rendered sound.

4. The method according to claim 3, characterized in that The preset conditions are: Switching from the first virtual scene to the second virtual scene, or The position and / or radiation direction of the sound source in the first virtual scene changes, wherein the sound source is the direct sound of the user.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Get the sound intensity level of the first direct sound; The generating a first rendered sound according to the first direct sound and the first virtual scene includes: A first rendered sound is generated according to the first direct sound, the sound intensity level of the first direct sound, and the first virtual scene.

6. The method according to any one of claims 1 to 5, characterized in that The first virtual scene includes a first virtual object, and the first virtual object is associated with the user. The method further includes: Acquire the timbre of the first virtual object; The generating a first rendered sound according to the first direct sound and the first virtual scene includes: A first rendered sound is generated according to the first direct sound, the timbre of the first virtual object and the first virtual scene. The first rendered sound also includes a first virtual direct sound. The first virtual direct sound is generated according to the first direct sound and the timbre of the first virtual object.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: generating a first noise reduction, where the first noise reduction is used to cancel a first reflected sound and / or noise of a real scene where the user is located, where the first reflected sound is formed by the first direct sound being reflected by the real scene; The first noise reduction is played.

8. The method according to claim 7, characterized in that The generating of the first noise reduction comprises: Acquiring a third acoustic parameter, wherein the third acoustic parameter includes an acoustic parameter of the real scene; The first direct sound is rendered according to the third acoustic parameter to generate a first noise reduction.

9. The method according to claim 8, characterized in that The obtaining of the third acoustic parameter comprises: The third acoustic parameter is acquired using one or more sensors, wherein the one or more sensors include one or more of the following: Inertial measurement unit IMU, speakers, microphones, ultrasonic radar, infrared radar, millimeter wave radar and cameras.

10. The method according to claim 8 or 9, characterized in that: The first acoustic parameter and the third acoustic parameter further include a head-related transfer function HRTF parameter of the user.

11. The method according to any one of claims 1 to 10, characterized in that The first electronic device is connected to a headset, and the first electronic device plays the first rendered sound through the headset. The method further includes: Detecting the type of the headset; When it is determined that the earphone is an in-ear earphone or a semi-in-ear earphone, a first virtual direct sound is added to the first rendered sound, where the first virtual direct sound is generated based on the first direct sound.

12. The method according to claim 11, characterized in that When the earphone is an in-ear earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a first ratio, and when the earphone is a semi-in-ear earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a second ratio, and the first ratio is greater than the second ratio.

13. The method according to any one of claims 1 to 10, characterized in that The first electronic device is connected to a headset, and the first electronic device plays the first rendered sound through the headset, the first rendered sound also includes a first virtual direct sound, and the first virtual direct sound is generated according to the first direct sound, and the method further includes: Detecting the type of the headset; The ratio of the first virtual direct sound to the first virtual reflected sound is determined according to the type of the earphone.

14. The method according to claim 13, characterized in that When the earphone is an in-ear earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a first ratio, when the earphone is a semi-in-ear earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a second ratio, and when the earphone is an open earphone, the ratio of the first virtual direct sound to the first virtual reflected sound is a third ratio, the first ratio is greater than the second ratio, and the second ratio is greater than the third ratio.

15. The method according to any one of claims 1 to 13, characterized in that The first electronic device is connected to the second electronic device, and the second electronic device displays the first virtual scene. The method further includes: receiving a third rendered sound sent by a second electronic device; The third rendered sound is played.

16. The method according to any one of claims 1 to 15, characterized in that The first virtual scene includes virtual scene sound, and the method further includes: Play the virtual scene sound.

17. The method according to any one of claims 1 to 16, characterized in that The method further comprises: The first virtual scene and the first rendered sound are synchronously recorded or live broadcasted.

18. An electronic device, characterized in that: The method comprises one or more processors; one or more memories; the one or more memories store one or more computer programs, the one or more computer programs include instructions, and when the instructions are executed by the one or more processors, the method according to any one of claims 1 to 17 is executed.

19. A chip, characterized in that: The chip includes a processor and a communication interface, wherein the communication interface is used to receive a signal and transmit the signal to the processor, and the processor processes the signal so that the method according to any one of claims 1 to 17 is executed.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the method according to any one of claims 1 to 17 is executed.

Images