Sound effect processing method and device

By obtaining the relative orientation information of the user relative to the sound source object in the virtual environment and performing sound effects processing on the sound source signal, the problem of insufficient sound fusion in virtual reality technology is solved, and the user's immersion experience and sound authenticity are improved.

CN112071326BActive Publication Date: 2024-11-22SAMSUNG ELECTRONICS CHINA R&D CENT +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010926470.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-09-07
Publication Date
2024-11-22
Estimated Expiration
2040-09-07

AI Technical Summary

Technical Problem

In the existing virtual reality technology, the external sound playback form is difficult to achieve the user's immersive experience, and the integration of sound and the environment is insufficient.

Method used

By obtaining the relative orientation information of the user relative to the sound source object in the virtual environment, the sound source signal is processed based on this information, including calculating the head-related unit impact response and convolution of the sound source signal, generating a real binaural signal, and Feedback to the user the processed signal.

Benefits of technology

It improves users' immersive experience in the virtual environment, makes the sound of sound source objects in the virtual environment more realistic, and enhances the spatial sense and reliability of the sound.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112071326B_ABST
    Figure CN112071326B_ABST
Patent Text Reader

Abstract

The present application discloses a sound effect processing method and device: obtaining the relative position information of a user relative to at least one other sound source object in a preset virtual environment; performing sound effect processing on the sound source signal of each of the at least one other sound source object based on the relative position information; and feeding back the sound source signal after the sound effect processing to the user. This implementation makes the sound of the sound source object in the virtual environment obtained by the user more realistic, which helps to enhance the user's immersive experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of virtual reality technology, specifically to the field of audio processing, and in particular to a method and device for sound effect processing. Background Art

[0002] VR (Virtual Reality) refers to the technology that uses computer graphics systems and various reality and control interface devices to provide an immersive feeling in a three-dimensional interactive environment generated by a computer. Currently, virtual reality technology has been widely promoted and applied in games, including shooting, puzzle, role-playing and other games.

[0003] In existing VR scenes, perfect visual simulation can be achieved through augmented reality, while the sound in virtual reality scenes is often achieved through external speakers. Although the fusion of environment and sound can be achieved, the user's immersive experience is poor. Summary of the invention

[0004] Embodiments of the present application provide a sound effect processing method, apparatus, device, and storage medium.

[0005] In a first aspect, an embodiment of the present application provides a sound effect processing method, the method comprising: obtaining the relative position information of a user relative to at least one other sound source object in a preset virtual environment; based on the relative position information, performing sound effect processing on the sound source signal of each of the at least one other sound source object to obtain the sound source signal after the sound effect processing, the sound source signal after the sound effect processing is used to characterize the sound signal of the sound source signal reaching the user's ear; and feeding back the sound source signal after the sound effect processing to the user.

[0006] In some embodiments, the relative orientation information includes: relative position and facial orientation, and based on the relative orientation information, performing sound effect processing on the sound source signal of each of the at least one other sound source object, including: updating the relative orientation information in response to changes in the user's relative position and / or facial orientation relative to the at least one other sound source object; and performing sound effect processing on the sound source signal of each of the at least one other sound source object based on the updated relative orientation information.

[0007] In some embodiments, based on the relative position information, sound effect processing is performed on the sound source signal of each sound source object among the at least one other sound source object, including: respectively calculating the convolution of the head-related unit impulse response HRIR corresponding to each relative position information and the sound source signal of the sound source object corresponding to the relative position information.

[0008] In some embodiments, there are multiple sound source objects, and based on the relative position information, sound effect processing is performed on the sound source signal of each sound source object in at least one other sound source object, including: based on the relative position information, sound effect processing is performed on the sound source signal of each sound source object, and the sound source signals after the sound effect processing are superimposed.

[0009] In some embodiments, the sound source objects include: other users, and obtaining the relative position information of the user relative to at least one other sound source object in a preset virtual environment includes: obtaining the relative position information of the user relative to at least one other user who meets preset conditions in the preset virtual environment, and the preset conditions include: the face of the other user is facing the user.

[0010] In a second aspect, an embodiment of the present application provides a sound effect processing device, which includes: an acquisition module, configured to obtain the relative position information of a user relative to at least one other sound source object in a preset virtual environment; a processing module, configured to perform sound effect processing on the sound source signal of each of the other at least one sound source object based on the relative position information, to obtain the sound source signal after the sound effect processing, and the sound source signal after the sound effect processing is used to characterize the sound signal of the sound source signal reaching the user's ear; a feedback module, configured to feed back the sound source signal after the sound effect processing to the user.

[0011] In some embodiments, the relative orientation information includes: relative position and face orientation, and the processing module is further configured to: update the relative orientation information in response to changes in the user's relative position and / or face orientation relative to at least one other sound source object; and perform sound effect processing on the sound source signal of each of the at least one other sound source object based on the updated relative orientation information.

[0012] In some embodiments, the processing module is further configured to respectively calculate the convolution of the head-related unit impulse response HRIR corresponding to each relative position information and the sound source signal of the sound source object corresponding to the relative position information.

[0013] In some embodiments, there are multiple sound source objects, and the processing module is further configured to: perform sound effect processing on the sound source signal of each sound source object based on the relative orientation information; and superimpose the sound source signals after the sound effect processing on the sound source signal of each sound source object.

[0014] In some embodiments, the sound source object includes: other users, and the acquisition module is further configured to: obtain the relative position information of the user relative to at least one other user that meets preset conditions in a preset virtual environment, and the preset conditions include: the face of the other user is facing the user.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, which includes one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement a sound processing method as in any embodiment of the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements a sound effect processing method as in any embodiment of the first aspect.

[0017] The present application obtains the relative position information of the user with respect to at least one other sound source object in a preset virtual environment; based on the relative position information, performs sound effect processing on the sound source signal of each of the at least one other sound source object; and feeds back the sound source signal after the sound effect processing to the user, so that the sound of the sound source object in the virtual environment obtained by the user can be different according to the relative position information, thereby making the obtained sound more realistic, which helps to enhance the user's immersive experience.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0020] Figure 2 is a flow chart of an embodiment of a sound effect processing method according to the present application;

[0021] Figure 3 A schematic diagram of an embodiment of a sound effect processing method according to the present application;

[0022] Figure 4 is a schematic diagram of an application scenario of the sound effect processing method according to the present application;

[0023] Figure 5 is a flow chart of another embodiment of the sound effect processing method according to the present application;

[0024] Figure 6 is a schematic diagram of an embodiment of a sound effect processing device according to the present application;

[0025] Figure 7 It is a structural diagram of a computer system suitable for implementing a server of an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0027] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0028] Figure 1 The system architecture 100 of a sound effect processing platform to which the sound effect processing method of the present application may be applied is shown.

[0029] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0030] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be installed with various applications with positioning functions, such as GPS positioning, Bluetooth positioning applications, etc.

[0031] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with positioning functions, including but not limited to Bluetooth headsets, mobile phones and glasses with positioning functions. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide sound processing services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0032] The server 105 can be a server that provides various services, for example, obtaining the relative position information of the terminal devices 101, 102, and 103 relative to at least one other sound source object, performing sound effect processing on the sound source signal of each of the at least one other sound source object based on the relative position information, and feeding back the sound source signal after the sound effect processing to the user.

[0033] It should be noted that the sound effect processing method provided in the embodiment of the present application is generally executed by the server 105 , and accordingly, the sound effect processing device is generally disposed in the server 105 .

[0034] It should be noted that the terminal devices 101, 102, 103 may also be installed with sound effect processing applications, and the terminal devices 101, 102, 103 may also perform sound effect processing based on relative position information. In this case, the sound effect processing method may also be executed by the terminal devices 101, 102, 103, and accordingly, the sound effect processing device may also be set in the terminal devices 101, 102, 103. In this case, the exemplary system architecture 100 may also not include the server 105 and the network 104.

[0035] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, for providing sound effect processing services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0036] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0037] Figure 2 A flowchart 200 of an embodiment of a sound effect processing method that can be applied to the present application is shown. The sound effect processing method comprises the following steps:

[0038] Step 201: Obtain relative position information of a user relative to at least one other sound source object in a preset virtual environment.

[0039] In this embodiment, the execution subject can obtain the relative position information of the user relative to one or more other sound source objects in the preset virtual environment. The relative position information can be the relative position information of the user relative to other sound source objects, the face orientation information of the user relative to other sound source objects, or the position information and face orientation information of the user relative to other sound source objects, which is not limited in this application.

[0040] Here, the sound source object may be a virtual sound source object in a preset virtual environment, or may be other users in the virtual environment, and this application does not limit this.

[0041] In addition, the execution subject may obtain relative position information in a variety of ways. For example, the execution subject may obtain the relative position information of the user relative to all sound source objects in the virtual environment, or may directly obtain the relative position information of the user relative to the sound source objects that meet the preset conditions in the virtual environment. The preset conditions may be determined based on experience, actual needs, and specific application scenarios, and this application does not limit this.

[0042] Specifically, the sound source object includes other users, and the execution subject can also determine whether to only obtain the relative position information of the user relative to other users who meet the preset conditions by setting the sound source signal acquisition mode. Among them, the sound source signal acquisition mode can include an open mode and a private mode, etc. If the sound source signal acquisition mode is an open mode, the execution subject can obtain the relative position information of the user relative to all other users, and then process the sound source signals of other users and feed them back to the user; if the acquisition mode is a private mode, the execution subject only obtains the relative position information of the user relative to other users who meet the preset conditions.

[0043] It should be pointed out that the execution entity can directly obtain the relative position information of the user relative to other sound source objects, or can separately obtain the position information of the user and other sound source objects in a preset coordinate system, and determine the relative position information based on the position information of the user and other sound source objects. This application does not limit this.

[0044] Specifically, since the virtual reality environment is usually a fixed space, the execution subject can directly create a rectangular coordinate system based on the fixed space. The origin of the rectangular coordinate system can be set according to experience, specific application scenarios and actual needs, and this application does not limit this. Specifically, if the virtual reality environment is a room, the corner point of the room can be used as the origin.

[0045] Furthermore, the execution entity can obtain the user's orientation information in a preset rectangular coordinate system and the orientation information of the sound source object, wherein the orientation information may include the position information and face orientation information of the sound source object in the preset rectangular coordinate system, and determine the relative orientation information of the user relative to at least one other sound source object in the preset rectangular coordinate system based on the above-mentioned user's orientation information and the orientation information of other sound source objects.

[0046] The position information of the user in the preset rectangular coordinate system can be obtained by using a positioning device in the existing technology or future development technology, such as a Bluetooth headset, glasses with a positioning function, and the like.

[0047] Specifically, the execution entity may determine the orientation information of the user in a preset rectangular coordinate system by sending a direction finding signal and a positioning signal to the Bluetooth headset worn by the user.

[0048] In addition, the position information of the virtual sound source object in the virtual environment may be manually preset position information.

[0049] In some optional methods, obtaining the relative position information of the user relative to at least one other sound source object in a preset virtual environment includes: obtaining the relative position information of the user relative to at least one other user who meets preset conditions in the preset virtual environment.

[0050] In this implementation, in the virtual environment, in addition to the current user, there are other users, and the execution subject can obtain the relative position information of the current user with respect to at least one other user.

[0051] Here, the execution subject may only obtain the relative position information of the current user relative to at least one other user that meets a preset condition. The preset condition may be that the face of the other user is facing the current user, that is, the execution subject only obtains the relative position information of the current user relative to the other user whose face is facing the current user.

[0052] It should be pointed out that the face of the current user can be facing any direction.

[0053] There are multiple ways for the execution subject to determine whether the faces of other users are facing the current user.

[0054] Specifically, Figure 3 As shown, the virtual environment includes user A301, other user B302 and other user C303, and user A301, other user B302 and other user C303 all wear Bluetooth headsets for positioning. The execution subject can first determine the coordinates of the midpoint position M of the line connecting the left-ear Bluetooth headset 304 and the right-ear Bluetooth headset 305 according to the coordinates of the left-ear Bluetooth headset 304 and the right-ear Bluetooth headset 305 of user A301, determine the coordinates of the midpoint position N of the line connecting the left-ear Bluetooth headset 306 and the right-ear Bluetooth headset 307 according to the coordinates of the left-ear Bluetooth headset 306 and the right-ear Bluetooth headset 307 of other user B302, and determine the coordinates of the midpoint position P of the line connecting the left-ear Bluetooth headset 308 and the right-ear Bluetooth headset 309 according to the coordinates of the left-ear Bluetooth headset 308 and the right-ear Bluetooth headset 309 of other user C303. Then, the execution subject further determines the face orientation of other users B302 and other users C303 relative to user A301.

[0055] The execution entity can calculate the difference between the angle between the line MN and the coordinates of the left ear Bluetooth headset 304 and the right ear Bluetooth headset 305 of user B302 and 90°, and determine whether the above difference is less than or equal to a preset angle value, for example, 10°. If it is less than or equal to 10°, it is determined that the face of other user B is facing user A.

[0056] Furthermore, the execution entity can use the above method to determine whether the face of other user C303 is facing user A301. If the difference between the angle between the connection line MP and the line connecting the left ear Bluetooth headset 304 and the right ear Bluetooth headset of user C303 and 90° is greater than 10°, it is determined that the face of other user C is not facing user A.

[0057] It can also be seen from the above that although other user C is closer to user A, the face of other user C is not facing user A, that is, other user C does not meet the preset conditions, so user A cannot receive the sound source signal of user C after sound effect processing.

[0058] This implementation method enables the user to obtain the sound source signal in a targeted manner by obtaining the relative position information of the user relative to at least one other user who meets the preset conditions in the preset virtual environment, while improving the reliability and effectiveness of the sound source signal acquisition.

[0059] Step 202: Based on the relative position information, perform sound effect processing on the sound source signal of each sound source object in the at least one other sound source object to obtain a sound source signal after the sound effect processing.

[0060] In this embodiment, after obtaining the relative position information of the user relative to at least one other sound source object, the execution subject can use a preset sound effect processing algorithm to perform sound effect processing on the sound source signal in each other sound source object according to the relative position information of the user relative to each other sound source object to obtain a sound source signal after sound effect processing. Here, the sound source signal after sound effect processing is mainly used to represent the sound signal of the sound source signal reaching the user's ears, that is, the binaural signal.

[0061] Here, the preset sound effect processing algorithm can adopt the sound effect processing algorithm in the existing technology or future development technology, for example, HRTF (Head Related Transfer Function) algorithm, HRIR-based synthesis algorithm, etc., and this application does not limit this.

[0062] Among them, HRTF is a sound localization algorithm, corresponding to HRIR (Head Related Impulse Responses) in the time domain. In the spatial sound image presentation based on binaural hearing, HRIR is the basis. It assumes that the sound transmission path from the sound source to the human ear is a linear time-invariant system, so it can be described by a unit impulse response.

[0063] In some optional embodiments, based on the relative orientation information, sound effect processing is performed on the sound source signal of each sound source object among at least one other sound source object, including: respectively calculating the convolution of the head-related unit impulse response HRIR corresponding to each relative orientation information and the sound source signal of the sound source object corresponding to the relative orientation information.

[0064] In this implementation, the execution subject can directly calculate the convolution of the head-related unit impulse response HRIR corresponding to each relative orientation information and the sound source signal of the sound source object corresponding to the relative orientation information to obtain binaural signals to present the sense of spatial orientation.

[0065] Among them, since HRIR is mainly obtained through acoustic measurement, and the measurement orientation always corresponds to a spatial orientation point and a time sampling with a certain accuracy, in order to ensure the accuracy of the obtained binaural signal, it is necessary to measure sufficiently dense spatial orientation points and the sampling rate must be high enough.

[0066] This implementation method obtains the sound source signal after sound effect processing by directly calculating the convolution of the head-related unit impulse response HRIR corresponding to each relative orientation information and the sound source signal of the sound source object corresponding to the relative orientation information, thereby improving the efficiency of obtaining the sound effect processing signal.

[0067] In some optional embodiments, there are multiple sound source objects, and based on the relative position information, sound effect processing is performed on the sound source signal of each sound source object among at least one other sound source object, including: based on the relative position information, sound effect processing is performed on the sound source signal of each sound source object; and the sound source signals after the sound effect processing are superimposed.

[0068] In this implementation, if there are multiple sound source objects, the sound source signal after the sound effect processing may be the sound source signal after the sound effect processing is performed on the sound source signals of the multiple sound source objects respectively. Further, the execution subject feeds back the superimposed sound source signal to the user, that is, feeds back the sound source signal after the sound effect processing is performed on the sound source signal of each sound source object to the user at the same time.

[0069] This implementation method obtains a superimposed sound source signal by superimposing the sound source signals of each sound source object after sound effect processing, and feeds the superimposed sound source signal back to the user, thereby improving the authenticity of the sound fed back to the user and helping to further enhance the user's immersive experience.

[0070] Step 203: Feedback the sound source signal after the sound effect processing to the user.

[0071] In this embodiment, the executing entity may feed back the sound source signals after the above sound effect processing to the user through the audio playback device respectively, or may superimpose the sound source signals after the above sound effect processing and feed back to the user through the audio playback device, and this application does not limit this.

[0072] Among them, the audio playback device can be a device used to convert sound source signals into sound in existing technology or future development technology, such as headphones, mobile phones, speakers, etc., and this application does not limit this.

[0073] Continue to see Figure 4 , Figure 4 It is a schematic diagram of an application scenario of the sound effect processing method according to this embodiment.

[0074] exist Figure 4 In an application scenario, in a specific example, in a preset virtual environment, a rectangular coordinate system 401 is set according to the fixed space corresponding to the virtual environment, and the execution subject 402 can set the origin of the coordinate system. At this time, there are a user 403 and sound source objects 404 and 405 in the virtual environment. First, the execution subject 402 can determine the user's orientation information in the rectangular coordinate system through the positioning device worn by the user 403, such as the left Bluetooth headset 406 and the right Bluetooth headset 407, and obtain the orientation information of the sound source objects 404 and 405 at the same time, and then determine the orientation information of the user 403 relative to the sound source object 404 and the orientation information of the user 403 relative to the sound source object 405. Furthermore, the execution subject 402 performs sound effect processing on the sound source signal of the sound source object 404 and the sound source signal of the sound source object 405 according to the above-mentioned relative position information and a preset sound effect positioning algorithm, respectively, to obtain the sound source signal after sound effect processing, and finally, feeds back the sound source signal after sound effect processing to the user through an audio playback device, such as a Bluetooth headset.

[0075] The sound effect processing method provided by the embodiments of the present disclosure obtains the relative position information of the user relative to at least one other sound source object in a preset virtual environment; based on the relative position information, performs sound effect processing on the sound source signal of each of the at least one other sound source object; and feeds back the sound source signal after the sound effect processing to the user, so that the sound of the sound source object in the virtual environment obtained by the user is more realistic, which helps to enhance the user's immersive experience.

[0076] Further references Figure 5 , which shows a process 500 of another embodiment of a sound effect processing method. In this embodiment, the relative position information includes: relative position and face orientation. The process 500 of the sound effect processing method of this embodiment may include the following steps:

[0077] Step 501: Obtain relative position information of a user relative to at least one other sound source object in a preset virtual environment.

[0078] In this embodiment, the implementation details and technical effects of step 501 can refer to the description of step 201 and will not be repeated here.

[0079] Step 502: In response to a change in the relative position and / or face orientation of the user relative to at least one other sound source object, the relative orientation information is updated.

[0080] In this embodiment, if the user's position and / or face orientation relative to the sound source object changes, the execution entity may use the changed user's position and / or face orientation information relative to the sound source object as updated relative orientation information.

[0081] Here, the change in the user's position and / or facial orientation relative to the sound source object may be caused by a change in the user's position and / or facial orientation, or may be caused by a change in the position and / or facial orientation of the sound source object, and this application does not limit this.

[0082] For example, a user is playing a shooting game in a virtual environment, and there is a sound source object A in the virtual environment. The executing entity first determines the relative orientation information based on the user's orientation information and the orientation information of the sound source object A obtained by the positioning device worn by the user. If at this time, the user changes the direction of his face and / or leaves the current position, and the orientation information of the sound source object A remains unchanged, then the relative orientation information between the user and the sound source object A changes, and then the updated relative orientation information is determined based on the orientation information of the sound source object A and the changed orientation information of the user.

[0083] For another example, a user is watching a performance in a virtual environment where there is a sound source object B. During the performance, the user's position and facial orientation remain unchanged, but the position and / or facial orientation of sound source object B changes. Then, the relative orientation information between the user and sound source object B changes. Then, based on the user's orientation information and the changed orientation information of sound source object B, the updated relative orientation information is determined.

[0084] Step 503: Based on the updated relative position information, perform sound effect processing on the sound source signal of each sound source object in the at least one other sound source object.

[0085] In this embodiment, after acquiring the updated relative position information, the execution subject may use a preset sound effect processing algorithm to perform sound effect processing on the sound source signal of each of the other at least one sound source object according to the updated relative position information.

[0086] Specifically, a user is playing a shooting game in a virtual environment, where there are sound source objects C and D. The user's initial position is located in the middle of the line connecting the sound source objects C and D, and the user's face is facing the sound source object D. If the user leaves the current position and moves along the line connecting the sound source objects C and D to a new position closer to the sound source object D, and the face direction remains unchanged, the execution subject updates the relative orientation information based on the new position of the user to obtain the updated relative orientation information, and further determines the sound source signal of the sound source object C and the sound source signal of the sound source object D after the sound effect processing according to the relative orientation information. At this time, since the user moves away from the sound source object C and approaches the sound source object D, the sound source signal of the sound source object C after the sound effect processing becomes smaller than the sound source signal heard by the user at the initial position, and the sound source signal of the virtual sound source object D after the sound effect processing becomes larger than the sound source signal heard by the user at the initial position.

[0087] Step 504: Feedback the sound source signal after the sound effect processing to the user.

[0088] In this embodiment, the implementation details and technical effects of step 504 can refer to the description of step 203 and will not be repeated here.

[0089] The above-mentioned embodiments of the present application update the relative position information in response to changes in the relative position and / or facial orientation of the user relative to at least one other sound source object; based on the updated relative position information, perform sound effect processing on the sound source signal of each of the at least one other sound source object, and feed back the sound source signal after sound effect processing to the user, so that when the user moves the position or changes the facial orientation, the sound experienced by the user will also change with the change of the position or facial orientation, just like the sound emitted by objects in the virtual space, further enhancing the user's immersive experience.

[0090] Further references Figure 6 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a sound effect processing device, and the device embodiment is Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0091] like Figure 6 As shown, the sound effect processing device 600 of this embodiment includes: an acquisition module 601 , a processing module 602 and a feedback module 603 .

[0092] The acquisition module 601 may be configured to acquire the relative position information of the user relative to at least one other sound source object in a preset virtual environment.

[0093] The processing module 602 can be configured to perform sound effect processing on the sound source signal of each sound source object in at least one other sound source object based on the relative orientation information to obtain the sound source signal after the sound effect processing. The sound source signal after the sound effect processing is used to represent the sound signal when the sound source signal reaches the user's ear.

[0094] The feedback module 603 may be configured to feed back the sound source signal after the sound effect processing to the user.

[0095] In some optional embodiments of this embodiment, the relative orientation information includes: relative position and face orientation, and the processing module is further configured to: update the relative orientation information in response to changes in the user's relative position and / or face orientation relative to at least one other sound source object; and perform sound effect processing on the sound source signal of each of the at least one other sound source object based on the updated relative orientation information.

[0096] In some optional aspects of this embodiment, the processing module is further configured to respectively calculate the convolution of the head-related unit impulse response HRIR corresponding to each relative position information and the sound source signal of the sound source object corresponding to the relative position information.

[0097] In some optional embodiments of this embodiment, there are multiple sound source objects, and the processing module is further configured to: perform sound effect processing on the sound source signal of each sound source object based on relative orientation information; and superimpose the sound source signals of each sound source object after the sound effect processing.

[0098] In some optional modes of this embodiment, the sound source object includes: other users, and the acquisition module is further configured to: obtain the relative position information of the user relative to at least one other user who meets preset conditions in a preset virtual environment, and the preset conditions include: the face of the other user is facing the user.

[0099] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0100] like Figure 7 , is a block diagram of an electronic device according to the sound effect processing method of an embodiment of the present application.

[0101] 700 is a block diagram of an electronic device according to the sound effect processing method of an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0102] like Figure 7 As shown, the electronic device includes: one or more processors 701, a memory 702, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 701 is taken as an example.

[0103] The memory 702 is a non-transitory computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the sound effect processing method provided in the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the sound effect processing method provided in the present application.

[0104] The memory 702 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the sound effect processing method in the embodiment of the present application (for example, the attached Figure 6 The processor 701 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 702, that is, the sound effect processing method in the above method embodiment is implemented.

[0105] The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created by the use of the electronic device for sound effect processing, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 702 may optionally include a memory remotely arranged relative to the processor 701, and these remote memories may be connected to the electronic device for sound effect processing via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0106] The electronic device of the sound effect processing method may further include: an input device 703 and an output device 704. The processor 701, the memory 702, the input device 703 and the output device 704 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0107] The input device 703 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device for quality monitoring of the live video stream, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 704 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0108] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0109] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0110] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0111] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0112] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0113] According to the technical solution of the embodiment of the present application, the user's immersive experience is effectively improved.

[0114] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0115] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for processing sound effects, the method include: Obtaining relative position information of the user relative to at least one other sound source object in a preset virtual environment; Based on the relative position information, performing sound effect processing on the sound source signal of each sound source object of the other at least one sound source object to obtain a sound source signal after the sound effect processing, wherein the sound source signal after the sound effect processing is used to represent the sound signal of the sound source signal reaching the user's ear; Feeding back the sound source signal after the sound effect processing to the user, Wherein, the sound source objects include other users, and Wherein, obtaining the relative position information of the user relative to at least one other sound source object in the preset virtual environment includes: Obtaining relative position information of the user relative to at least one other user that meets a preset condition in a preset virtual environment, wherein the preset condition includes: the face of the other user is facing the user, The position information of the user and the other users is obtained through positioning devices worn by each of them, and the relative position information is obtained through the position information of the user and the other users.

2. The method according to claim 1, It is characterized in that The relative position information includes: relative position and face orientation, and the sound effect processing of the sound source signal of each of the at least one other sound source object based on the relative position information includes: In response to a change in a relative position of the user relative to at least one other sound source object and / or a change in a face orientation, updating the relative orientation information; Based on the updated relative position information, sound effect processing is performed on the sound source signal of each of the at least one other sound source object.

3. The method according to any one of claims 1 or 2, It is characterized in that The performing sound effect processing on the sound source signal of each of the at least one other sound source object based on the relative position information includes: The convolution of the head-related unit impulse response HRIR corresponding to each relative position information and the sound source signal of the sound source object corresponding to the relative position information is calculated respectively.

4. The method according to claim 1, It is characterized in that The number of the sound source objects is multiple, and the sound effect processing is performed on the sound source signal of each of the at least one other sound source object based on the relative position information, including: Based on the relative position information, performing sound effect processing on the sound source signal of each sound source object; The sound source signals of each sound source object are superimposed after being processed with sound effects.

5. A sound effect processing device, the device include: An acquisition module, configured to acquire relative position information of the user relative to at least one other sound source object in a preset virtual environment; a processing module configured to perform sound effect processing on the sound source signal of each of the at least one other sound source object based on the relative position information to obtain a sound source signal after the sound effect processing, wherein the sound source signal after the sound effect processing is used to represent a sound signal of the sound source signal reaching a user's ear; A feedback module is configured to feed back the sound source signal after the sound effect processing to the user, The sound source objects include other users, and Wherein, the acquisition module is further configured to: Obtaining relative position information of the user relative to at least one other user that meets a preset condition in a preset virtual environment, wherein the preset condition includes: the face of the other user is facing the user, The position information of the user and the other users is obtained through positioning devices worn by each of them, and the relative position information is obtained through the position information of the user and the other users.

6. The device according to claim 5, It is characterized in that The relative position information includes: relative position and face orientation, and the processing module is further configured to: In response to a change in a relative position of the user relative to at least one other sound source object and / or a change in a face orientation, updating the relative orientation information; Based on the updated relative position information, sound effect processing is performed on the sound source signal of each of the at least one other sound source object.

7. The device according to any one of claims 5 or 6, It is characterized in that The processing module is further configured to: The convolution of the head-related unit impulse response HRIR corresponding to each relative position information and the sound source signal of the sound source object corresponding to the relative position information is calculated respectively.

8. The device according to claim 5, It is characterized in that The number of the sound source objects is multiple, and the processing module is further configured to: Based on the relative position information, performing sound effect processing on the sound source signal of each sound source object; The sound source signals of each sound source object are superimposed after being processed with sound effects.

9. An electronic device, It is characterized in that include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium storing computer instructions, It is characterized in that The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 4.