Method, storage medium and electronic device for identifying sound source of audio data

Abstract

The present invention discloses a method, storage medium and electronic device for identifying the sound source of audio data. The method comprises: receiving audio data and first position data from multiple first virtual images in a preset virtual space, wherein the current identity of the multiple first virtual images is the speaker identity; determining the spatial position relationship using the first position data and the second position data, wherein the second position data is the position data of the second virtual image, the current identity of the second virtual image is the listener identity, and the spatial position relationship is used to determine the relative position relationship between the second virtual image and each of the multiple first virtual images; identifying the sound source of the audio data based on the spatial position relationship. The present invention solves the technical problems of mixed sound sources and poor user experience when multiple speakers speak at the same time in a virtual space.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method for identifying the sound source of audio data, a storage medium and an electronic device. Background Art

[0002] Remote conference is a common form of conference. In traditional remote conference, since the screen of the participants is displayed, the current speaker can be identified by: marking the current speaker by changing the screen layout, setting screen highlight, etc. in the remote conference product design; or quickly distinguishing the current speaker based on the screen of the participants through information such as movements and lip shape.

[0003] In recent years, more and more virtual conference rooms have appeared in remote conference scenarios. In virtual conference rooms, the screens of participants are replaced by their corresponding virtual images, and the scene layout of the virtual conference room is bound to the virtual images of the participants. At this time, it is difficult to mark the current speaker by changing the scene layout in product design, and it is also difficult for the audience in the virtual conference room to quickly distinguish the current speaker.

[0004] In addition, in a virtual conference room, multiple participants may speak at the same time. At this time, the conference voice is relatively mixed, and it is difficult for the audience to clearly distinguish the voice, resulting in a poor user experience.

[0005] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0006] The embodiments of the present invention provide a method, a storage medium and an electronic device for identifying the sound source of audio data, so as to at least solve the technical problems of mixed sound sources and poor user experience when multiple speakers speak simultaneously in a virtual space.

[0007] According to one aspect of an embodiment of the present invention, a method for identifying the sound source of audio data is provided, comprising: receiving audio data and first position data from multiple first virtual images in a preset virtual space, wherein the current identities of the multiple first virtual images are speaker identities; determining a spatial position relationship using the first position data and the second position data, wherein the second position data is the position data of a second virtual image, the current identity of the second virtual image is a listener identity, and the spatial position relationship is used to determine the relative position relationship between the second virtual image and each of the multiple first virtual images; and identifying the sound source of the audio data based on the spatial position relationship.

[0008] According to another aspect of an embodiment of the present invention, a method for identifying the sound source of audio data is also provided, including: receiving audio data and first position data from multiple first virtual images in a virtual conference room, wherein the current identities of the multiple first virtual images are conference speakers; determining a spatial position relationship using the first position data and the second position data, wherein the second position data is the position data of a second virtual image, the current identity of the second virtual image is a conference listener, and the spatial position relationship is used to determine the relative position relationship between the second virtual image and each of the multiple first virtual images; identifying the sound source of the audio data in the virtual conference room based on the spatial position relationship.

[0009] According to another aspect of an embodiment of the present invention, a method for identifying the sound source of audio data is also provided, including: receiving audio data and first position data from multiple first virtual images in a virtual classroom, wherein the current identities of the multiple first virtual images are classroom speakers; determining a spatial position relationship using the first position data and the second position data, wherein the second position data is the position data of a second virtual image, the current identity of the second virtual image is a classroom listener, and the spatial position relationship is used to determine the relative position relationship between the second virtual image and each of the multiple first virtual images; identifying the sound source of the audio data in the virtual classroom based on the spatial position relationship.

[0010] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned methods for identifying the sound source of audio data.

[0011] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising: a processor; and a memory, connected to the processor, for providing the processor with instructions for processing the following processing steps: receiving audio data and first position data from multiple first virtual images in a preset virtual space, wherein the current identities of the multiple first virtual images are speaker identities; determining a spatial position relationship using the first position data and the second position data, wherein the second position data is the position data of the second virtual image, the current identity of the second virtual image is the listener identity, and the spatial position relationship is used to determine the relative position relationship between the second virtual image and each of the multiple first virtual images; and identifying the sound source of the audio data based on the spatial position relationship.

[0012] In an embodiment of the present invention, audio data and first position data are received from a plurality of first virtual images in a preset virtual space, wherein the current identities of the plurality of first virtual images are speaker identities, and a spatial position relationship is determined by utilizing the first position data and the second position data, wherein the second position data is position data of a second virtual image, and the current identity of the second virtual image is a listener identity, and the spatial position relationship is used to determine a relative position relationship between the second virtual image and each of the plurality of first virtual images, and further identify a sound source of the audio data based on the spatial position relationship.

[0013] It is easy to notice that, through the embodiments of the present invention, when there are multiple current speakers in the preset virtual space, the sound sources of the audio data of the multiple current speakers are identified through the relative position relationship between the multiple current speakers and the listeners, and the binaural effect of humans can be simulated in the preset virtual space to determine the positions of the speakers, thereby achieving the purpose of identifying the sound sources of the audio data of the multiple speaker virtual images according to the relative position relationship between the multiple speaker virtual images and the listener virtual images in the preset virtual space, thereby achieving the technical effect of enhancing the recognition of audio data and improving user experience when multiple speakers are speaking at the same time in the virtual space, and further solving the technical problem of mixed sound sources and poor user experience when multiple speakers are speaking at the same time in the virtual space. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0015] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for identifying the sound source of audio data is shown;

[0016] Figure 2 is a schematic diagram of a hardware environment of a virtual reality device according to a method for identifying a sound source of audio data according to an embodiment of the present invention;

[0017] Figure 3 is a structural block diagram of a computing environment of a method for identifying a sound source of audio data according to an embodiment of the present invention;

[0018] Figure 4 is a flow chart of a method for identifying a sound source of audio data according to an embodiment of the present invention;

[0019] Figure 5 is a schematic diagram of a graphical user interface of a virtual conference room according to an embodiment of the present invention;

[0020] Figure 6 is a schematic diagram of a signal weight calculation method according to an embodiment of the present invention;

[0021] Figure 7 is a schematic diagram of a multi-channel audio playback process in a virtual conference room according to an embodiment of the present invention;

[0022] Figure 8 is a schematic diagram of a dual-channel playback method for audio data according to an embodiment of the present invention;

[0023] Fig. 9 is a flow chart of another method for identifying the sound source of audio data according to an embodiment of the present invention;

[0024] Fig.10 is a flow chart of another method for identifying the sound source of audio data according to an embodiment of the present invention;

[0025] Fig.11 is a schematic structural diagram of a device for identifying a sound source of audio data according to an embodiment of the present invention;

[0026] Fig.12 is a schematic structural diagram of an optional device for identifying the sound source of audio data according to an embodiment of the present invention;

[0027] Fig.13 is a schematic structural diagram of another optional device for identifying the sound source of audio data according to an embodiment of the present invention;

[0028] Fig.14 is a schematic structural diagram of another device for identifying the sound source of audio data according to an embodiment of the present invention;

[0029] Fig.15 is a schematic structural diagram of another optional device for identifying the sound source of audio data according to an embodiment of the present invention;

[0030] Fig.16 is a schematic structural diagram of another device for identifying the sound source of audio data according to an embodiment of the present invention;

[0031] Fig.17 is a structural block diagram of another computer terminal according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] First, some nouns or terms that appear in the process of describing the embodiments of the present invention are subject to the following explanations:

[0035] Virtual conference room: refers to an application product that realizes conference scenes through online virtual digital space. Through the virtual conference room, users can remotely join virtual meetings. The user terminal can restore the conference scene for users through technologies such as 3D rendering and virtual reality (VR). Users can also participate in virtual meetings (such as speaking or listening) through devices such as cameras, microphones, and speakers associated with the user terminal.

[0036] Binaural effect: refers to the effect that humans can judge the direction of sound by relying on the volume difference, time difference and timbre difference between the two ears.

[0037] Spatial audio: refers to audio playback technology that uses multi-channel, surround sound and other technologies to provide spatial location information that can perceive the source of sound.

[0038] Example 1

[0039] According to an embodiment of the present invention, there is also provided an embodiment of a method for identifying the sound source of audio data. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] The method embodiment provided in the first embodiment of the present invention may be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for identifying the sound source of audio data is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (shown as 102a, 102b, ..., 102n in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor (Microcontroller Unit, MCU) or a programmable logic device (Field Programmable Gate Array, FPGA)), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, the computer terminal 10 may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a cursor control device (such as a mouse, a touchpad, etc.), a keyboard, a power supply and / or a camera.

[0041] It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0042] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As involved in the embodiments of the present invention, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0043] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for identifying the sound source of audio data in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned method for identifying the sound source of audio data is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0044] The transmission device 106 is used to connect to a network via a network interface to receive or send data. The specific examples of the above network may include a wired and / or wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0045] like Figure 1 The display shown may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0046] It should be noted that, in some optional embodiments, the above Figure 1 The computer device (or mobile device) shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the above-described computer device (or mobile device).

[0047] It should be noted that, in some embodiments, the above Figure 1 The computer device (or mobile device) shown has a touch display (also called a "touch screen" or "touch display screen"). In some embodiments, the above Figure 1The computer device (or mobile device) shown has a graphical user interface (GUI), and a user can perform human-computer interaction with the GUI through finger contacts and / or gestures on the touch-sensitive surface. The human-computer interaction functions here optionally include the following interactions: creating web pages, drawing, word processing, making electronic documents, games, video conferencing, instant messaging, sending and receiving emails, call interface, playing digital videos, playing digital music and / or web browsing, etc. The executable instructions for performing the above-mentioned human-computer interaction functions are configured / stored in a computer program product executable by one or more processors or a readable storage medium.

[0048] The method embodiment provided in the first embodiment of the present invention can be executed in a virtual reality (VR) device or an augmented reality (AR) device. Taking the VR device as an example, Figure 2 FIG. 1 is a schematic diagram of a hardware environment of a virtual reality device for a method of identifying the sound source of audio data according to an embodiment of the present invention. Figure 2 As shown, the virtual reality device 204 is connected to the terminal 206, and the terminal 206 is connected to the server 202 through a network. The above-mentioned virtual reality device 204 is not limited to: virtual reality helmets, virtual reality glasses, virtual reality all-in-one machines, etc., and the above-mentioned terminal 204 is not limited to PCs, mobile phones, tablet computers, etc. The server 202 can be a server corresponding to a media file operator, and the above-mentioned network includes but is not limited to: a wide area network, a metropolitan area network or a local area network.

[0049] Optionally, the virtual reality device 204 of the embodiment of the present invention includes: a memory 2041, a processor 2042 and a transmission device 2043 ( Figure 2 The memory 2041 is used to store an application program, which can be used to execute the method for identifying the sound source of audio data provided in the embodiment of the present invention. The processor 2042 can call and execute the application program stored in the memory 2041 to obtain the sound source of the identified audio data through the method for identifying the sound source of audio data provided in the embodiment of the present invention.

[0050] Optionally, the terminal 206 of the embodiment of the present invention can also be used to execute the method for identifying the sound source of audio data, and display the sound source information of the audio data through the display screen of the VR device or AR device, or play the audio generated in the process of executing the method for identifying the sound source of audio data through an audio output device associated with the VR device or AR device.

[0051] Figure 2 The hardware structure block diagram shown can not only serve as an exemplary block diagram of the hardware environment of the above-mentioned AR / VR device (or mobile device), but also as an exemplary block diagram of the above-mentioned server.

[0052] Figure 3 is a structural block diagram of a computing environment of a method for identifying a sound source of audio data according to an embodiment of the present invention, Figure 3 The block diagram shows the use of the above Figure 2 The VR device (or mobile device) shown is an embodiment of a computing node in the computing environment 301.

[0053] Still like Figure 3 As shown, the computing environment 301 includes multiple ( Figure 3 310-1, 310-2, ..., are used to illustrate computing nodes (such as servers). Each computing node includes local processing and memory resources, and end users 302 can remotely run applications or store data in computing environment 301. Applications can be provided as multiple services 320-1 (representing service A), 320-2 (representing service D), 320-3 (representing service E) and 320-4 (representing service H) in computing environment 301.

[0054] Still like Figure 3 As shown, the end user 302 can provide and access services through a web browser or other software application on the client, and in some embodiments, the provision and / or request of the end user 302 can be provided to the entry gateway 330. The entry gateway 330 may include a corresponding agent to handle the provision and / or request for the service 320 (one or more services provided in the computing environment 301).

[0055] Still like Figure 3 As shown, service 320 is provided or deployed according to various virtualization technologies supported by computing environment 301. In some embodiments, service 320 can be provided according to virtualization based on virtual machine (VM), virtualization based on container and / or similar methods. Virtualization based on virtual machine can be to simulate a real computer by initializing a virtual machine, and execute programs and applications without directly contacting any actual hardware resources. While the virtual machine virtualizes the machine, according to container-based virtualization, the container can be started to virtualize the entire operating system (OS) so that multiple workloads can run on a single operating system instance.

[0056] In an embodiment based on container virtualization, several containers of service 320 can be assembled into a POD (e.g., Kubernetes POD). Figure 3As shown, service 320-2 can be equipped with one or more PODs 340-1, 340-2, ..., 340-N (collectively referred to as POD 340). Each POD 340 can include a proxy 345 and one or more containers 342-1, 342-2, ..., 342-M (collectively referred to as containers 342). One or more containers 342 in POD 340 process requests related to one or more corresponding functions of the service, and the proxy 345 generally controls network functions related to the service, such as routing, load balancing, etc. Other services 320 can also be PODs similar to POD 340.

[0057] During operation, executing a user request from an end user 302 may require invoking one or more services 320 in the computing environment 301, and executing one or more functions of a service 320 may require invoking one or more functions of another service 320. Figure 3 As shown, service 320-1 (representing service A) receives a user request from terminal user 302 from the entry gateway 330, service 320-1 (representing service A) can call service 320-2 (representing service D), and service 320-2 (representing service D) can request service 320-3 (representing service E) to perform one or more functions.

[0058] The computing environment described above can be a cloud computing environment, where the allocation of resources is managed by the cloud service provider, allowing the development of functions without considering the implementation, adjustment or expansion of servers. The computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be divided into a set of functions that can be automatically and independently scaled, rather than expanding a single hardware device to handle potential loads.

[0059] Under the above operating environment, the present invention provides Figure 4 A method for identifying the sound source of audio data is shown. Figure 4 is a flow chart of a method for identifying the sound source of audio data according to an embodiment of the present invention. Figure 4 As shown, the method for identifying the sound source of audio data includes:

[0060] Step S41, receiving audio data and first position data from a plurality of first virtual images in a preset virtual space, wherein the current identities of the plurality of first virtual images are speaker identities;

[0061] Step S42, determining a spatial position relationship using the first position data and the second position data, wherein the second position data is position data of the second virtual image, the current identity of the second virtual image is the identity of the listener, and the spatial position relationship is used to determine a relative position relationship between the second virtual image and each of the plurality of first virtual images;

[0062] Step S43, identifying the sound source of the audio data based on the spatial position relationship.

[0063] The above-mentioned preset virtual space may be a virtual digital space corresponding to a virtual conference room. In the embodiment of the present invention, a virtual conference room is taken as an example to illustrate the technical solution below.

[0064] In an embodiment of the present invention, the first virtual image may be a virtual image corresponding to the speaker at the current moment in the virtual conference room. The multiple first virtual images may be multiple virtual images corresponding to multiple speakers simultaneously existing in the virtual conference room. The audio data of the multiple first virtual images may be audio data emitted by multiple speakers in the virtual conference room. The first position data of the multiple first virtual images may be spatial coordinate position information of each virtual object in the multiple first virtual objects in the virtual conference room.

[0065] In an embodiment of the present invention, the second virtual image may be a virtual image corresponding to a current listener in the virtual conference room. The second position data may be spatial coordinate position information of the second virtual object in the virtual conference room.

[0066] The spatial position relationship can be determined by using the first position data and the second position data. The spatial position relationship can be a relative position relationship (e.g., including direction, distance, etc.) between the virtual image corresponding to the listener at the current moment in the virtual conference room (equivalent to the second virtual image) and the virtual image corresponding to each of the multiple speakers (equivalent to the first virtual image).

[0067] In the embodiment of the present invention, based on the above spatial position relationship, the sound source of the audio data can be identified. For example, in a virtual conference room, based on the relative position relationship between the virtual image corresponding to the listener and the virtual image corresponding to each of the multiple speakers, it can be identified which sound in the audio data comes from which speaker among the multiple speakers.

[0068] It is easy to understand that the focus of the above method provided by the embodiment of the present invention is: it can imitate the human binaural effect and realize audio space source identification in a preset virtual space. Specifically, it can help listeners quickly identify the current speaker in a virtual conference room.

[0069] Through the above-mentioned steps S41 to S43 of the embodiment of the present invention, audio data and first position data are received from multiple first virtual images in a preset virtual space, wherein the current identities of the multiple first virtual images are the speaker identities, and the spatial position relationship is determined by utilizing the first position data and the second position data, wherein the second position data is the position data of the second virtual image, and the current identity of the second virtual image is the listener identity. The spatial position relationship is used to determine the relative position relationship between the second virtual image and each of the multiple first virtual images, and further identify the sound source of the audio data based on the spatial position relationship.

[0070] It is easy to notice that, through the embodiments of the present invention, when there are multiple current speakers in the preset virtual space, the sound sources of the audio data of the multiple current speakers are identified through the relative position relationship between the multiple current speakers and the listeners, and the binaural effect of humans can be simulated in the preset virtual space to determine the positions of the speakers, thereby achieving the purpose of identifying the sound sources of the audio data of the multiple speaker virtual images according to the relative position relationship between the multiple speaker virtual images and the listener virtual images in the preset virtual space, thereby achieving the technical effect of enhancing the recognition of audio data and improving user experience when multiple speakers are speaking at the same time in the virtual space, and further solving the technical problem of mixed sound sources and poor user experience when multiple speakers are speaking at the same time in the virtual space.

[0071] The above method of the embodiment of the present invention is further introduced below.

[0072] In an optional embodiment, in step S42, determining the spatial position relationship using the first position data and the second position data includes the following method steps:

[0073] Step S421, performing a difference operation on the first coordinate position of each first virtual image in the preset virtual space and the second coordinate position of the second virtual image in the preset virtual space to obtain a spatial position relationship, wherein the spatial position relationship includes: a relative orientation relationship between the second virtual image and each first virtual image in the multiple first virtual images, and a relative distance relationship between the second virtual image and each first virtual image in the multiple first virtual images.

[0074] In the above optional embodiment, the difference calculation performed may be a spatial vector difference calculation. A spatial vector is determined for the first coordinate position of each first virtual image in the preset virtual space and the second coordinate position of the second virtual image in the preset virtual space among the multiple first virtual images, and the length of the spatial vector may be used to represent the relative distance relationship between each first virtual image and the second virtual image in the preset virtual space, and the direction of the spatial vector may be used to represent the relative orientation relationship between each first virtual image and the second virtual image in the preset virtual space, wherein the direction of the spatial vector may be represented by the angle between the spatial vector and the channel direction (such as the left channel direction or the right channel direction of the current listener).

[0075] The method for identifying the sound source of audio data provided by the embodiment of the present invention can be applied to virtual conferences in e-commerce service scenarios, education and training scenarios, medical and health scenarios, and scientific research and academic scenarios. The following takes the virtual conference room in the e-commerce service scenario as an example to specifically describe the technical solution of the embodiment of the present invention.

[0076] Figure 5 is a schematic diagram of a graphical user interface of a virtual conference room according to an embodiment of the present invention. Figure 5 As shown, in the virtual conference room M1 in the e-commerce service scenario, it can include: a virtual conference hall, multiple virtual seats, and multiple virtual images of participants ( Figure 5 Speaker A, speaker B and participant C are displayed in the figure. In this example, participant C is a virtual character corresponding to the current user, and the current identity of participant C is a listener.

[0077] Still like Figure 5 As shown, in the e-commerce service scenario, the application product for providing the virtual conference room M1 can provide the following functions: "Invite to the lobby" function (used to invite people who are not currently in the meeting to join the current virtual conference room M1), "Sit down" function (used to control the virtual image corresponding to the current user to sit in the designated virtual seat), "Mute" function (used to turn off the microphone of the current user), "Volume" function (used to adjust the volume of the current user's speaker or headphones), "Share" function (used to broadcast the current user's screen to other participants in the virtual conference room M1), "Settings" function (used to support the current user to adjust the system preferences for the virtual conference room M1), and "Exit" function (used to exit the virtual conference room M1).

[0078] Still like Figure 5 As shown, in the virtual conference room M1, speaker A is located on the left side of participant C, and speaker B is located on the right side of participant C.

[0079] Exemplarily, in a virtual conference room M1 in an e-commerce service scenario, the process of identifying the sound source of audio data for participant C may include: receiving position coordinates XA and audio data SA sent by speaker A, and receiving position coordinates XB and audio data SB sent by speaker B.

[0080] It should be noted that the position coordinate XA is the spatial coordinate of speaker A in the virtual conference room M1, the audio data SA is the sound data emitted by speaker A, the position coordinate XB is the spatial coordinate of speaker B in the virtual conference room M1, and the audio data SB is the sound data emitted by speaker B.

[0081] In the virtual conference room M1 , the current device used by the participant C can provide the location coordinates XC of the participant C.

[0082] Using the above-mentioned position coordinates XA of speaker A and the position coordinates XC of participant C, the spatial position relationship between speaker A and participant C can be determined in the virtual conference room M1; using the above-mentioned position coordinates XB of speaker B and the position coordinates XC of participant C, the spatial position relationship between speaker B and participant C can be determined in the virtual conference room M1.

[0083] It should be noted that the virtual conference room M1 can be a virtual two-dimensional conference room or a virtual three-dimensional conference room. In a virtual two-dimensional conference room, the spatial coordinates of all participants (including speakers and listeners) are two-dimensional coordinates; in a virtual three-dimensional conference room, the spatial coordinates of all participants (including speakers and listeners) are three-dimensional coordinates.

[0084] In an optional embodiment, in step S43, identifying the sound source of the audio data based on the spatial position relationship includes the following method steps:

[0085] Step S431, determining a weight coefficient corresponding to each channel component of the audio data on a target playback device by using a spatial position relationship, wherein the target playback device is a playback device associated with the second avatar and the target playback device supports multi-channel spatial audio playback;

[0086] Step S432, synthesizing multi-channel spatial audio based on the weight coefficient corresponding to each channel component on the target playback device;

[0087] Step S433: identifying the sound source of the audio data through multi-channel spatial audio.

[0088] In the above optional embodiment, the target playback device may be an audio output device used by the user who is currently a listener, and the audio output device may support multi-channel spatial audio playback. The weight coefficient corresponding to each channel component of the audio data of the multiple first virtual images on the above target playback device may be determined by using the relative position relationship and relative distance relationship between the second virtual image and each of the multiple first virtual images.

[0089] The weight coefficients can be used to represent the signal strength of the audio data corresponding to each first virtual image among the multiple first virtual images on each channel of the target playback device. For example, the larger the weight coefficient of a first virtual image on the left channel of the target playback device, the clearer the current listener can hear the sound of the first virtual image in the left channel (e.g., the louder the volume).

[0090] It is easy to understand that by synthesizing multi-channel spatial audio based on the weight coefficient corresponding to each channel component on the target playback device, and identifying the sound source of the audio data through multi-channel spatial audio, it is possible to combine the spatial position relationship of multiple speakers in the virtual conference room to help the current listener quickly distinguish the sound source.

[0091] In an optional embodiment, in step S431, the weight coefficient corresponding to each channel component of the audio data on the target playback device is determined by using the spatial position relationship, including the following method steps:

[0092] Step S4311, determining a target length and a target angle using a spatial position relationship, wherein the target length is the length of a target vector formed by a listening position corresponding to each channel component on a target playback device and a first coordinate position of each first virtual image in a plurality of first virtual images, and the target angle is the angle between a direction corresponding to the target vector and a direction corresponding to each channel component on the target playback device;

[0093] Step S4312, calculating a first attenuation coefficient corresponding to each channel component on the target playback device based on the target length, and calculating a second attenuation coefficient corresponding to each channel component on the target playback device based on the target angle, wherein the first attenuation coefficient is a distance attenuation coefficient, and the second attenuation coefficient is a directivity attenuation coefficient;

[0094] Step S4313: The first attenuation coefficient and the second attenuation coefficient are used to calculate a weight coefficient corresponding to each channel component on the target playback device.

[0095] In the above optional embodiments, the listening position corresponding to each channel component on the target playback device may correspond to multiple listening positions of the current listener, and the direction corresponding to the target vector may correspond to the listening direction (such as binaural orientation) of the current listener. For example, the listening position of the left channel component may correspond to the left ear position of the current listener, the listening position of the right channel component may correspond to the right ear position of the current listener, the listening direction of the left channel may correspond to the left ear orientation of the current listener, and the listening direction of the right channel may correspond to the right ear orientation of the current listener. The left ear position, the right ear position, the left ear orientation, and the right ear orientation may be calculated based on the position coordinates of the current listener using preset rules, or may be acquired in real time using other smart devices used by the current listener (such as smart wearable devices: smart glasses, smart watches, smart headsets, etc., which are associated with the smart phone used by the current listener).

[0096] Specifically, for example, if the current listener is wearing a smart headset associated with a smartphone, the smart headset is equipped with a gyroscope, and the current listener's motion is captured through the camera of the smartphone, and the current listener's facial orientation is obtained using an optical recognition algorithm. Based on the facial orientation, the real-time orientation of the current listener's ears is obtained in real time using the smart headset (or in combination with a camera), and the real-time orientation is used to determine the spatial position relationship of the current listener in the virtual conference room, and then the attenuation coefficient and channel weight are calculated. For example, when the head of the current listener C deflects to the left to face the speaker A, the relative orientation relationship between the speaker A and the listener C changes from "A is on the left side of C" to "A is directly in front of C". In this way, the current listener's sense of immersion in the meeting can be increased.

[0097] The above optional embodiments of the present invention are still described in detail by taking the example of identifying the sound source of audio data for participant C in the virtual conference room M1.

[0098] Using the position coordinates XA of the speaker A and the position coordinates XC of the participant C, the space vector can be determined in the virtual conference room M1. Space Vector The length of represents the relative distance between speaker A and participant C. Since speaker A is located on the left side of participant C, the space vector The angle between the speaker A and the left channel L of participant C represents the relative position relationship between speaker A and participant C.

[0099] Similarly, using the position coordinates XB of speaker B and XC of participant C, the space vector can be determined in the virtual conference room M1. Space Vector The length of represents the relative distance between speaker B and participant C. Since speaker B is located on the right side of participant C, the space vector The angle between speaker B and participant C's right channel R indicates the relative position relationship between speaker B and participant C.

[0100] Exemplarily, in a virtual conference room M1 under an e-commerce service scenario, the process of identifying the sound source of audio data for participant C may include: calculating the weights of the signal amplitudes of speakers A and B in the left channel L and right channel R of participant C based on the position coordinates XA of speaker A, the position coordinates XB of speaker B, and the position coordinates XC of participant C.

[0101] It should be noted that the above signal amplitude weights include: the signal weight W of speaker A in the left channel L of participant C LA , the signal weight W of speaker A in the right channel R of participant C RA , the signal weight W of speaker B in the left channel L of participant C LB , the signal weight W of speaker B in the right channel R of participant C RB .

[0102] Figure 6 is a schematic diagram of a signal weight calculation method according to an embodiment of the present invention. Figure 6 As shown, the signal weight W of speaker A in the left channel L of participant C is LA Take this as an example, and calculate the signal weight W LA The specific implementation method may include the following method steps:

[0103] Step E51, according to a preset human ear offset model (which can be predetermined by a technician with reference to binaural effects in real scenes according to the method provided by the present invention), the following formula (1) is used to calculate the listening position C of the left ear of the participant C in the virtual conference room M1: L :

[0104] C L =XC+(-1,0)×r Formula (1)

[0105] In the above formula (1), XC is the spatial coordinate of participant C in the virtual conference room M1, (-1, 0) represents the direction of the left channel L of participant C, and r represents a preset constant;

[0106] Step E52, based on the listening position C of the left ear of participant C L and the position coordinates XA of speaker A, determine the vector (From C L The length d of the vector pointing to XA) and the determination of the vector The angle θ between the direction of the left channel L of participant C;

[0107] Step E53, according to the propagation law of the point sound source in the real scene in the free space, a preset inverse function is used for fitting to obtain the attenuation coefficient a1 (equivalent to the above-mentioned first attenuation coefficient) of the audio data SA of speaker A in the space of the virtual conference room M1. In this example, a1=d -1 ;

[0108] Step E54, according to the directivity law of the human ear in the real scene, a preset cardioid function is used for fitting to obtain the directivity attenuation coefficient a2 (equivalent to the above-mentioned second attenuation coefficient) of the audio data SA of speaker A. In this example,

[0109] Step E55, multiply the attenuation coefficient a1 by the directivity attenuation coefficient a2 to obtain the signal weight W of speaker A in the left channel L of participant C. LA =a1×a2.

[0110] It should be noted that the preset inverse function used in the above step E53 and the preset cardioid function used in the above step E54 are only possible examples. In specific application scenarios, other possible functions can be used for fitting and weight calculation according to scenario requirements.

[0111] In an optional embodiment, in step S4312, the second attenuation coefficient corresponding to each channel component on the target playback device is calculated based on the target angle, including the following method steps:

[0112] Step S4314, performing a fast Fourier transform on the audio data corresponding to each channel component on the target playback device to obtain a transformation result;

[0113] Step S4315: Based on the transformation result, the target angle is used to calculate the second attenuation coefficient corresponding to each channel component on the target playback device.

[0114] In the above optional embodiment, when the directional attenuation coefficient corresponding to each channel component on the target playback device is calculated based on the target angle (equivalent to the above-mentioned second attenuation coefficient), factors such as the phase difference and spectral influence caused by different sound sources in the virtual space can also be taken into consideration to obtain a more realistic spatial audio effect.

[0115] The above optional embodiments of the present invention are still described in detail by taking the example of identifying the sound source of audio data for participant C in the virtual conference room M1.

[0116] When calculating the signal weight W of speaker A in the left channel L of participant C LAIn the above step E54, in the process of calculating the directivity attenuation coefficient a2 (equivalent to the above second attenuation coefficient), since the human ear has different sound receiving directivities in different frequency domains, the audio data SA can be firstly subjected to a fast Fourier transform (FFT) to obtain the changed audio data SA2, and then based on the audio data SA2, the audio data SA2 can be further converted into the audio data SA2 by the method of the method described above. Different directivity attenuation functions a2 corresponding to different frequency points are calculated.

[0117] In an optional embodiment, in step S432, the multi-channel spatial audio is synthesized based on the weight coefficient corresponding to each channel component on the target playback device, including the following method steps:

[0118] Step S4321: Perform a weighted sum operation based on the weight coefficient corresponding to each channel component on the target playback device and the audio data corresponding to each channel component to obtain multi-channel spatial audio.

[0119] In the above optional embodiment, the weighted sum calculation performed may be a linear weighted calculation.

[0120] The above optional embodiments of the present invention are still described in detail by taking the example of identifying the sound source of audio data for participant C in the virtual conference room M1.

[0121] For example, in a virtual conference room M1 in an e-commerce service scenario, the process of identifying the sound source of audio data for participant C may include: after calculating the weights of the signal amplitudes of speaker A and speaker B in the left channel L and the right channel R of participant C, weighted synthesis is performed on each signal in the left channel L and the right channel R to obtain the left channel L playback audio Audio of participant C. L And the right channel R plays back the audio Audio R .

[0122] Specifically, for example, in the virtual conference room M2, there are N current speakers speaking at the same time, and the audio data of the N current speakers received by the current listener are recorded as S1, S2, ..., S N The signal weight of the N current speakers in the left channel L of the current listener is recorded as W L1 , W L2 , ..., W LN The signal weight of the N current speakers in the right channel R of the current listener is recorded as W R1 , W R2 , ..., W RN When playing back the audio of virtual conference room M2 for the current listener, the playback audio of left channel L Right channel R playback audio

[0123] Figure 7 is a schematic diagram of a multi-channel audio playback process in a virtual conference room according to an embodiment of the present invention. Figure 7 As shown, in the virtual conference room, the speech waveforms and spatial coordinates of speaker A and speaker B are received through the microphone and the network; then based on the spatial coordinates of speaker A and speaker B, the spatial restoration of speaker A, speaker B and participant C is performed to obtain the weight A of speaker A to participant C (for example, it can include W LA =0.8 and W RA = 0.2) and the weight B of speaker B to participant C (for example, it can include W LB =0.3 and W RB =0.7); further, performing weighted synthesis based on the speech waveforms of speaker A and speaker B, weight A and weight B to obtain multi-channel playback audio for the current listener; and playing the multi-channel playback audio in a speaker or headset supporting multi-channel associated with the current listener.

[0124] It is easy to understand that in the above virtual conference room M1, speaker A is located on the left side of participant C. Therefore, speaker A has a higher weight on the left channel L of participant C, that is, W LA >W LB , participant C mainly listens to the voice from speaker A on the left channel L; similarly, speaker B is located on the right side of participant C, so speaker B has a higher weight on the right channel R of participant C, that is, W RB >W RA , participant C mainly listens to the voice from speaker B on the right channel R.

[0125] Figure 8 is a schematic diagram of a dual-channel playback method for audio data according to an embodiment of the present invention. Figure 8 As shown, participant C mainly listens to the voice from speaker A on the left channel L, and mainly listens to the voice from speaker B on the right channel R.

[0126] It is easy to notice that, through the above method provided by the embodiment of the present invention, participant C in the virtual conference room M1 can more easily distinguish that the sound on the "left side" (that is, the sound mainly listened to in the left channel L) comes from speaker A, and the sound on the "right side" (that is, the sound mainly listened to in the right channel R) comes from speaker B, and when speaker A and speaker B speak at the same time, participant C can also distinguish the voices of speaker A and speaker B through the sounds of the left and right channels, thereby eliminating the problem of mixed sounds in the virtual conference room in the prior art.

[0127] It is easy to notice that in traditional offline meetings, when there are multiple speakers at the same time, the meeting listeners can distinguish the sounds from different sources through the binaural effect, and focus their attention on the source of the sound of interest through the brain (filtering or ignoring other sound sources). In an embodiment of the present invention, the recognition of audio data can be enhanced through the spatial relative relationship between virtual images (which can be speakers or listeners) in the virtual conference room scene. In other words, the binaural effect of humans can be imitated to determine which speaker the sound of the audio data comes from, thereby solving the problem of mixed sounds when there are multiple speakers at the same time in the virtual conference room scene, and improving the meeting experience of participants in the virtual conference room.

[0128] In an optional embodiment, the method for identifying the sound source of audio data further includes the following method steps:

[0129] Step S44, obtaining a phase difference between different channel components of the audio data on a target playback device, wherein the phase difference is used to assist in determining the position of each first virtual image among the plurality of first virtual images;

[0130] Step S45: inserting a phase difference in the time domain corresponding to each channel component on the target playback device.

[0131] In the above optional embodiment, due to the different binaural listening positions of people, when playing back audio data in multi-channel, the phase difference between different channel components of the audio data on the target playback device can also be considered, thereby improving the quality of multi-channel playback audio and enhancing the user's listening experience.

[0132] Still taking the example of identifying the sound source of audio data for participant C in the virtual conference room M1, the above optional embodiment of the present invention is specifically described. The distance between speaker A and participant C is d AC , the distance between speaker B and participant C is d BC Since usually d AC With d BC are not equal, the phase difference can be considered when playing back the multi-channel spatial audio data for participant C.

[0133] Specifically, assuming that d AC >d BC , calculate the phase difference according to the following formula (2):

[0134]

[0135] In the above formula (2), f represents the frequency of the sound (which can be preset by a technician), and v represents the speed of sound propagation (which can be preset by a technician).

[0136] When playing back the multi-channel spatial audio data for participant C, the above phase difference is added to the audio of the phase-lag channel. Speaker A is on the left side of participant C. For the audio data SA of speaker A, the right channel R of participant C is a phase-lag channel. R When the phase of speaker A's audio data SA is shifted backward Similarly, speaker B is located on the right side of participant C. For speaker B's audio data SB, participant C's left channel L is a phase-lag channel. L When the phase of speaker B's audio data SB is shifted backward In this way, the synthesized multi-channel playback audio can have a more realistic effect, thereby improving the user's conference experience.

[0137] In an optional embodiment, a graphical user interface is provided by a terminal device, and the content displayed by the graphical user interface at least partially includes a virtual space scene. The method for identifying the sound source of audio data further includes the following method steps:

[0138] Step S461, in response to a first touch operation on the graphical user interface, adding a plurality of first virtual images to a preset virtual space;

[0139] Step S462, in response to a second touch operation on the graphical user interface, enabling a virtual speaker control corresponding to the second virtual image;

[0140] Step S463: Perform spatial audio playback on the audio data of the multiple first virtual images through the target playback device corresponding to the virtual speaker control to identify the sound source of the audio data.

[0141] In the above optional embodiment, a virtual space scene may be displayed in the graphical user interface.

[0142] In the above optional embodiment, the user can perform a first touch operation on the graphical user interface. The user can touch the "invite new member" button or the plus button in the graphical user interface, or perform a specified first touch gesture in a preset touch area to implement the following process: adding multiple first virtual images to the preset virtual space, the multiple first virtual images are virtual images corresponding to multiple participants (can be speakers or listeners) in the preset virtual space.

[0143] In the above optional embodiment, the user can perform a first touch operation on the graphical user interface. The user can touch the "turn on microphone" button or microphone button in the graphical user interface, or perform a specified second touch gesture in a preset touch area to implement the following process: enable the virtual speaker control corresponding to the second virtual image. The second virtual image can be a virtual image whose current identity is a speaker in a preset virtual space. After enabling the virtual speaker control corresponding to the second virtual image, the sound emitted by the speaker corresponding to the second virtual image can be broadcast to other participants in the preset virtual space.

[0144] In response to the above-mentioned second touch operation, after enabling the virtual speaker control corresponding to the second virtual image, the above-mentioned method for identifying the sound source of audio data provided by an embodiment of the present invention can be used to perform spatial audio playback of the audio data of multiple first virtual images through the target playback device corresponding to the virtual speaker control, so as to help the current listener identify the sound source of the audio data.

[0145] In particular, the first touch operation and the second touch operation can both be operations in which a user touches the display screen of the terminal device with a finger and touches the terminal device. The touch operation can include single-point touch and multi-point touch, wherein the touch operation of each touch point can include click, long press, heavy press, swipe, etc. The first touch operation and the second touch operation can also be touch operations implemented by input devices such as a mouse and a keyboard.

[0146] Under the above operating environment, the present invention provides Fig. 9 A method for identifying the sound source of audio data is shown. Fig. 9 FIG. 4 is a flowchart of another method for identifying the sound source of audio data according to an embodiment of the present invention. Fig. 9 As shown, the method for identifying the sound source of audio data includes:

[0147] Step S91, receiving audio data and first position data from a plurality of first virtual images in a virtual conference room, wherein the current identities of the plurality of first virtual images are conference speakers;

[0148] Step S92, determining a spatial position relationship using the first position data and the second position data, wherein the second position data is position data of the second virtual image, the current identity of the second virtual image is the identity of a conference listener, and the spatial position relationship is used to determine a relative position relationship between the second virtual image and each of the plurality of first virtual images;

[0149] Step S93: Identify the sound source of the audio data in the virtual conference room based on the spatial position relationship.

[0150] In an embodiment of the present invention, the first virtual image may be a virtual image corresponding to the current conference speaker in the virtual conference room. The multiple first virtual images may be multiple virtual images corresponding to multiple conference speakers simultaneously existing in the virtual conference room. The audio data of the multiple first virtual images may be audio data emitted by multiple conference speakers in the virtual conference room. The first position data of the multiple first virtual images may be spatial coordinate position information of each virtual object in the multiple first virtual objects in the virtual conference room.

[0151] In an embodiment of the present invention, the second virtual image may be a virtual image corresponding to a conference listener at the current moment in the virtual conference room. The second position data may be spatial coordinate position information of the second virtual object in the virtual conference room.

[0152] The spatial position relationship can be determined by using the first position data and the second position data. The spatial position relationship can be a relative position relationship (e.g., including direction, distance, etc.) between the virtual image corresponding to the conference listener at the current moment in the virtual conference room (equivalent to the second virtual image) and the virtual image corresponding to each of the multiple conference speakers (equivalent to the first virtual image).

[0153] In the embodiment of the present invention, based on the above spatial position relationship, the sound source of the audio data in the virtual conference room can be identified. In the virtual conference room, based on the relative position relationship between the virtual image corresponding to the conference listener and the virtual image corresponding to each of the multiple conference speakers, it can be identified which sound in the audio data comes from which conference speaker among the multiple conference speakers.

[0154] It is easy to understand that the focus of the above method provided by the embodiment of the present invention is: it can imitate the human binaural effect and realize audio space source recognition in a virtual conference room. Specifically, it can help conference listeners quickly identify the current conference speaker in the virtual conference room.

[0155] Through the above-mentioned steps S91 to S93 of the embodiment of the present invention, audio data and first position data are received from multiple first virtual images in the virtual conference room, wherein the current identities of the multiple first virtual images are conference speakers, and the spatial position relationship is determined by utilizing the first position data and the second position data, wherein the second position data is the position data of the second virtual image, and the current identity of the second virtual image is the conference listener. The spatial position relationship is used to determine the relative position relationship between the second virtual image and each of the multiple first virtual images, and further identify the sound source of the audio data in the virtual conference room based on the spatial position relationship.

[0156] It is easy to notice that, through the embodiments of the present invention, when there are multiple current conference speakers in the virtual conference room, the sound sources of the audio data of the multiple current conference speakers in the virtual conference room are identified through the relative position relationship between the multiple current conference speakers and the conference listeners, and the positions of the conference speakers can be determined by simulating the binaural effect of humans in the virtual conference room, thereby achieving the purpose of identifying the sound sources of the audio data of the multiple conference speaker virtual images in the virtual conference room according to the relative position relationship between the multiple conference speaker virtual images and the conference listeners virtual images in the virtual conference room, thereby achieving the technical effect of enhancing the recognition of audio data and improving user experience when multiple conference speakers speak at the same time in the virtual space, and further solving the technical problems of mixed sound sources and poor user experience when multiple speakers speak at the same time in the virtual space.

[0157] In traditional conference room products provided by related technologies, the current speaker is usually identified by adjusting the layout of the participant's screen or by highlighting the screen. However, for complex scenarios in virtual conference rooms (where multiple speakers are speaking at the same time), the method of identifying the current speaker in related technologies cannot solve the problem of mixed voices, resulting in a poor experience for participants in virtual conference rooms.

[0158] Compared with the related art, the method provided by the embodiment of the present invention can imitate the binaural effect of humans in a preset virtual space, and provide the position information of the current speaker in the preset virtual space to the listener through a multi-channel playback device, so that the listener can more easily identify the current speaker in the preset virtual space. In addition, the above method provided by the embodiment of the present invention can also alleviate the mixing problem of multi-source audio in the preset virtual space, and can improve the recognition of audio data in the preset virtual space under the same network conditions. In general, the above method provided by the embodiment of the present invention can be applied to related products such as remote conferencing and virtual conferencing, which is conducive to improving the overall conference audio quality and enhancing the conference experience of participants (such as increasing the immersion in the conference).

[0159] In an optional embodiment, a graphical user interface is provided by a terminal device, and the content displayed by the graphical user interface at least partially includes a virtual conference room scene. The method for identifying the sound source of audio data further includes the following method steps:

[0160] Step S94, in response to a first touch operation on the graphical user interface, adding a plurality of first virtual images to the virtual conference room;

[0161] Step S95, in response to a second touch operation on the graphical user interface, enabling a virtual speaker control corresponding to the second virtual image;

[0162] Step S96, performing spatial audio playback on the audio data of the multiple first virtual images through the target playback device corresponding to the virtual speaker control to identify the sound source of the audio data in the virtual conference room.

[0163] In the above optional embodiment, a virtual conference room scene may be displayed in the graphical user interface.

[0164] In the above optional embodiment, the user can perform a first touch operation on the graphical user interface. The user can touch the "invite new member" button or the plus button in the graphical user interface, or perform a specified first touch gesture in a preset touch area to implement the following process: adding multiple first virtual images to the virtual conference room, the multiple first virtual images are virtual images corresponding to multiple participants (who can be speakers or listeners) in the virtual conference room.

[0165] In the above optional embodiment, the user can perform a first touch operation on the graphical user interface. The user can touch the "turn on microphone" button or microphone button in the graphical user interface, or perform a specified second touch gesture in a preset touch area to implement the following process: enable the virtual speaker control corresponding to the second virtual image. The second virtual image can be a virtual image of the current speaker in the virtual conference room. After enabling the virtual speaker control corresponding to the second virtual image, the sound emitted by the speaker corresponding to the second virtual image can be broadcast to other participants in the virtual conference room.

[0166] In response to the second touch operation, after enabling the virtual speaker control corresponding to the second virtual image, the method for identifying the sound source of audio data provided in an embodiment of the present invention can be used to perform spatial audio playback on the audio data of multiple first virtual images through the target playback device corresponding to the virtual speaker control to identify the sound source of the audio data in the virtual conference room.

[0167] In particular, the first touch operation and the second touch operation can both be operations in which a user touches the display screen of the terminal device with a finger and touches the terminal device. The touch operation can include single-point touch and multi-point touch, wherein the touch operation of each touch point can include click, long press, heavy press, swipe, etc. The first touch operation and the second touch operation can also be touch operations implemented by input devices such as a mouse and a keyboard.

[0168] Under the above operating environment, the present invention provides Fig.10 A method for identifying the sound source of audio data is shown. Fig.10 FIG. 4 is a flowchart of another method for identifying the sound source of audio data according to an embodiment of the present invention. Fig.10As shown, the method for identifying the sound source of audio data includes:

[0169] Step S1001, receiving audio data and first position data from a plurality of first virtual images in a virtual classroom, wherein the current identities of the plurality of first virtual images are classroom speakers;

[0170] Step S1002, determining a spatial position relationship using the first position data and the second position data, wherein the second position data is position data of a second virtual image, the current identity of the second virtual image is an identity of a class listener, and the spatial position relationship is used to determine a relative position relationship between the second virtual image and each of the plurality of first virtual images;

[0171] Step S1003, identifying the sound source of the audio data in the virtual classroom based on the spatial position relationship.

[0172] In an embodiment of the present invention, the first virtual image may be a virtual image corresponding to the current classroom speaker in the virtual classroom. The multiple first virtual images may be multiple virtual images corresponding to multiple classroom speakers simultaneously existing in the virtual classroom. The audio data of the multiple first virtual images may be audio data emitted by multiple classroom speakers in the virtual classroom. The first position data of the multiple first virtual images may be the spatial coordinate position information of each virtual object in the multiple first virtual objects in the virtual classroom.

[0173] In an embodiment of the present invention, the second virtual image may be a virtual image corresponding to a classroom listener at the current moment in the virtual classroom. The second position data may be spatial coordinate position information of the second virtual object in the virtual classroom.

[0174] The spatial position relationship can be determined by using the first position data and the second position data. The spatial position relationship can be a relative position relationship (e.g., including direction, distance, etc.) between the virtual image corresponding to the classroom listener at the current moment in the virtual classroom (equivalent to the second virtual image) and the virtual image corresponding to each of the multiple classroom speakers (equivalent to the first virtual image).

[0175] In the embodiment of the present invention, based on the above spatial position relationship, the sound source of the audio data in the virtual classroom can be identified. In the virtual classroom, based on the relative position relationship between the virtual image corresponding to the classroom listener and the virtual image corresponding to each of the multiple classroom speakers, it can be identified which sound in the audio data comes from which classroom speaker among the multiple classroom speakers.

[0176] It is easy to understand that the focus of the above method provided by the embodiment of the present invention is: it can imitate the human binaural effect and realize audio space source recognition in a virtual classroom. Specifically, it can help classroom listeners quickly identify the current classroom speaker in a virtual classroom.

[0177] Through the above-mentioned steps S1001 to S1003 of the embodiment of the present invention, audio data and first position data are received from multiple first virtual images in the virtual classroom, wherein the current identities of the multiple first virtual images are classroom speakers, and the spatial position relationship is determined by utilizing the first position data and the second position data, wherein the second position data is the position data of the second virtual image, and the current identity of the second virtual image is the classroom listener. The spatial position relationship is used to determine the relative position relationship between the second virtual image and each of the multiple first virtual images, and further identify the sound source of the audio data in the virtual classroom based on the spatial position relationship.

[0178] It is easy to notice that, through the embodiments of the present invention, when there are multiple current classroom speakers in the virtual classroom, the sound sources of the audio data of the multiple current classroom speakers in the virtual classroom are identified through the relative position relationship between the multiple current classroom speakers and the classroom listeners, and the binaural effect of humans can be simulated in the virtual classroom to determine the positions of the classroom speakers, thereby achieving the purpose of identifying the sound sources of the audio data of the multiple classroom speaker virtual images in the virtual classroom according to the relative position relationship between the multiple classroom speaker virtual images and the classroom listeners virtual images in the virtual classroom, thereby achieving the technical effect of enhancing the recognition of audio data and improving user experience when multiple classroom speakers speak at the same time in the virtual space, and further solving the technical problems of mixed sound sources and poor user experience when multiple speakers speak at the same time in the virtual space.

[0179] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0180] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0181] Example 2

[0182] According to an embodiment of the present invention, there is also provided an apparatus embodiment for implementing the above method for identifying the sound source of audio data. Fig.11 is a schematic diagram of the structure of a device for identifying the sound source of audio data according to an embodiment of the present invention. Fig.11 As shown, the device includes: a receiving module 1101, a determining module 1102 and an identifying module 1103, wherein:

[0183] The receiving module 1101 is used to receive audio data and first position data from a plurality of first virtual images in a preset virtual space, wherein the current identities of the plurality of first virtual images are speaker identities;

[0184] A determination module 1102 is used to determine a spatial position relationship using the first position data and the second position data, wherein the second position data is position data of the second virtual image, the current identity of the second virtual image is the identity of the listener, and the spatial position relationship is used to determine a relative position relationship between the second virtual image and each of the plurality of first virtual images;

[0185] The identification module 1103 is used to identify the sound source of the audio data based on the spatial position relationship.

[0186] Optionally, the above-mentioned determination module 1102 is also used to: perform a difference operation on the first coordinate position of each first virtual image in the preset virtual space and the second coordinate position of the second virtual image in the preset virtual space to obtain a spatial position relationship, wherein the spatial position relationship includes: the relative orientation relationship between the second virtual image and each first virtual image in the multiple first virtual images, and the relative distance relationship between the second virtual image and each first virtual image in the multiple first virtual images.

[0187] Optionally, the above-mentioned identification module 1103 is also used to: determine the weight coefficient corresponding to each channel component of the audio data on the target playback device by utilizing the spatial position relationship, wherein the target playback device is the playback device associated with the second virtual image and the target playback device supports multi-channel spatial audio playback; synthesize multi-channel spatial audio based on the weight coefficient corresponding to each channel component on the target playback device; and identify the sound source of the audio data through the multi-channel spatial audio.

[0188] Optionally, the above-mentioned identification module 1103 is also used to: determine the target length and the target angle by using the spatial position relationship, wherein the target length is the length of the target vector formed by the listening position corresponding to each channel component on the target playback device and the first coordinate position of each first virtual image in the multiple first virtual images, and the target angle is the angle between the direction corresponding to the target vector and the direction corresponding to each channel component on the target playback device; calculate the first attenuation coefficient corresponding to each channel component on the target playback device based on the target length, and calculate the second attenuation coefficient corresponding to each channel component on the target playback device based on the target angle, wherein the first attenuation coefficient is the distance attenuation coefficient, and the second attenuation coefficient is the directional attenuation coefficient; use the first attenuation coefficient and the second attenuation coefficient to calculate the weight coefficient corresponding to each channel component on the target playback device.

[0189] Optionally, the identification module 1103 is further used to: perform fast Fourier transform on the audio data corresponding to each channel component on the target playback device to obtain a transformation result; based on the transformation result, obtain a second attenuation coefficient corresponding to each channel component on the target playback device using a target angle calculation.

[0190] Optionally, the identification module 1103 is further configured to perform a weighted sum operation based on a weight coefficient corresponding to each channel component on a target playback device and audio data corresponding to each channel component to obtain multi-channel spatial audio.

[0191] Optionally, Fig.12 FIG. 1 is a schematic diagram of a structure of an optional device for identifying the sound source of audio data according to an embodiment of the present invention. Fig.12 As shown, the device includes Fig.11 In addition to all the modules shown, it also includes: an insertion module 1104, which is used to obtain the phase difference between different channel components of the audio data on the target playback device, wherein the phase difference is used to assist in determining the orientation of each first virtual image in multiple first virtual images; and insert the phase difference in the time domain corresponding to each channel component on the target playback device.

[0192] Optionally, Fig.13 FIG. 1 is a schematic diagram of the structure of another optional device for identifying the sound source of audio data according to an embodiment of the present invention. Fig.13As shown, the device includes Fig.12 In addition to all the modules shown, it also includes: a display module 1105, which is used to: respond to a first touch operation applied to the graphical user interface to add multiple first virtual images to a preset virtual space; respond to a second touch operation applied to the graphical user interface to enable a virtual speaker control corresponding to the second virtual image; and perform spatial audio playback of the audio data of the multiple first virtual images through a target playback device corresponding to the virtual speaker control to identify the sound source of the audio data.

[0193] It should be noted that the receiving module 1101, the determining module 1102 and the identifying module 1103 correspond to steps S41 to S43 in Example 1, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0194] In an embodiment of the present invention, audio data and first position data are received from a plurality of first virtual images in a preset virtual space, wherein the current identities of the plurality of first virtual images are speaker identities, and a spatial position relationship is determined by utilizing the first position data and the second position data, wherein the second position data is position data of a second virtual image, and the current identity of the second virtual image is a listener identity, and the spatial position relationship is used to determine a relative position relationship between the second virtual image and each of the plurality of first virtual images, and further identify a sound source of the audio data based on the spatial position relationship.

[0195] It is easy to notice that, through the embodiments of the present invention, when there are multiple current speakers in the preset virtual space, the sound sources of the audio data of the multiple current speakers are identified through the relative position relationship between the multiple current speakers and the listeners, and the binaural effect of humans can be simulated in the preset virtual space to determine the positions of the speakers, thereby achieving the purpose of identifying the sound sources of the audio data of the multiple speaker virtual images according to the relative position relationship between the multiple speaker virtual images and the listener virtual images in the preset virtual space, thereby achieving the technical effect of enhancing the recognition of audio data and improving user experience when multiple speakers are speaking at the same time in the virtual space, and further solving the technical problem of mixed sound sources and poor user experience when multiple speakers are speaking at the same time in the virtual space.

[0196] According to an embodiment of the present invention, there is also provided an embodiment of a device for implementing the above-mentioned method for identifying the sound source of audio data. Fig.14 FIG. 4 is a schematic diagram of the structure of another device for identifying the sound source of audio data according to an embodiment of the present invention. Fig.14As shown, the device includes: a receiving module 1401, a determining module 1402 and an identifying module 1403, wherein:

[0197] The receiving module 1401 is used to receive audio data and first position data from a plurality of first virtual images in a virtual conference room, wherein the current identities of the plurality of first virtual images are conference speakers;

[0198] A determination module 1402 is used to determine a spatial position relationship using the first position data and the second position data, wherein the second position data is position data of the second virtual image, the current identity of the second virtual image is the identity of the conference listener, and the spatial position relationship is used to determine a relative position relationship between the second virtual image and each of the plurality of first virtual images;

[0199] The identification module 1403 is used to identify the sound source of the audio data in the virtual conference room based on the spatial position relationship.

[0200] Optionally, Fig.15 FIG. 1 is a schematic diagram of the structure of another optional device for identifying the sound source of audio data according to an embodiment of the present invention. Fig.15 As shown, the device includes Fig.14 In addition to all the modules shown, it also includes: a display module 1404, which is used to respond to a first touch operation applied to the graphical user interface to add multiple first virtual images to the virtual conference room; respond to a second touch operation applied to the graphical user interface to enable a virtual speaker control corresponding to the second virtual image; and perform spatial audio playback of the audio data of the multiple first virtual images through a target playback device corresponding to the virtual speaker control to identify the sound source of the audio data in the virtual conference room.

[0201] It should be noted that the receiving module 1401, the determining module 1402 and the identifying module 1403 correspond to steps S91 to S93 in Embodiment 1, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Embodiment 1.

[0202] In an embodiment of the present invention, audio data and first position data are received from multiple first virtual images in a virtual conference room, wherein the current identities of the multiple first virtual images are conference speakers, and a spatial position relationship is determined by utilizing the first position data and the second position data, wherein the second position data is the position data of the second virtual image, and the current identity of the second virtual image is the conference listener. The spatial position relationship is used to determine the relative position relationship between the second virtual image and each of the multiple first virtual images, and further identify the sound source of the audio data in the virtual conference room based on the spatial position relationship.

[0203] It is easy to notice that, through the embodiments of the present invention, when there are multiple current conference speakers in the virtual conference room, the sound sources of the audio data of the multiple current conference speakers in the virtual conference room are identified through the relative position relationship between the multiple current conference speakers and the conference listeners, and the positions of the conference speakers can be determined by simulating the binaural effect of humans in the virtual conference room, thereby achieving the purpose of identifying the sound sources of the audio data of the multiple conference speaker virtual images in the virtual conference room according to the relative position relationship between the multiple conference speaker virtual images and the conference listeners virtual images in the virtual conference room, thereby achieving the technical effect of enhancing the recognition of audio data and improving user experience when multiple conference speakers speak at the same time in the virtual space, and further solving the technical problems of mixed sound sources and poor user experience when multiple speakers speak at the same time in the virtual space.

[0204] According to an embodiment of the present invention, there is also provided an apparatus embodiment for implementing the above-mentioned method for identifying the sound source of audio data. Fig.16 FIG. 4 is a schematic diagram of the structure of another device for identifying the sound source of audio data according to an embodiment of the present invention. Fig.16 As shown, the device includes: a receiving module 1601, a determining module 1602 and an identifying module 1603, wherein:

[0205] The receiving module 1601 is used to receive audio data and first position data from a plurality of first virtual images in a virtual classroom, wherein the current identities of the plurality of first virtual images are classroom speakers;

[0206] A determination module 1602 is used to determine a spatial position relationship using the first position data and the second position data, wherein the second position data is position data of the second virtual image, the current identity of the second virtual image is the identity of a class listener, and the spatial position relationship is used to determine a relative position relationship between the second virtual image and each of the plurality of first virtual images;

[0207] The identification module 1603 is used to identify the sound source of the audio data in the virtual classroom based on the spatial position relationship.

[0208] It should be noted that the receiving module 1601, the determining module 1602 and the identifying module 1603 correspond to steps S1001 to S1003 in Example 1, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0209] Receive audio data and first position data from multiple first virtual images in a virtual classroom, wherein the current identities of the multiple first virtual images are classroom speakers, determine a spatial position relationship by using the first position data and the second position data, wherein the second position data is the position data of the second virtual image, and the current identity of the second virtual image is the classroom listener, and the spatial position relationship is used to determine the relative position relationship between the second virtual image and each of the multiple first virtual images, and further identify the sound source of the audio data in the virtual classroom based on the spatial position relationship.

[0210] It is easy to notice that, through the embodiments of the present invention, when there are multiple current classroom speakers in the virtual classroom, the sound sources of the audio data of the multiple current classroom speakers in the virtual classroom are identified through the relative position relationship between the multiple current classroom speakers and the classroom listeners, and the binaural effect of humans can be simulated in the virtual classroom to determine the positions of the classroom speakers, thereby achieving the purpose of identifying the sound sources of the audio data of the multiple classroom speaker virtual images in the virtual classroom according to the relative position relationship between the multiple classroom speaker virtual images and the classroom listeners virtual images in the virtual classroom, thereby achieving the technical effect of enhancing the recognition of audio data and improving user experience when multiple classroom speakers speak at the same time in the virtual space, and further solving the technical problems of mixed sound sources and poor user experience when multiple speakers speak at the same time in the virtual space.

[0211] It should be noted that the preferred implementation of this embodiment can refer to the relevant description in Example 1, which will not be repeated here.

[0212] Example 3

[0213] According to an embodiment of the present invention, an embodiment of an electronic device is also provided, and the electronic device can be any computing device in a computing device group. The electronic device includes: a processor and a memory, wherein:

[0214] A memory is connected to the processor and is used to provide the processor with instructions for processing the following processing steps: receiving audio data and first position data from multiple first virtual images in a preset virtual space, wherein the current identities of the multiple first virtual images are speaker identities; determining a spatial position relationship using the first position data and the second position data, wherein the second position data is position data of a second virtual image, the current identity of the second virtual image is a listener identity, and the spatial position relationship is used to determine a relative position relationship between the second virtual image and each of the multiple first virtual images; and identifying a sound source of the audio data based on the spatial position relationship.

[0215] In an embodiment of the present invention, audio data and first position data are received from a plurality of first virtual images in a preset virtual space, wherein the current identities of the plurality of first virtual images are speaker identities, and a spatial position relationship is determined by utilizing the first position data and the second position data, wherein the second position data is position data of a second virtual image, and the current identity of the second virtual image is a listener identity, and the spatial position relationship is used to determine a relative position relationship between the second virtual image and each of the plurality of first virtual images, and further identify a sound source of the audio data based on the spatial position relationship.

[0216] It is easy to notice that, through the embodiments of the present invention, when there are multiple current speakers in the preset virtual space, the sound sources of the audio data of the multiple current speakers are identified through the relative position relationship between the multiple current speakers and the listeners, and the binaural effect of humans can be simulated in the preset virtual space to determine the positions of the speakers, thereby achieving the purpose of identifying the sound sources of the audio data of the multiple speaker virtual images according to the relative position relationship between the multiple speaker virtual images and the listener virtual images in the preset virtual space, thereby achieving the technical effect of enhancing the recognition of audio data and improving user experience when multiple speakers are speaking at the same time in the virtual space, and further solving the technical problem of mixed sound sources and poor user experience when multiple speakers are speaking at the same time in the virtual space.

[0217] It should be noted that the preferred implementation of this embodiment can refer to the relevant description in Example 1, which will not be repeated here.

[0218] Example 4

[0219] The embodiment of the present invention can provide a computer terminal, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0220] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0221] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the method for identifying the sound source of audio data: receiving audio data and first position data from multiple first virtual images in a preset virtual space, wherein the current identity of the multiple first virtual images is the speaker identity; determining the spatial position relationship using the first position data and the second position data, wherein the second position data is the position data of the second virtual image, the current identity of the second virtual image is the listener identity, and the spatial position relationship is used to determine the relative position relationship between the second virtual image and each of the multiple first virtual images; identifying the sound source of the audio data based on the spatial position relationship.

[0222] Optionally, Fig.17 is a structural block diagram of another computer terminal according to an embodiment of the present invention. Fig.17 As shown, the computer terminal may include: one or more (only one is shown in the figure) processors 122 , a memory 124 , and a peripheral interface 126 .

[0223] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for identifying the sound source of audio data in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method for identifying the sound source of audio data. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0224] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receiving audio data and first position data from multiple first virtual images in a preset virtual space, wherein the current identity of the multiple first virtual images is the speaker identity; using the first position data and the second position data to determine the spatial position relationship, wherein the second position data is the position data of the second virtual image, the current identity of the second virtual image is the listener identity, and the spatial position relationship is used to determine the relative position relationship between the second virtual image and each of the multiple first virtual images; identifying the sound source of the audio data based on the spatial position relationship.

[0225] Optionally, the processor may also execute program code of the following steps: performing a difference operation between the first coordinate position of each first virtual image in a preset virtual space and the second coordinate position of the second virtual image in the preset virtual space to obtain a spatial position relationship, wherein the spatial position relationship includes: a relative orientation relationship between the second virtual image and each first virtual image in a plurality of first virtual images, and a relative distance relationship between the second virtual image and each first virtual image in a plurality of first virtual images.

[0226] Optionally, the processor may also execute program code of the following steps: determining a weight coefficient corresponding to each channel component of the audio data on a target playback device by using a spatial position relationship, wherein the target playback device is a playback device associated with a second virtual image and the target playback device supports multi-channel spatial audio playback; synthesizing multi-channel spatial audio based on the weight coefficient corresponding to each channel component on the target playback device; and identifying the sound source of the audio data through the multi-channel spatial audio.

[0227] Optionally, the processor may also execute program code of the following steps: determining a target length and a target angle using a spatial position relationship, wherein the target length is the length of a target vector formed by a listening position corresponding to each channel component on a target playback device and a first coordinate position of each first virtual image in a plurality of first virtual images, and the target angle is the angle between a direction corresponding to the target vector and a direction corresponding to each channel component on the target playback device; obtaining a first attenuation coefficient corresponding to each channel component on the target playback device based on the target length, and obtaining a second attenuation coefficient corresponding to each channel component on the target playback device based on the target angle, wherein the first attenuation coefficient is a distance attenuation coefficient and the second attenuation coefficient is a directional attenuation coefficient; and obtaining a weight coefficient corresponding to each channel component on the target playback device using the first attenuation coefficient and the second attenuation coefficient.

[0228] Optionally, the processor may also execute the following program code: performing a fast Fourier transform on the audio data corresponding to each channel component on the target playback device to obtain a transformation result; based on the transformation result, using the target angle calculation to obtain a second attenuation coefficient corresponding to each channel component on the target playback device.

[0229] Optionally, the processor may further execute program code of the following steps: performing a weighted sum operation based on a weight coefficient corresponding to each channel component on a target playback device and audio data corresponding to each channel component to obtain multi-channel spatial audio.

[0230] Optionally, the processor may also execute program code of the following steps: obtaining a phase difference between different channel components of audio data on a target playback device, wherein the phase difference is used to assist in determining the orientation of each first virtual image among a plurality of first virtual images; and inserting a phase difference in the time domain corresponding to each channel component on the target playback device.

[0231] Optionally, the processor may also execute program codes of the following steps: in response to a first touch operation applied to a graphical user interface, adding multiple first virtual images to a preset virtual space; in response to a second touch operation applied to the graphical user interface, enabling a virtual speaker control corresponding to the second virtual image; and performing spatial audio playback of the audio data of the multiple first virtual images through a target playback device corresponding to the virtual speaker control to identify the sound source of the audio data.

[0232] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receiving audio data and first position data from multiple first virtual images in the virtual conference room, wherein the current identities of the multiple first virtual images are conference speakers; determining the spatial position relationship using the first position data and the second position data, wherein the second position data is the position data of the second virtual image, and the current identity of the second virtual image is the conference listener, and the spatial position relationship is used to determine the relative position relationship between the second virtual image and each of the multiple first virtual images; identifying the sound source of the audio data in the virtual conference room based on the spatial position relationship.

[0233] Optionally, the processor may also execute program code of the following steps: in response to a first touch operation applied to the graphical user interface, adding multiple first virtual images to the virtual conference room; in response to a second touch operation applied to the graphical user interface, enabling a virtual speaker control corresponding to the second virtual image; and performing spatial audio playback of the audio data of the multiple first virtual images through a target playback device corresponding to the virtual speaker control to identify the sound source of the audio data in the virtual conference room.

[0234] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receiving audio data and first position data from multiple first virtual images in the virtual classroom, wherein the current identities of the multiple first virtual images are classroom speakers; determining the spatial position relationship using the first position data and the second position data, wherein the second position data is the position data of the second virtual image, the current identity of the second virtual image is the classroom listener, and the spatial position relationship is used to determine the relative position relationship between the second virtual image and each of the multiple first virtual images; identifying the sound source of the audio data in the virtual classroom based on the spatial position relationship.

[0235] In an embodiment of the present invention, audio data and first position data are received from a plurality of first virtual images in a preset virtual space, wherein the current identities of the plurality of first virtual images are speaker identities, and a spatial position relationship is determined by utilizing the first position data and the second position data, wherein the second position data is position data of a second virtual image, and the current identity of the second virtual image is a listener identity, and the spatial position relationship is used to determine a relative position relationship between the second virtual image and each of the plurality of first virtual images, and further identify a sound source of the audio data based on the spatial position relationship.

[0236] It is easy to notice that, through the embodiments of the present invention, when there are multiple current speakers in the preset virtual space, the sound sources of the audio data of the multiple current speakers are identified through the relative position relationship between the multiple current speakers and the listeners, and the binaural effect of humans can be simulated in the preset virtual space to determine the positions of the speakers, thereby achieving the purpose of identifying the sound sources of the audio data of the multiple speaker virtual images according to the relative position relationship between the multiple speaker virtual images and the listener virtual images in the preset virtual space, thereby achieving the technical effect of enhancing the recognition of audio data and improving user experience when multiple speakers are speaking at the same time in the virtual space, and further solving the technical problem of mixed sound sources and poor user experience when multiple speakers are speaking at the same time in the virtual space.

[0237] It can be understood by those skilled in the art that Fig.17 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig.17 It does not limit the structure of the above electronic device. For example, the computer terminal may also include Fig.17 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.17 Different configurations are shown.

[0238] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0239] According to an embodiment of the present invention, an embodiment of a computer-readable storage medium is also provided. Optionally, in this embodiment, the computer-readable storage medium can be used to store program codes executed by the method for identifying the sound source of audio data provided in the above embodiment 1.

[0240] Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the mobile terminals in a mobile terminal group.

[0241] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: receiving audio data and first position data from multiple first virtual images in a preset virtual space, wherein the current identities of the multiple first virtual images are speaker identities; determining a spatial position relationship using the first position data and the second position data, wherein the second position data is the position data of the second virtual image, the current identity of the second virtual image is the listener identity, and the spatial position relationship is used to determine the relative position relationship between the second virtual image and each of the multiple first virtual images; identifying the sound source of the audio data based on the spatial position relationship.

[0242] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: receiving audio data and first position data from multiple first virtual images in a preset virtual space, wherein the current identities of the multiple first virtual images are speaker identities; determining a spatial position relationship using the first position data and the second position data, wherein the second position data is the position data of the second virtual image, the current identity of the second virtual image is the listener identity, and the spatial position relationship is used to determine the relative position relationship between the second virtual image and each of the multiple first virtual images; identifying the sound source of the audio data based on the spatial position relationship.

[0243] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: performing a difference operation on a first coordinate position of each first virtual image in a preset virtual space and a second coordinate position of a second virtual image in a preset virtual space to obtain a spatial position relationship, wherein the spatial position relationship includes: a relative orientation relationship between the second virtual image and each first virtual image in the multiple first virtual images, and a relative distance relationship between the second virtual image and each first virtual image in the multiple first virtual images.

[0244] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: determining a weight coefficient corresponding to each channel component of the audio data on a target playback device using a spatial position relationship, wherein the target playback device is a playback device associated with the second virtual image and the target playback device supports multi-channel spatial audio playback; synthesizing multi-channel spatial audio based on the weight coefficient corresponding to each channel component on the target playback device; and identifying the sound source of the audio data through the multi-channel spatial audio.

[0245] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: determining a target length and a target angle using a spatial position relationship, wherein the target length is the length of a target vector formed by a listening position corresponding to each channel component on a target playback device and a first coordinate position of each first virtual image in a plurality of first virtual images, and the target angle is the angle between a direction corresponding to the target vector and a direction corresponding to each channel component on the target playback device; calculating a first attenuation coefficient corresponding to each channel component on the target playback device based on the target length, and calculating a second attenuation coefficient corresponding to each channel component on the target playback device based on the target angle, wherein the first attenuation coefficient is a distance attenuation coefficient and the second attenuation coefficient is a directional attenuation coefficient; and calculating a weight coefficient corresponding to each channel component on the target playback device using the first attenuation coefficient and the second attenuation coefficient.

[0246] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: performing a fast Fourier transform on the audio data corresponding to each channel component on the target playback device to obtain a transformation result; based on the transformation result, using the target angle calculation to obtain a second attenuation coefficient corresponding to each channel component on the target playback device.

[0247] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: performing a weighted sum operation based on a weight coefficient corresponding to each channel component on a target playback device and audio data corresponding to each channel component to obtain multi-channel spatial audio.

[0248] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining a phase difference between different channel components of audio data on a target playback device, wherein the phase difference is used to assist in determining the orientation of each first virtual image among multiple first virtual images; and inserting a phase difference in the time domain corresponding to each channel component on the target playback device.

[0249] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for performing the following steps: in response to a first touch operation applied to a graphical user interface, adding multiple first virtual images to a preset virtual space; in response to a second touch operation applied to the graphical user interface, enabling virtual speaker controls corresponding to the second virtual images; and performing spatial audio playback of audio data of the multiple first virtual images through a target playback device corresponding to the virtual speaker controls to identify the sound source of the audio data.

[0250] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: receiving audio data and first position data from multiple first virtual images in a virtual conference room, wherein the current identities of the multiple first virtual images are conference speakers; determining a spatial position relationship using the first position data and the second position data, wherein the second position data is the position data of the second virtual image, and the current identity of the second virtual image is a conference listener, and the spatial position relationship is used to determine the relative position relationship between the second virtual image and each of the multiple first virtual images; identifying the sound source of the audio data in the virtual conference room based on the spatial position relationship.

[0251] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for performing the following steps: in response to a first touch operation applied to a graphical user interface, adding multiple first virtual images to a virtual conference room; in response to a second touch operation applied to the graphical user interface, enabling virtual speaker controls corresponding to the second virtual images; and performing spatial audio playback of audio data of the multiple first virtual images through a target playback device corresponding to the virtual speaker controls to identify the sound source of the audio data in the virtual conference room.

[0252] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: receiving audio data and first position data from multiple first virtual images in a virtual classroom, wherein the current identities of the multiple first virtual images are classroom speakers; determining a spatial position relationship using the first position data and the second position data, wherein the second position data is the position data of the second virtual image, and the current identity of the second virtual image is the classroom listener, and the spatial position relationship is used to determine the relative position relationship between the second virtual image and each of the multiple first virtual images; identifying the sound source of the audio data in the virtual classroom based on the spatial position relationship.

[0253] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0254] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0255] In the several embodiments provided by the present invention, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0256] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0257] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0258] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0259] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for identifying the sound source of audio data, characterized in that: include: Receiving audio data and first position data from a plurality of first virtual images in a preset virtual space, wherein current identities of the plurality of first virtual images are speaker identities; Determine a spatial position relationship using the first position data and the second position data, wherein the second position data is position data of a second virtual image, the current identity of the second virtual image is the identity of the listener, and the spatial position relationship is used to determine a relative position relationship between the second virtual image and each of the plurality of first virtual images; Determine a weight coefficient corresponding to each channel component of the audio data on a target playback device by using the spatial position relationship, wherein the weight coefficient is calculated by a first attenuation coefficient corresponding to a target length and a second attenuation coefficient corresponding to a target angle, and the target length and the target angle are determined by the spatial position relationship; The sound source of the audio data is identified by synthesizing the multi-channel spatial audio based on the weight coefficients.

2. The method according to claim 1, characterized in that Determining the spatial position relationship using the first position data and the second position data includes: The spatial position relationship is obtained by performing a difference operation on the first coordinate position of each first virtual image among the multiple first virtual images in the preset virtual space and the second coordinate position of the second virtual image in the preset virtual space, wherein the spatial position relationship includes: the relative orientation relationship between the second virtual image and each first virtual image among the multiple first virtual images, and the relative distance relationship between the second virtual image and each first virtual image among the multiple first virtual images.

3. The method according to claim 1, characterized in that The target playback device is a playback device associated with the second avatar and the target playback device supports multi-channel spatial audio playback; Identifying the sound source of the audio data by synthesizing the multi-channel spatial audio based on the weight coefficients includes: synthesizing the multi-channel spatial audio based on the weight coefficient corresponding to each channel component on the target playback device; The sound source is identified through the multi-channel spatial audio.

4. The method according to claim 3, characterized in that Determining the weight coefficient corresponding to each channel component of the audio data on the target playback device by using the spatial position relationship includes: Determine a target length and a target angle using the spatial position relationship, wherein the target length is the length of a target vector formed by a listening position corresponding to each channel component on the target playback device and a first coordinate position of each first virtual image in the plurality of first virtual images, and the target angle is the angle between a direction corresponding to the target vector and a direction corresponding to each channel component on the target playback device; A first attenuation coefficient corresponding to each channel component on the target playback device is calculated based on the target length, and a second attenuation coefficient corresponding to each channel component on the target playback device is calculated based on the target angle, wherein the first attenuation coefficient is a distance attenuation coefficient, and the second attenuation coefficient is a directivity attenuation coefficient; The first attenuation coefficient and the second attenuation coefficient are used to calculate a weight coefficient corresponding to each channel component on the target playback device.

5. The method according to claim 4, characterized in that Calculating the second attenuation coefficient corresponding to each channel component on the target playback device based on the target angle includes: Performing a fast Fourier transform on the audio data corresponding to each channel component on the target playback device to obtain a transformation result; Based on the transformation result, the second attenuation coefficient corresponding to each channel component on the target playback device is calculated using the target angle.

6. The method according to claim 4, characterized in that Synthesizing the multi-channel spatial audio based on the weight coefficient corresponding to each channel component on the target playback device includes: The multi-channel spatial audio is obtained by performing a weighted sum operation based on the weight coefficient corresponding to each channel component on the target playback device and the audio data corresponding to each channel component.

7. The method according to claim 3, characterized in that The method further comprises: Acquire a phase difference between different channel components of the audio data on a target playback device, wherein the phase difference is used to assist in determining the position of each of the plurality of first virtual images; The phase difference is inserted in the time domain corresponding to each channel component on the target playback device.

8. The method according to claim 1, characterized in that A graphical user interface is provided through a terminal device, wherein the content displayed by the graphical user interface at least partially includes a virtual space scene, and the method further includes: In response to a first touch operation on the graphical user interface, adding the plurality of first virtual images to the preset virtual space; In response to a second touch operation applied to the graphical user interface, enabling a virtual speaker control corresponding to the second virtual image; The audio data of the multiple first virtual images are spatially played back through a target playback device corresponding to the virtual speaker control to identify the sound source of the audio data.

9. A method for identifying the sound source of audio data, characterized in that: include: Receiving audio data and first position data from a plurality of first virtual images in a virtual conference room, wherein current identities of the plurality of first virtual images are conference speaker identities; Determine a spatial position relationship using the first position data and the second position data, wherein the second position data is position data of a second virtual image, the current identity of the second virtual image is the identity of a conference listener, and the spatial position relationship is used to determine a relative position relationship between the second virtual image and each of the plurality of first virtual images; Determine a weight coefficient corresponding to each channel component of the audio data on a target playback device by using the spatial position relationship, wherein the weight coefficient is calculated by a first attenuation coefficient corresponding to a target length and a second attenuation coefficient corresponding to a target angle, and the target length and the target angle are determined by the spatial position relationship; The sound source of the audio data in the virtual conference room is identified by synthesizing the multi-channel spatial audio based on the weight coefficients.

10. The method according to claim 9, characterized in that A graphical user interface is provided through a terminal device, wherein the content displayed by the graphical user interface at least partially includes a virtual conference room scene, and the method further includes: In response to a first touch operation on the graphical user interface, adding the plurality of first virtual images to the virtual conference room; In response to a second touch operation applied to the graphical user interface, enabling a virtual speaker control corresponding to the second virtual image; The audio data of the multiple first virtual images are spatially played back through a target playback device corresponding to the virtual speaker control to identify the sound source of the audio data in the virtual conference room.

11. A method for identifying the sound source of audio data, characterized in that: include: Receiving audio data and first position data from a plurality of first virtual images in a virtual classroom, wherein current identities of the plurality of first virtual images are classroom speakers; Determine a spatial position relationship using the first position data and the second position data, wherein the second position data is position data of a second virtual image, the current identity of the second virtual image is a classroom listener, and the spatial position relationship is used to determine a relative position relationship between the second virtual image and each of the plurality of first virtual images; Determine a weight coefficient corresponding to each channel component of the audio data on a target playback device by using the spatial position relationship, wherein the weight coefficient is calculated by a first attenuation coefficient corresponding to a target length and a second attenuation coefficient corresponding to a target angle, and the target length and the target angle are determined by the spatial position relationship; The sound source of the audio data in the virtual classroom is identified by synthesizing the multi-channel spatial audio based on the weight coefficients.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for identifying the sound source of audio data according to any one of claims 1 to 11.

13. An electronic device, characterized in that: include: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Receiving audio data and first position data from a plurality of first virtual images in a preset virtual space, wherein current identities of the plurality of first virtual images are speaker identities; Determine a spatial position relationship using the first position data and the second position data, wherein the second position data is position data of a second virtual image, the current identity of the second virtual image is the identity of the listener, and the spatial position relationship is used to determine a relative position relationship between the second virtual image and each of the plurality of first virtual images; Determine a weight coefficient corresponding to each channel component of the audio data on a target playback device by using the spatial position relationship, wherein the weight coefficient is calculated by a first attenuation coefficient corresponding to a target length and a second attenuation coefficient corresponding to a target angle, and the target length and the target angle are determined by the spatial position relationship; The sound source of the audio data is identified by synthesizing the multi-channel spatial audio based on the weight coefficients.

Images