Multi-person instant messaging method, system, medium and device

By using SFU and MCU servers in multi-person instant communication systems to filter and push the audio stream with the largest volume, the problems of inaccurate sound audible and bandwidth pressure in multi-person communication scenarios are solved, and the communication quality is improved.

CN115883501BActive Publication Date: 2024-11-12武汉斗鱼鱼乐网络科技有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211582052.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-12-08
Publication Date
2024-11-12
Estimated Expiration
2042-12-08

AI Technical Summary

Technical Problem

In multi-person instant communication scenarios, when there are many users, the prior art is difficult to ensure communication quality, resulting in unrecognizable user voice and increasing bandwidth pressure, affecting the communication effect.

Method used

The user audio stream is filtered by a selective forwarding unit (SFU) server to obtain the reference audio stream, and the multi-point control unit (MCU) server is used to filter out the target audio stream with the largest volume, push it to the SFU server, and then SFU The server pushes it to other users, reducing bandwidth pressure.

Benefits of technology

By filtering and pushing the audio stream with the largest volume, the number of audio streams that each user needs to push and pull is reduced, bandwidth pressure is reduced, and the instant communication quality is ensured, especially in massive user scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115883501B_ABST
    Figure CN115883501B_ABST
Patent Text Reader

Abstract

The present invention provides a multi-person instant communication method, system, medium and device. The method comprises: using an SFU server to screen audio streams of N users to obtain multiple reference audio streams; using an MCU server to pull and screen the multiple reference audio streams to obtain M target audio streams with the largest current volume, and pushing the M target audio streams to the SFU server; using the SFU server to push the M target audio streams to the N users; in this way, based on the SFU+MCU architecture, the MCU can select only M target audio streams from the N audios and push them to the SFU server. In an instant communication scenario, the number of users with the loudest voices is about 3 to 5, so it is equivalent to that each user only needs to push one audio to the SFU server, and at the same time pull 3 to 5 target audio streams from the SFU, which greatly reduces the bandwidth pressure and ensures the communication quality even in a communication scenario with a large number of users.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of audio and video communication technology, and in particular to a multi-person instant communication method, system, medium and device. Background Art

[0002] Establishing instant messaging relationships is an important means in social software to ensure effective information exchange between users. After a user establishes an instant messaging relationship with other users on the social software, he or she can conduct online chatting, information sharing and other activities with the users who have established the instant messaging relationship.

[0003] At present, various instant messaging tools mainly forward WebRTC client audio and video data streams through the conference solution of the Selective Forwarding Unit (SFU) to achieve multi-terminal online voice interaction. However, if there are many people communicating, the user's voice will not be heard clearly, so this solution needs to strictly limit the number of users. If the number of users is not limited, in the scenario of massive user communication, the user bandwidth pressure will increase, and there will be sound jamming and other phenomena, which will seriously affect the communication quality.

[0004] Based on this, how to improve the quality of multi-person instant messaging is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the problems existing in the prior art, the embodiments of the present invention provide a multi-person instant messaging method, system, medium and device to solve or partially solve the technical problem in the prior art that the quality of multi-person instant messaging cannot be improved without increasing the user bandwidth pressure.

[0006] A first aspect of the present invention provides a multi-person instant messaging method, the method comprising:

[0007] Using a selective forwarding unit SFU server to screen the audio streams of N users to obtain multiple reference audio streams;

[0008] Using a multipoint control unit MCU server to pull and filter the multiple reference audio streams, obtain M target audio streams with the largest current volumes, and push the M target audio streams with the largest current volumes to the SFU server;

[0009] The SFU server is used to push the M target audio streams with the highest current volumes to the N different users respectively.

[0010] In the above scheme, the selective forwarding unit SFU server is used to screen the audio streams of N users to obtain multiple reference audio streams, including:

[0011] Obtain the audio stream of each user using the SFU server, and determine the volume information of each audio stream;

[0012] Determine the audio stream in a muted state according to the volume information of each audio stream;

[0013] Audio streams in a silent state are filtered out from the audio streams of N users to obtain the multiple reference audio streams.

[0014] In the above scheme, the multipoint control unit MCU server is used to pull and screen the multiple reference audio streams to obtain M target audio streams with the largest current volume, including:

[0015] Pulling the multiple reference audio streams from the SFU server;

[0016] Acquire volume information of the multiple reference audio streams based on extended header information of each reference audio stream;

[0017] sorting the plurality of reference audio streams based on volume information of each reference audio stream;

[0018] Among the sorted multiple reference audio streams, M audio streams with the highest current volume are selected.

[0019] In the above scheme, the using of the SFU server to push the M target audio streams with the largest current volumes to the N users includes:

[0020] Parsing the RTP extension headers of the M target audio streams with the highest current volumes to obtain a stream ID of each target audio stream;

[0021] For the current user, if it is determined that the user ID of the current user is consistent with the stream ID of the current target audio stream, the remaining M-1 target audio streams except the current target audio stream are pushed to the current user; the current user is any one of the N different users.

[0022] In the above solution, the SFU server is used to push the M target audio streams with the largest current volumes to the N different users respectively, including:

[0023] The M target audio streams with the highest current volumes are pushed to the N different users in sequence; and a push interval between two adjacent target audio streams is less than or equal to a preset time threshold.

[0024] In the above solution, obtaining the volume information of each audio stream includes:

[0025] For any audio stream, decode the audio stream to obtain each frame of speech signal;

[0026] Determine the speech sampling point based on the sampling frequency and the frame length of each frame of the speech signal;

[0027] Determine a first volume according to a pulse modulation coding PCM value of each of the speech sampling points;

[0028] The volume information V′ of the audio stream is determined based on the formula V′=10*log10V; wherein V is the first volume.

[0029] In the above solution, determining the first volume according to the PCM value of each speech sampling point includes:

[0030] According to the formula Determine the first volume V; wherein the V S is the PCM value of the speech sampling point, and s is the sequence number of the speech sampling point.

[0031] A second aspect of the present invention provides a multi-person instant messaging system, the system comprising:

[0032] The SFU server is used to screen the audio streams of N users to obtain multiple reference audio streams;

[0033] The multipoint control unit MCU server is used to pull and filter the multiple reference audio streams, obtain M target audio streams with the largest current volumes, and push the M target audio streams with the largest current volumes to the SFU server;

[0034] The SFU server is further used to push the M target audio streams with the highest current volumes to the N different users respectively.

[0035] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0036] According to a third aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the methods described in the first aspect are implemented.

[0037] The present invention provides a multi-person instant communication method, system, medium and device, the method comprising: using a selective forwarding unit SFU server to screen the audio streams of N users to obtain multiple reference audio streams; using a multipoint control unit MCU server to pull and screen the multiple reference audio streams to obtain M target audio streams with the largest current volumes, and pushing the M target audio streams to the SFU server; using the SFU server to push the M target audio streams to the N different users respectively; in this way, based on the SFU+MCU architecture, the MCU can select only the M target audio streams with the largest current volumes from the N audios and push them to the SFU server, and then the SFU server pushes them to the users; generally speaking, in an instant communication scenario, the number of users with the largest voices is about 3 to 5, so it is equivalent to that each user only needs to push one channel of its own audio to the SFU server, and at the same time pull 3 to 5 target audio streams from the SFU, which greatly reduces the bandwidth pressure, and even in a communication scenario with a large number of users, the communication quality can be ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] By reading the detailed description of the preferred embodiment below, various other advantages and benefits will become clear to those of ordinary skill in the art. The accompanying drawings are only used for the purpose of illustrating the preferred embodiment and are not considered to be limitations of the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings.

[0039] In the attached picture:

[0040] Figure 1 A schematic diagram of the structure of a multi-person instant messaging system in the prior art is shown;

[0041] Figure 2 A schematic diagram of the structure of a multi-person instant messaging system according to an embodiment of the present invention is shown;

[0042] Figure 3 A schematic diagram of data interaction in a multi-person instant messaging system according to an embodiment of the present invention is shown;

[0043] Figure 4 A schematic diagram of a multi-person instant messaging method flow chart according to an embodiment of the present invention is shown;

[0044] Figure 5 A schematic diagram of the RTP extension header structure according to an embodiment of the present invention is shown;

[0045] Figure 6 A voice signal diagram is shown when two user voices frequently switch according to an embodiment of the present invention;

[0046] Figure 7 A schematic diagram of the structure of a computer device according to an embodiment of the present invention is shown;

[0047] Figure 8 A schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0048] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0049] In order to better understand the technical solution of this embodiment, the traditional voice chat logic is first introduced here. Figure 1 In the traditional voice chat architecture, assuming that there are N people having a voice chat, a single user needs to push one of his own audio streams to the stream server, and at the same time needs to obtain N-1 stream information of all other users in the current room through room server 1, and pull N-1 audio streams through stream server 2 to listen to the voices of other users. Generally, the bit rate of one audio stream is around 64kbps to 192kbps. When the number of users N is 100, about 6M to 20M bandwidth is required, which is unbearable for most users, especially mobile users. If it is necessary to support voice chats of thousands or even tens of thousands of people, this solution is even more unfeasible. Therefore, in traditional multi-person voice chats, the number of chatters is generally limited, such as not more than 9 people.

[0050] Based on this, in order to solve the above problems, this embodiment provides a multi-person instant messaging system, which mainly realizes multi-person instant messaging by building an SFU+MCU architecture. Figure 2 As shown, the system includes: an SFU server, an MCU server and a room server 1; wherein,

[0051] The Selective Forwarding Unit (SFU) server does not need to mix the audio and video. After receiving the audio and video stream shared by a terminal, it directly forwards the audio and video stream to other terminals in the room. The SFU server is actually an audio and video routing forwarder.

[0052] The Multipoint Conferencing Unit (MCU) server will mix the audio and video streams of all terminals in the same room, and finally generate a mixed audio and video stream and send it to each terminal, so that each terminal can see / hear the audio and video of other terminals. In fact, the MCU server can be understood as an audio and video mixer.

[0053] The SFU server and MCU server can be a single server or a cluster server. When the SFU server and MCU server are cluster servers, refer to Figure 3 The SFU server can be a cluster of many stream servers, and the MCU server can be a cluster of many mixing servers.

[0054] The SFU server is used to screen the audio streams of N users to obtain multiple reference audio streams; and send the multiple reference audio streams to the multipoint control unit MCU server;

[0055] The MCU server is used to screen multiple reference audio streams, obtain M target audio streams with the largest current volumes, and push the M target audio streams with the largest current volumes to the SFU server;

[0056] The SFU server is also used to push M target audio streams with the largest current volume to N different users, where the value range of M is 3-5.

[0057] Specifically, the data interaction process between the user, SFU server, MCU server and room server 1 can be referred to Figure 2 Assuming the number of users is N, the steps are as follows:

[0058] Step 1.1: User joins the room;

[0059] Step 2: The user pushes his stream data to the SFU server;

[0060] Step 2.1: The SFU server sends stream notification information to the room server;

[0061] Step 2.1.1: The room server initiates a mixed stream request to the MCU server based on the stream notification information;

[0062] Step 3: The MCU server obtains N stream data from the SFU server based on the mixed stream request, and selects M SFU single streams;

[0063] Step 3.1: The MCU server pushes M SFU single streams to the SFU server;

[0064] Step 3.1.1: The SFU server sends M stream notification information to the room server;

[0065] Step 3.1.1.1: The room server forwards M stream notification information to the user;

[0066] Step 4: The user obtains M stream data from the SFU server.

[0067] When a new user enters the room, the stream data processing process of the new user is the same as above. The difference is that the MCU server will update M stream data and send the new M stream data to the SFU server.

[0068] Among them, the specific implementation logic of the SFU server and the MCU server will be described in detail in subsequent embodiments, so it will not be repeated here.

[0069] It can be seen that in the SFU+MCU-based architecture, the MCU can select only M target audio streams with the current loudest volume from N audios and push them to the SFU server, which then pushes them to the user. Generally speaking, in instant messaging scenarios, the number of users with the loudest voices is about 3 to 5, so M is 3 to 5, which means that each user only needs to push one channel of his own audio to the SFU server and pull 3 to 5 target audios from the SFU at the same time, which greatly reduces the bandwidth pressure and ensures the communication quality even in massive user communication scenarios.

[0070] It is also worth noting that although the MCU server has a mixing function, the MCU server in this embodiment does not need to mix the multiple reference audio streams obtained, and the output target audio stream is a single stream, which will further reduce the pressure on the MCU server and improve the communication quality.

[0071] It should be noted that the multi-person instant communication method of this embodiment is applicable to multi-person voice calls, multi-person video calls, and other multi-person instant communication application scenarios. Multi-person instant communication between users is not restricted by the terminal, that is, users between the mobile terminal and the web terminal can have smooth audio interaction, and users between the browser terminal and the client terminal can also have smooth audio interaction.

[0072] Based on the same inventive concept as the above-mentioned embodiment, this embodiment provides a multi-person instant messaging method, such as Figure 4 As shown, the method comprises the following steps:

[0073] S410, using a selective forwarding unit SFU server to screen the audio streams of N users to obtain multiple reference audio streams;

[0074] In one implementation, a selective forwarding unit SFU server is used to screen the audio streams of N users to obtain multiple reference audio streams, including:

[0075] Use the SFU server to obtain the audio stream of each user and determine the volume information of each audio stream;

[0076] Determine the audio stream in a muted state according to the volume information of each audio stream;

[0077] Audio streams in a silent state are filtered out from the audio streams of N users to obtain multiple reference audio streams.

[0078] In one implementation, obtaining volume information of each audio stream includes:

[0079] For any audio stream, decode the audio stream to obtain each frame of voice signal;

[0080] Determine the speech sampling point based on the sampling frequency and the frame length of each frame of the speech signal;

[0081] Determine a first volume according to the PCM value of each speech sampling point;

[0082] The volume information V′ of the audio stream is determined based on the formula V′=10*log10V, wherein V is the first volume.

[0083] In one implementation, determining the first volume according to the PCM value of each speech sampling point includes:

[0084] According to the formula Determine a first volume V; wherein V S is the PCM value of the speech sampling point, and s is the sequence number of the speech sampling point.

[0085] Specifically, when the speech sampling point is determined based on the sampling frequency and the frame length of each frame of the speech signal, it is equivalent to converting the analog audio into digital audio. This embodiment is mainly implemented based on pulse code modulation (PCM), so each speech sampling point corresponds to a PCM value.

[0086] If the MCU server directly pulls the audio streams of all users from the SFU server, since the bit rate of one audio stream is around 64kbps to 192kbps, when there are a large number of users, pulling the stream will seriously occupy the bandwidth and reduce the communication quality.

[0087] Based on this, this embodiment takes into account that most users are usually in a silent state and only speak on the microphone for a short time. Therefore, the SFU server can be used to filter the audio streams of N users, determine the users in a silent state based on the volume information, filter out the audio streams in the silent state, and obtain a reference audio stream.

[0088] Here, since the stream data is transmitted via the webRTC protocol in this embodiment, in order to determine the volume information, the volume information can be set in the RTP extension header of each audio stream. Then the volume information of each audio stream can be obtained through the extension header information of each audio stream. Among them, the RTP extension header in this embodiment is in the Two-Byte header format. For the specific format diagram, please refer to Figure 5 ,exist Figure 5 In the data structure, data is used to store specific information of the audio stream.

[0089] In addition, the RTP extension header also carries the stream ID of each audio stream, and the user to which each audio stream belongs can be determined based on the stream ID. For example, when the stream ID is consistent with a user ID, it means that the audio stream belongs to the user.

[0090] The volume information uses the protobuf protocol and is defined as follows:

[0091]

[0092]

[0093] When you need to determine the volume information of the audio stream, you can parse the data field in the RTP extension header. Since the duration of each frame of voice in WebRTC is generally 10ms, based on a sampling rate of 48k, it contains a total of 480 sampling points. So according to the formula When determining the first volume V, S should be 480.

[0094] In this way, by using the SFU server to filter all audio streams, a small number of reference audio streams in a non-silent state are finally obtained. Then, when the MCU pulls the reference audio stream, the bandwidth pressure can be greatly reduced.

[0095] S411, using the multipoint control unit MCU server to pull and filter the multiple reference audio streams, obtain M target audio streams with the largest current volumes, and push the M target audio streams with the largest current volumes to the SFU server;

[0096] In this embodiment, since the MCU also needs time in the streaming process, there may be a high delay from speaking on the microphone to the transmission of the sound to other users. In order to reduce the sound delay rate and improve the communication quality, it is also necessary to use the MCU server to screen the multiple reference audio streams, obtain M target audio streams with the current largest volume, and push the M target audio streams with the current largest volume to the SFU server.

[0097] For example, assuming there are 10 reference audio streams, the MCU server needs to pull 10 reference audio streams, and then continue to filter based on the volume information of the reference audio streams, and finally obtain M target audio streams with the highest current volume.

[0098] Generally speaking, whether in a voice conference or a family voice group chat, when more than 3 to 5 people are talking at the same time, the communication experience will deteriorate. Therefore, the value of M is generally 3 to 5.

[0099] In this way, the MCU server only needs to pull multiple reference audio streams from the SFU server and push M reference audio streams to the SFU server. Therefore, the bandwidth occupied by the stream data interaction between the SFU server and the MCU server will be greatly reduced, thereby improving the communication quality.

[0100] It is worth noting that although the MCU server has a stream mixing function, in this embodiment, the MCU only needs to screen the reference audio stream and does not need to mix the stream. The M target audio streams pushed are also single streams, which further reduces bandwidth occupancy.

[0101] S412: Use the SFU server to push the M target audio streams with the highest current volumes to the N different users respectively.

[0102] After the SFU server obtains the M target audio streams, it pushes the M target audio streams with the highest current volume to the N users.

[0103] In one implementation, using the SFU server to push M target audio streams with the highest current volume to N users includes:

[0104] Push M target audio streams with the highest current volume to N different users in sequence; the push interval between two adjacent target audio streams is less than or equal to a preset time threshold.

[0105] Specifically, refer to Figure 6 In the dashed box part, if there are two users speaking and switching frequently, the voices of the two people will be heard intermittently in the M target audio streams. Therefore, this embodiment optimizes the communication quality by adding smooth switching.

[0106] For example, when you choose to push the target audio stream of the current user, you can switch to the target audio stream of other users for push only after the preset push interval. Even if the volume of other users is louder than that of the current user, it will not switch immediately, so that the sound sounds smoother. The preset push interval is 1 to 2 seconds, which can also be set based on actual conditions and is not limited here.

[0107] Furthermore, when pushing M target audio streams to the user, in order to prevent the user from hearing his own voice, in one implementation, the SFU server is used to push the M target audio streams with the largest current volume to the N different users, including:

[0108] Parse the RTP extension headers of the M target audio streams with the highest current volume to obtain the stream ID of each target audio stream;

[0109] For the current user, if it is determined that the user ID of the current user is consistent with the stream ID of the current target audio stream, the remaining M-1 target audio streams except the current target audio stream are pushed to the current user; the current user is any one of N different users.

[0110] That is, when the SFU server pushes the target audio stream, it will not push the user's own audio stream to the user himself.

[0111] For example, assuming that the target audio stream includes the audio stream of user A, when the SFU server pushes the target audio stream to user A, it will filter out the audio stream of user A itself, and then push the remaining M-1 target audio streams to N users.

[0112] In this way, when the target audio stream is pushed to the user, the communication quality is ensured by smoothly switching the sound, and the user can be prevented from hearing his own voice, thereby improving the user experience.

[0113] Based on the same inventive concept, this embodiment provides a computer device 700, such as Figure 7 As shown, it includes a memory 710, a processor 720, and a computer program 711 stored in the memory 710 and executable on the processor 720. When the processor 720 executes the computer program 711, any step of the method described above is implemented.

[0114] Based on the same inventive concept, this embodiment provides a computer-readable storage medium 800, such as Figure 8 As shown, a computer program 811 is stored thereon, and when the computer program 811 is executed by a processor, the steps of any of the methods described above are implemented.

[0115] Through one or more embodiments of the present invention, the present invention has the following beneficial effects or advantages:

[0116] The present invention provides a multi-person instant communication method, system, medium and device, the method comprising: using a selective forwarding unit SFU server to screen the audio streams of N users to obtain multiple reference audio streams; using a multipoint control unit MCU server to pull and screen the multiple reference audio streams to obtain M target audio streams with the largest current volumes, and pushing the M target audio streams to the SFU server; using the SFU server to push the M target audio streams to the N different users; in this way, based on the SFU+MCU architecture, the MCU can select only the M target audio streams with the largest current volumes from the N audios and push them to the SFU server, and then the SFU server pushes them to the users; generally speaking, in an instant communication scenario, the number of users with the largest voices is about 3 to 5, so it is equivalent to that each user only needs to push one channel of its own audio to the SFU server, and at the same time pull 3 to 5 target audio streams from the SFU, which greatly reduces the bandwidth pressure, and even in a communication scenario with a large number of users, the communication quality can be ensured.

[0117] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the best mode of the present invention.

[0118] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0119] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0120] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0121] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0122] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components in a gateway, proxy server, or system according to an embodiment of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0123] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0124] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0125] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-person instant messaging method, characterized in that: The method comprises: Using a selective forwarding unit SFU server to screen the audio streams of N users to obtain multiple reference audio streams; Using a multipoint control unit MCU server to pull and filter the multiple reference audio streams, obtain M target audio streams with the largest current volumes, and push the M target audio streams with the largest current volumes to the SFU server; Using the SFU server, the M target audio streams with the largest current volumes are pushed to the N different users respectively; The selective forwarding unit SFU server is used to screen the audio streams of N users to obtain multiple reference audio streams, including: Obtain the audio stream of each user using the SFU server, and determine the volume information of each audio stream; Determine the audio stream in a muted state according to the volume information of each audio stream; Filter out the audio streams in a silent state from the audio streams of N users to obtain the multiple reference audio streams; The multipoint control unit MCU server is used to pull and filter the multiple reference audio streams to obtain M target audio streams with the largest current volume, including: Pulling the multiple reference audio streams from the SFU server; Acquire volume information of the multiple reference audio streams based on extended header information of each reference audio stream; sorting the plurality of reference audio streams based on volume information of each reference audio stream; Filter out M audio streams with the highest current volume from the sorted multiple reference audio streams; The using the SFU server to push the M target audio streams with the largest current volumes to the N different users includes: Parsing the RTP extension headers of the M target audio streams with the highest current volumes to obtain a stream ID of each target audio stream; For the current user, if it is determined that the user ID of the current user is consistent with the stream ID of the current target audio stream, the remaining M-1 target audio streams except the current target audio stream are pushed to the current user; the current user is any one of the N different users; Using the SFU server to push the M target audio streams with the largest current volumes to the N different users respectively includes: The M target audio streams with the largest current volumes are pushed to the N different users in sequence; the push interval between two adjacent target audio streams is less than or equal to a preset time threshold; The step of obtaining the volume information of each audio stream includes: For any audio stream, decode the audio stream to obtain each frame of speech signal; Determine the speech sampling point based on the sampling frequency and the frame length of each frame of the speech signal; Determine a first volume according to a pulse modulation coding PCM value of each of the speech sampling points; The volume information V′ of the audio stream is determined based on the formula V′=10*log10V; wherein V is the first volume.

2. The method according to claim 1, characterized in that The determining the first volume according to the PCM value of each of the speech sampling points includes: According to the formula Determine the first volume V; wherein the V S is the PCM value of the speech sampling point, and s is the sequence number of the speech sampling point.

3. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to claim 1 are implemented.

4. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to claim 1 are implemented.