Multi-channel audio and video mixing method and multi-channel audio and video mixing system

Abstract

This application provides a multi-channel audio and video mixing method and system. The method includes: the Storage Unit (SFU) acquiring audio and video stream data and status parameters of each terminal device; determining the processed audio and video stream data corresponding to each terminal device based on the status parameters and audio and video stream data of each terminal device; the SFU generating multiple push messages based on the processed audio and video stream data corresponding to each terminal device and sending them to the mixing engine; the mixing engine creating media stream pipelines corresponding to each terminal device based on each push message; pulling the processed audio and video stream data corresponding to each terminal device from the SFU and inputting it into the media stream pipelines; mixing the processed audio and video stream data based on each media stream pipeline and the integration unit in the mixing engine to generate mixed audio and video stream data and sending it to the SFU and each terminal device; the SFU recording and storing the mixed audio and video stream data, and each terminal device rendering the mixed audio and video stream data to improve the mixing effect.

Description

Technical Field

[0001] This application relates to the field of communication technology, and more specifically, to a multi-channel audio and video mixing method and a multi-channel audio and video mixing system. Background Technology

[0002] In real-time audio and video communication scenarios based on Web Real-Time Communication (WebRTC), a Selective Forwarding Unit (SFU) architecture is typically used to implement multi-party audio and video communication.

[0003] Currently, the SFU architecture mainly consists of one SFU and multiple terminals. The SFU selectively forwards the audio and video streams uploaded by each terminal without performing multi-stream mixing. Each terminal needs to independently pull multiple streams and render them locally, leading to problems such as high terminal resource consumption, audio and video desynchronization, and complex window management. In addition, current multi-stream mixing recording for multiple audio and video streams usually relies on third-party cloud recording services, which are costly and lack flexibility.

[0004] Therefore, the existing SFU architecture has certain limitations in mixing and recording multiple audio and video streams. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a multi-channel audio and video mixing method and system, so as to solve the practical problem that the existing SFU architecture has certain limitations in mixing and recording multi-channel audio and video streams.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0007] In a first aspect, embodiments of this application provide a multi-channel audio and video mixing method, the method being applied to a multi-channel audio and video stream mixing system, the multi-channel audio and video stream mixing system including an SFU, a mixing engine, and multiple terminal devices; the method includes:

[0008] SFU acquires the audio and video stream data of each terminal device and the status parameters of each terminal device, and determines the processed audio and video stream data corresponding to each terminal device based on the status parameters of each terminal device and the audio and video stream data.

[0009] The SFU generates multiple push messages based on the processed audio and video stream data corresponding to each terminal device, and sends each push message to the mixing engine;

[0010] The mixing engine creates media stream pipelines corresponding to each terminal device based on each push message, pulls the processed audio and video stream data corresponding to each terminal device from the SFU and inputs it into the media stream pipeline, mixes the processed audio and video stream data based on each media stream pipeline and the integration unit in the mixing engine, generates mixed audio and video stream data, and sends the mixed audio and video stream data to the SFU and each terminal device respectively.

[0011] The SFU records and stores the mixed audio and video stream data, and each of the terminal devices renders the mixed audio and video stream data.

[0012] As an optional implementation, determining the processed audio and video stream data corresponding to each terminal device based on the status parameters of each terminal device and the audio and video stream data includes:

[0013] The SFU performs transcoding and single-channel mixing processing on the audio and video stream data of each terminal device according to the status parameters of each terminal device, and determines the processed audio and video stream data corresponding to each terminal device.

[0014] As an optional implementation, the SFU performs transcoding and single-channel mixing processing on the audio and video stream data of each terminal device sequentially based on the status parameters of each terminal device, thereby determining the processed audio and video stream data corresponding to each terminal device, including:

[0015] The SFU decodes the audio and video stream data of the first terminal device to obtain the first decoded video stream data and the first decoded audio stream data, wherein the first terminal device is any terminal device in the multi-channel audio and video stream mixing system.

[0016] The SFU determines whether to adjust the resolution and / or frame rate of the first decoded video stream data based on the status parameters of the first terminal device.

[0017] If so, the SFU will reduce the resolution and / or frame rate of the first decoded video stream data to a preset resolution and / or frame rate to obtain the first adjusted video stream data. After clock synchronization of the first adjusted video stream data and the first decoded audio stream data, the SFU will encapsulate them into the first processed audio and video stream data, wherein the first processed audio and video stream data is the processed audio and video stream data corresponding to the first terminal device.

[0018] Otherwise, the SFU performs clock synchronization on the first decoded video stream data and the first decoded audio stream data, and then encapsulates them into the first processed audio and video stream data.

[0019] As an optional implementation, the SFU generates multiple push messages based on the processed audio and video stream data corresponding to each terminal device, including:

[0020] The SFU generates metadata for the processed audio and video stream data corresponding to each terminal device based on the identifier of each terminal device and the resolution and / or frame rate of the video stream in the processed audio and video stream data corresponding to each terminal device.

[0021] The SFU generates each push message based on the metadata of the processed audio and video stream data corresponding to each terminal device.

[0022] As an optional implementation, the mixing engine creates media stream pipelines corresponding to each terminal device based on each push message, pulls the processed audio and video stream data corresponding to each terminal device from the SFU and inputs it into the media stream pipeline, including:

[0023] The mixing engine determines, based on the first push message, whether the resolution and / or frame rate of the video stream in the first processed audio and video stream data is greater than the preset resolution and / or preset frame rate.

[0024] If so, the mixing engine creates a first media stream pipeline corresponding to the first terminal device according to the first push message, and pulls the first processed audio and video stream data from the SFU and inputs it into the first media stream pipeline;

[0025] Otherwise, the mixing engine discards the first push message.

[0026] As an optional implementation, the mixing engine creates a first media stream pipeline corresponding to the first terminal device based on the first push message, and pulls the first processed audio and video stream data from the SFU and inputs it into the first media stream pipeline, including:

[0027] The mixing engine creates a first media stream pipeline corresponding to the first terminal device based on the identifier of the first terminal device in the first push message;

[0028] The mixing engine generates a first stream filter in the first media stream pipeline, and uses a coroutine mode to cyclically pull the first processed audio and video stream data from the SFU and input it into the first media stream pipeline. The first stream filter separates the first processed audio stream and the first processed video stream from the first processed audio and video stream data.

[0029] As an optional implementation, before mixing the processed audio and video stream data based on the media stream pipelines and the integrated unit in the mixing engine, the following steps are included:

[0030] The mixing engine creates a first video compositer and a first audio mixer in the first media stream pipeline;

[0031] The first video compositer assembles the first processed video streams at a preset time interval to obtain the first assembled video stream, and the first audio mixer assembles the first processed audio streams at a preset time interval to obtain the first assembled audio stream.

[0032] The mixing engine clock-synchronizes the first assembled video stream and the first assembled audio stream and then encapsulates them into first media stream data.

[0033] As an optional implementation, the step of mixing the processed audio and video stream data based on the integrated unit in each media stream pipeline and the mixing engine to generate mixed audio and video stream data, and sending the mixed audio and video stream data to the SFU and each of the terminal devices respectively, includes:

[0034] The media switches in each media stream pipeline detect the transmission continuity and data integrity of each media stream data to obtain the detection status value of each media stream data;

[0035] The integration unit receives each streaming media stream data to be mixed with a detection status value of the first detection status value, performs clock synchronization and merging processing on each streaming media stream data to be mixed, generates the mixed audio and video stream data, and sends the mixed audio and video stream data to the media output device in the mixing engine;

[0036] The media output device sends the mixed audio and video stream data to the SFU and each of the terminal devices respectively.

[0037] As an optional implementation, the SFU is externally connected to a memory;

[0038] The SFU records and stores the mixed audio and video stream data, including:

[0039] The SFU records the mixed audio and video stream data to obtain recorded mixed audio and video stream data, and saves the recorded mixed audio and video stream data in the memory.

[0040] Secondly, embodiments of this application provide a multi-channel audio and video stream mixing system, which includes an SFU, a mixing engine, and multiple terminal devices. The SFU and the mixing engine are both communicatively connected to each of the terminal devices, and the SFU is communicatively connected to the mixing engine.

[0041] The multi-channel audio and video stream mixing system is used to perform the steps of the multi-channel audio and video stream mixing method described in the first aspect above.

[0042] The beneficial effects of this application are:

[0043] This application provides a multi-channel audio and video stream mixing method and system. The Storage Unit (SFU) acquires audio and video stream data and status parameters from each terminal device, and determines the processed audio and video stream data corresponding to each terminal device based on these parameters and data. The SFU generates multiple push messages based on the processed audio and video stream data for each terminal device and sends them to the mixing engine. The mixing engine creates media stream pipelines corresponding to each terminal device based on the push messages, pulls the processed audio and video stream data from the SFU, inputs it into the media stream pipelines, and mixes the processed audio and video stream data based on the media stream pipelines and the integrated unit in the mixing engine, generating mixed audio and video stream data, which is then sent to the SFU and each terminal device respectively. The SFU records and stores the mixed audio and video stream data, and each terminal device renders the mixed audio and video stream data. By introducing a mixing engine into the multi-channel audio and video stream mixing system, SFU performs initial filtering of the multi-channel audio and video stream data. Then, a secondary, deeper filtering is applied to the media stream pipelines of each terminal device within the mixing engine. Finally, the mixing engine's integration unit mixes the filtered audio and video streams to generate a hybrid audio and video stream, significantly improving the mixing effect and quality. This greatly reduces network resource and performance consumption on each terminal device, enhances their stability, and enables mixed recording without relying on third-party cloud recording. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of the architecture of a multi-channel audio and video stream mixing system provided in an embodiment of this application;

[0046] Figure 2 A schematic diagram illustrating a scenario of real-time multi-party audio and video communication provided in an embodiment of this application;

[0047] Figure 3 This is a schematic diagram of the structure of the hybrid engine provided in the embodiments of this application;

[0048] Figure 4 A flowchart illustrating the multi-channel audio and video stream mixing method provided in the embodiments of this application. Figure 1 ;

[0049] Figure 5 A flowchart illustrating the multi-channel audio and video stream mixing method provided in the embodiments of this application. Figure 2 ;

[0050] Figure 6 A flowchart illustrating the multi-channel audio and video stream mixing method provided in the embodiments of this application. Figure 3 ;

[0051] Figure 7 A flowchart illustrating the multi-channel audio and video stream mixing method provided in the embodiments of this application. Figure 4 ;

[0052] Figure 8 A flowchart illustrating the multi-channel audio and video stream mixing method provided in the embodiments of this application. Figure 5 ;

[0053] Figure 9 A flowchart illustrating the multi-channel audio and video stream mixing method provided in the embodiments of this application. Figure 6 ;

[0054] Figure 10 A flowchart illustrating the multi-channel audio and video stream mixing method provided in the embodiments of this application. Figure 7 . Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0056] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0057] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0058] In WebRTC-based real-time audio and video communication scenarios, the SFU (Streaming Support Unit) architecture is typically used to implement multi-party audio and video communication. Currently, the SFU architecture mainly consists of one SFU and multiple terminals. The SFU selectively forwards the audio and video streams uploaded by each terminal without performing multi-stream mixing. Each terminal needs to independently pull multiple streams and render them locally, leading to problems such as high terminal resource consumption, audio and video desynchronization, and complex window management. In addition, current multi-stream mixing and recording for multi-audio and video streams usually relies on third-party cloud recording services, which are costly and lack flexibility. In other words, the existing SFU architecture has certain limitations in handling multi-stream mixing and recording of audio and video streams.

[0059] This application provides a method for mixing multiple audio and video streams, addressing the aforementioned problems. By utilizing a mixing engine within a multi-channel audio and video stream mixing system, it achieves efficient mixing of multiple audio and video streams and improves the mixing effect. Furthermore, an external memory connected to the SFU (Streaming Unit) enables effective storage of the recorded mixed audio and video data.

[0060] Figure 1 This is a schematic diagram of the architecture of a multi-channel audio and video stream mixing system provided in an embodiment of this application, as shown below. Figure 1 As shown, the multi-channel audio and video stream mixing system includes an SFU (Streaming Unit), a mixing engine, and multiple terminal devices. Both the SFU and the mixing engine are communicatively connected to each terminal device, and the SFU is also communicatively connected to the mixing engine.

[0061] Figure 2 This is a schematic diagram of a multi-party real-time audio and video communication scenario provided in an embodiment of this application, with reference to... Figure 1 and Figure 2 In multi-party conferences and other multi-party real-time audio and video communication scenarios, taking N terminal devices as an example, terminal device 1, terminal device 2, terminal device 3, ..., terminal device N access the same video virtual room (like a virtual conference room) through the network. After access, the N terminal devices transmit the locally collected audio and video streams to the SFU respectively.

[0062] SFU synchronously acquires the status parameters of N terminal devices, and processes the audio and video stream data of the N terminal devices according to the status parameters of the N terminal devices to obtain the processed audio and video stream data corresponding to the N terminal devices. The purpose of SFU in this process is to initially filter out the audio and video stream data of the terminal devices with poor status among the N terminal devices to avoid affecting the subsequent mixing quality and effect.

[0063] The mixing engine identifies the N terminal devices whose status meets the mixing requirements and creates a media stream pipeline for each terminal device. It pulls audio and video stream data from the SFU (System-on-Functional Unit) from these N devices and inputs it into the corresponding media stream pipeline. Based on its internal structure, the mixing engine performs secondary filtering on the processed audio and video stream data, primarily targeting encoding anomalies and other deep-level data filtering. After this secondary filtering, it performs mixing to generate hybrid audio and video stream data, which is then sent to the SFU and the N terminal devices, achieving efficient and high-quality mixing of multiple audio and video streams.

[0064] N terminal devices receive mixed audio and video streams from the mixing engine and render them locally. The video feed displays split-screen / focused images of all participants, and the audio output is a mixed sound from all participants, ensuring real-time and consistent rendering across all terminal devices. In other words, each terminal device can clearly and smoothly see the video feed of all participants and hear their clear conversations locally, with synchronized video and audio across all devices without delay.

[0065] Figure 3 This is a schematic diagram of the structure of the hybrid engine provided in the embodiments of this application, as shown below. Figure 3 As shown, the mixing engine creates media stream pipelines corresponding to each terminal device, and each media stream pipeline deploys a stream filter, a video compositer, an audio mixer, and a media switch. The stream filter separates the processed audio and video data pulled by the mixing engine. The video compositer assembles the separated video streams at preset time intervals, and the audio mixer assembles the separated audio streams at preset time intervals. The mixing engine clock-synchronizes the assembled video and audio streams and encapsulates them into media stream data. The media switch detects the status of the media stream data.

[0066] Continue to refer to Figure 3 The mixing engine also includes an integration unit and a media output device. The detected media stream data is fed into the integration unit, which then mixes the detected media stream data to generate mixed audio and video stream data, which is then output to the SFU and N terminal devices through the media output device.

[0067] Figure 4 A flowchart illustrating the multi-channel audio and video stream mixing method provided in the embodiments of this application. Figure 1 The subject of this method is the aforementioned Figure 1 The aforementioned multi-channel audio and video stream mixing system. For example... Figure 4 As shown, the method includes:

[0068] S101 and SFU acquire audio and video stream data and status parameters of each terminal device, and determine the processed audio and video stream data corresponding to each terminal device based on the status parameters and audio and video stream data of each terminal device.

[0069] Optionally, the SFU establishes a connection with each terminal device via RTC technology, uses the Real-time Transport Protocol (RTP) to acquire audio and video stream data from each terminal device in real time, and collects status parameters of each terminal device. These status parameters characterize the network and device status of the terminal device. For example, the status parameters of the terminal device may include its transmission bandwidth, latency, packet loss rate, camera resolution, etc.

[0070] SFU processes the audio and video stream data of each terminal device according to the status parameters of each terminal device, and determines the processed audio and video stream data corresponding to each terminal device. Specifically, SFU can dynamically adjust relevant parameters in the audio and video stream data of each terminal device according to the status parameters of each terminal device, such as video resolution / frame rate, so as to initially filter out the audio and video stream data of terminal devices with poor network or device status before mixing, so as to avoid affecting the subsequent mixing quality and effect.

[0071] S102 and SFU generate multiple push messages based on the processed audio and video stream data corresponding to each terminal device, and send each push message to the mixing engine.

[0072] Optionally, SFU generates each push message based on the processed audio and video stream data corresponding to each terminal device. That is, SFU generates a standardized push message for the processed audio and video stream data corresponding to each terminal device.

[0073] SFU sends each push message to the mixing engine through a message queue, triggering the mixing engine to start the mixing task.

[0074] S103 The mixing engine creates media stream pipelines corresponding to each terminal device based on each push message, pulls the processed audio and video stream data corresponding to each terminal device from the SFU and inputs it into the media stream pipeline, mixes the processed audio and video stream data based on each media stream pipeline and the integration unit in the mixing engine, generates mixed audio and video stream data, and sends the mixed audio and video stream data to the SFU and each terminal device respectively.

[0075] Optionally, the mixing engine creates an independent media stream pipeline for each terminal device based on each push message, and pulls the processed audio and video stream data corresponding to each terminal device from the SFU in real time and inputs it into the corresponding media stream pipeline. In other words, the media stream pipeline corresponding to each terminal device can isolate the processed audio and video stream data of different terminal devices.

[0076] The media stream pipelines of the mixing engine are used for the transmission and secondary filtering of processed audio and video stream data corresponding to each terminal device. The secondary filtering mainly involves deep filtering of data with encoding abnormalities to prevent abnormal stream data with packet loss, frame loss, etc. from entering the mixing stage.

[0077] The integrated unit in the mixing engine mixes the multiple normal stream data after secondary filtering to generate mixed audio and video stream data. The mixing engine then sends the generated mixed audio and video stream data to the SFU and each terminal device.

[0078] S104 and SFU record and store the mixed audio and video stream data, and each terminal device renders the mixed audio and video stream data.

[0079] Optionally, SFU records and persistently stores the mixed audio and video stream data output by the mixing engine in real time, without relying on third-party cloud recording.

[0080] Each terminal device renders the mixed audio and video stream data locally in real time, ensuring that the video screen displays split-screen / focused images of all participants, and the audio output contains the mixed sound of all participants, guaranteeing that the rendered content on each terminal device is real-time and consistent. In other words, each terminal device can clearly and smoothly see the video of all participants and hear their clear conversations locally, with no delay in the synchronization of video and audio across all terminals.

[0081] In this way, each terminal device does not need to fetch audio and video data from other terminal devices. It can achieve audio and video communication with other terminal devices by rendering the mixed audio and video stream data output by the mixing engine. This greatly reduces the network resources and performance consumption of each terminal device, avoids application anomalies, crashes or freezes that may occur when terminal devices fetch audio and video data from other terminal devices, and improves the stability of each terminal device.

[0082] In this embodiment, the SFU acquires the audio and video stream data and status parameters of each terminal device, and determines the processed audio and video stream data corresponding to each terminal device based on the status parameters and audio and video stream data. The SFU generates multiple push messages based on the processed audio and video stream data corresponding to each terminal device and sends them to the mixing engine. The mixing engine creates media stream pipelines corresponding to each terminal device based on each push message, pulls the processed audio and video stream data corresponding to each terminal device from the SFU and inputs it into the media stream pipelines, and mixes the processed audio and video stream data based on each media stream pipeline and the integration unit in the mixing engine to generate mixed audio and video stream data, which is then sent to the SFU and each terminal device respectively. The SFU records and stores the mixed audio and video stream data, and each terminal device renders the mixed audio and video stream data. By introducing a mixing engine into the multi-channel audio and video stream mixing system, SFU performs initial filtering of the multi-channel audio and video stream data. Then, a secondary, deeper filtering is applied to the media stream pipelines of each terminal device within the mixing engine. Finally, the mixing engine's integration unit mixes the filtered audio and video streams to generate a hybrid audio and video stream, significantly improving the mixing effect and quality. This greatly reduces network resource and performance consumption on each terminal device, enhances their stability, and enables mixed recording without relying on third-party cloud recording.

[0083] As an optional implementation, step S101 above, which determines the processed audio and video stream data corresponding to each terminal device based on the status parameters of each terminal device and the audio and video stream data, includes:

[0084] SFU performs transcoding and single-channel mixing processing on the audio and video stream data of each terminal device according to the status parameters of each terminal device, and determines the processed audio and video stream data corresponding to each terminal device.

[0085] Optionally, SFU performs transcoding and single-channel mixing processing on the audio and video stream data of each terminal device in sequence according to the status parameters such as transmission bandwidth, latency, packet loss rate and / or camera resolution of each terminal device, and determines the processed audio and video stream data corresponding to each terminal device.

[0086] During the transcoding process, SFU dynamically adjusts relevant parameters in the audio and video stream data of each terminal device based on the status parameters of each terminal device, such as video resolution / frame rate, so as to initially filter out the audio and video stream data of terminal devices with poor network or device status before mixing, so as to avoid affecting the subsequent mixing quality and effect, and ensure the smoothness and clarity of the video after mixing.

[0087] In this embodiment, the SFU performs transcoding and single-channel mixing processing on the audio and video stream data of each terminal device sequentially based on the status parameters of each terminal device, thereby determining the processed audio and video stream data corresponding to each terminal device. The SFU dynamically adjusts relevant parameters in the audio and video stream data of each terminal device based on the status parameters of each terminal device, and performs preliminary filtering on the audio and video stream data of each terminal device to ensure the smoothness and clarity of the mixed video.

[0088] Figure 5 A flowchart illustrating the multi-channel audio and video stream mixing method provided in the embodiments of this application. Figure 2 ,like Figure 5 As shown, in the above steps, SFU performs transcoding and single-channel mixing processing on the audio and video stream data of each terminal device according to the status parameters of each terminal device, and determines the processed audio and video stream data corresponding to each terminal device, including:

[0089] S201 and SFU decode the audio and video stream data of the first terminal device to obtain the first decoded video stream data and the first decoded audio stream data, wherein the first terminal device is any terminal device in the multi-channel audio and video stream mixing system.

[0090] Optionally, taking the first terminal device as an example, the transcoding and single-channel mixing process of SFU will be explained. Here, the first terminal device is any terminal device in a multi-channel audio / video stream mixing system, i.e. Figure 1 or Figure 2 Any terminal device among N terminal devices.

[0091] SFU decodes the audio and video stream data of the first terminal device, separating the audio and video stream data of the first terminal device into independent first decoded video stream data and first decoded audio stream data.

[0092] Specifically, SFU decodes and restores the compressed video data in the audio and video stream data of the first terminal device to the original video data, generating the first decoded video stream data, and decodes and restores the compressed audio data in the audio and video stream data of the first terminal device to the original audio data, generating the first decoded audio stream data.

[0093] S202 and SFU determine whether to adjust the resolution and / or frame rate of the first decoded video stream data based on the status parameters of the first terminal device.

[0094] Optionally, SFU determines whether the network status and device status of the first terminal device will reduce the smoothness and clarity of the video after mixing, based on status parameters such as the transmission bandwidth, latency, packet loss rate and / or camera resolution of the first terminal device.

[0095] Based on the judgment result, the SFU determines whether to adjust the resolution and / or frame rate of the first decoded video stream data. Specifically, if the SFU determines, based on the status parameters of the first terminal device, that the network status and device status of the first terminal device are poor, which will reduce the smoothness and clarity of the video after mixing, then the SFU determines that it is necessary to adjust the resolution and / or frame rate of the first decoded video stream data in order to filter out the audio and video stream data of the first terminal device before mixing, so as to prevent the audio and video stream data of the first terminal device from entering the subsequent mixing stage and reducing the smoothness and clarity of the video after mixing.

[0096] Accordingly, if the SFU determines that the network status and device status of the first terminal device are good based on the status parameters of the first terminal device, and will not reduce the smoothness and clarity of the video after mixing, then the SFU determines that there is no need to adjust the resolution and / or frame rate of the first decoded video stream data, and the audio and video stream data of the first terminal device can enter the subsequent mixing stage to participate in the mixing.

[0097] S203. If so, the SFU lowers the resolution and / or frame rate of the first decoded video stream data to a preset resolution and / or preset frame rate to obtain the first adjusted video stream data. After clock synchronization of the first adjusted video stream data and the first decoded audio stream data, the SFU encapsulates them into the first processed audio and video stream data, wherein the first processed audio and video stream data is the processed audio and video stream data corresponding to the first terminal device.

[0098] Optionally, if the SFU determines that the resolution and / or frame rate of the first decoded video stream data needs to be adjusted, a resolution and / or frame rate adjustment mechanism is triggered to reduce the resolution and / or frame rate of the first decoded video stream data to a preset resolution and / or preset frame rate, thereby generating the first adjusted video stream data.

[0099] After clock synchronization of the first adjusted video stream data and the first decoded audio stream data, SFU ensures that the timestamps of the first adjusted video stream data and the first decoded audio stream data are precisely aligned to avoid audio-visual asynchrony. The clock-synchronized first adjusted video stream data and the first decoded audio stream data are then encapsulated into first processed audio-video stream data, which is the processed audio-video stream data corresponding to the first terminal device.

[0100] If the video stream data at the preset resolution and / or preset frame rate belongs to a terminal device whose processed audio and video stream data cannot be retrieved by the mixing engine, it cannot participate in the mixing process, thus failing to guarantee mixing quality and effect. In other words, if the network and device conditions of the first terminal device are poor, after the resolution and / or frame rate of the first adjusted video stream data is lowered to the preset resolution and / or preset frame rate, the first processed audio and video stream data cannot be retrieved by the mixing engine and therefore cannot participate in the mixing process.

[0101] S204. Otherwise, after clock synchronization of the first decoded video stream data and the first decoded audio stream data, the SFU encapsulates them into the first processed audio and video stream data.

[0102] Optionally, if the SFU determines that there is no need to adjust the resolution and / or frame rate of the first decoded video stream data, then the first decoded video stream data and the first decoded audio stream data are clock-synchronized directly to ensure that the timestamps of the first decoded video stream data and the first decoded audio stream data are accurately aligned to avoid audio-visual asynchrony. The clock-synchronized first decoded video stream data and the first decoded audio stream data are then encapsulated into the first processed audio-video stream data.

[0103] Because the resolution and / or frame rate of the first decoded video stream data are high, the first processed audio and video stream data can be pulled by the mixing engine and used for mixing without reducing the smoothness and clarity of the mixed video.

[0104] In this embodiment, the SFU decodes the audio and video stream data of the first terminal device to obtain first decoded video stream data and first decoded audio stream data. Based on the status parameters of the first terminal device, it determines whether to adjust the resolution and / or frame rate of the first decoded video stream data. If so, the SFU lowers the resolution and / or frame rate of the first decoded video stream data to a preset resolution and / or preset frame rate to obtain first adjusted video stream data. The SFU then clock-synchronizes the first adjusted video stream data and the first decoded audio stream data and encapsulates them into first processed audio and video stream data. Otherwise, the SFU clock-synchronizes the first decoded video stream data and the first decoded audio stream data and encapsulates them into first processed audio and video stream data. By dynamically adjusting the decoded video stream data in the audio and video stream data of the terminal device through the SFU, the processed audio and video streams of terminal devices with poor network or device conditions are prevented from participating in the mixing process, thus ensuring the smoothness and clarity of the mixed video.

[0105] Figure 6 A flowchart illustrating the multi-channel audio and video stream mixing method provided in the embodiments of this application. Figure 3 ,like Figure 6 As shown, in step S102 above, the SFU generates multiple push messages based on the processed audio and video stream data corresponding to each terminal device, including:

[0106] S301 and SFU generate metadata for the processed audio and video stream data corresponding to each terminal device based on the identifier of each terminal device and the resolution and / or frame rate of the video stream in the processed audio and video stream data corresponding to each terminal device.

[0107] Optionally, SFU can be configured based on the identifier of each terminal device. The resolution and / or frame rate of the video stream in the processed audio and video stream data corresponding to each terminal device are used to generate metadata for the processed audio and video stream data corresponding to each terminal device.

[0108] In other words, the fields included in the metadata are the identifiers of each terminal device. The resolution and / or frame rate of the video stream in the processed audio and video stream data corresponding to each terminal device.

[0109] S302 and SFU generate push messages based on the metadata of the processed audio and video stream data corresponding to each terminal device.

[0110] Optionally, SFU encapsulates the metadata of the processed audio and video stream data corresponding to each terminal device into a push message. This allows the mixing engine to push messages before fetching data. Accurately determine whether it is necessary to pull push messages from SFU The processed audio and video stream data corresponding to the terminal device.

[0111] In this embodiment, the SFU generates metadata for the processed audio and video stream data corresponding to each terminal device based on the identifier of each terminal device and the resolution and / or frame rate of the video stream in the processed audio and video stream data corresponding to each terminal device. Based on this metadata, the SFU generates push messages. This allows the mixing engine to accurately determine whether it needs to pull the processed audio and video stream data corresponding to the terminal device to which the push message belongs from the SFU before pulling data.

[0112] Figure 7 A flowchart illustrating the multi-channel audio and video stream mixing method provided in the embodiments of this application. Figure 4 ,like Figure 7 As shown, in step S103 above, the mixing engine creates media stream pipelines corresponding to each terminal device based on each push message, pulls the processed audio and video stream data corresponding to each terminal device from the SFU and inputs it into the media stream pipeline, including:

[0113] S401, the mixing engine determines, based on the first push message, whether the resolution and / or frame rate of the video stream in the first processed audio and video stream data is greater than the preset resolution and / or preset frame rate.

[0114] Optionally, the first push message corresponding to the first terminal device. Taking this as an example, the process of the mixing engine creating a media stream pipeline corresponding to the terminal device based on the push message, pulling the processed audio and video stream data corresponding to the terminal device from the SFU and inputting it into the media stream pipeline is explained in detail.

[0115] The mixing engine parses the first push message. Extract the resolution and / or frame rate of the video stream and the identifier of the first terminal device from the metadata of the processed audio and video stream data corresponding to the first terminal device. .

[0116] The mixing engine determines whether the resolution and / or frame rate of the video stream in the first processed audio and video stream data is greater than the preset resolution and / or frame rate, and determines whether to create the first media stream pipeline corresponding to the first terminal device and whether to pull the first processed audio and video stream data from the SFU based on the determination result.

[0117] S402. If so, the mixing engine creates a first media stream pipeline corresponding to the first terminal device according to the first push message, and pulls the first processed audio and video stream data from the SFU and inputs it into the first media stream pipeline.

[0118] Optionally, if the resolution and / or frame rate of the video stream in the first processed audio and video stream data is greater than the preset resolution and / or preset frame rate, the mixing engine will adjust the settings according to the identifier of the first terminal device. Create a corresponding first media stream pipeline for the first terminal device, for example Figure 3 Media stream pipe 1 is shown.

[0119] The first terminal device creates a corresponding first media stream pipeline for isolating and transmitting the first processed audio and video stream data corresponding to the first terminal device. Specifically, the mixing engine pulls the first processed audio and video stream data from the SFU in real time and inputs it into the first media stream pipeline.

[0120] S403. Otherwise, the mixing engine discards the first push message.

[0121] Optionally, if the resolution and / or frame rate of the video stream in the first processed audio and video stream data does not exceed the preset resolution and / or frame rate, the mixing engine discards the first push message. There is no need to create a corresponding first media stream pipeline for the first terminal device, nor is it necessary to pull the first processed audio and video stream data from the SFU. This avoids the first processed audio and video stream data from participating in the mixing process, which would affect the smoothness and clarity of the mixed video.

[0122] In this embodiment, the mixing engine determines, based on the first push message, whether the resolution and / or frame rate of the video stream in the first processed audio and video stream data is greater than a preset resolution and / or preset frame rate. If so, the mixing engine creates a first media stream pipeline corresponding to the first terminal device based on the first push message, and pulls the first processed audio and video stream data from the SFU and inputs it into the first media stream pipeline. Otherwise, the mixing engine discards the first push message. This ensures mixing quality and guarantees the smoothness and clarity of the mixed video.

[0123] Figure 8A flowchart illustrating the multi-channel audio and video stream mixing method provided in the embodiments of this application. Figure 5 ,like Figure 8 As shown, in step S402 above, the mixing engine creates a first media stream pipeline corresponding to the first terminal device based on the first push message, and pulls the first processed audio and video stream data from the SFU and inputs it into the first media stream pipeline, including:

[0124] S501, The mixing engine creates a first media stream pipeline corresponding to the first terminal device based on the identifier of the first terminal device in the first push message.

[0125] Optionally, the mixing engine determines the amount of data based on the first push message. Identification of the first terminal device To create a dedicated first media stream pipeline for the first terminal device, namely Figure 3 Media stream pipe 1 is shown.

[0126] S502, the mixing engine generates a first stream filter in the first media stream pipeline, and uses a coroutine mode to cyclically pull the first processed audio and video stream data from the SFU and input it into the first media stream pipeline. The first stream filter separates the first processed audio stream and the first processed video stream from the first processed audio and video stream data.

[0127] Optionally, continue to refer to Figure 3 The mixing engine generates the first stream filter in the first media stream pipeline. First-class filter Used for separating audio and video streams and filtering out media impurities.

[0128] The mixing engine determines the first push message based on the mixing engine. The system uses a coroutine model to cyclically pull the first processed audio and video stream data from the SFU, inputs the pulled first processed audio and video stream data into the first media stream pipeline, and flows through the first stream filter. .

[0129] First-class filter The first processed audio stream and the first processed video stream are separated from the first processed audio and video stream data of the single-channel mixed data, and media impurities are filtered out.

[0130] In this embodiment, the mixing engine creates a first media stream pipeline corresponding to the first terminal device based on the identifier of the first terminal device in the first push message, generates a first stream filter in the first media stream pipeline, and uses a coroutine mode to cyclically pull the first processed audio and video stream data from the SFU and input it into the first media stream pipeline. The first stream filter separates the first processed audio stream and the first processed video stream from the first processed audio and video stream data. By creating an independent media stream pipeline for each terminal device, the mixing engine avoids mutual interference between the processed audio and video stream data corresponding to different terminal devices, and achieves audio and video separation and media impurity filtering of a single-channel mixed audio and video stream through the stream filter in the media stream pipeline.

[0131] Figure 9 A flowchart illustrating the multi-channel audio and video stream mixing method provided in the embodiments of this application. Figure 6 ,like Figure 9 As shown, before mixing the processed audio and video stream data based on each media stream pipeline and the integrated unit in the mixing engine in step S103 above, the following steps are included:

[0132] S601, the mixing engine creates a first video compositer and a first audio mixer in the first media stream pipeline.

[0133] Optionally, the mixing engine creates the first video compositer in the first media stream pipeline. and the first audio mixer Among them, the first video compositer First audio mixer used to assemble a first processed video stream at a preset time interval. Used to assemble the first processed audio stream at a preset time interval.

[0134] S602. The first video compositer assembles the first processed video stream at a preset time interval to obtain the first assembled video stream, and the first audio mixer assembles the first processed audio stream at a preset time interval to obtain the first assembled audio stream.

[0135] Optionally, by the first video compositer Maintain a video stream buffer queue within a time window (i.e., a preset time interval). Assemble the first processed video stream within the preset time interval into a continuous video stream in chronological order to obtain the first assembled video stream.

[0136] Accordingly, by the first audio mixer Maintain an audio stream buffer queue within a time window (i.e., a preset time interval). Assemble the first processed audio stream within the preset time interval into a continuous audio stream in chronological order to obtain the first assembled audio stream.

[0137] The S603 mixing engine clock-synchronizes the first assembled video stream and the first assembled audio stream and then encapsulates them into the first media stream data.

[0138] Optionally, the mixing engine performs clock synchronization on the first assembled video stream and the first assembled audio stream to ensure that their timestamps are consistent. The mixing engine then encapsulates the clock-synchronized first assembled video stream and the first assembled audio stream into a first media stream data.

[0139] In this embodiment, the mixing engine creates a first video compositer and a first audio mixer in the first media stream pipeline. The first video compositer assembles a first processed video stream at a preset time interval to obtain a first assembled video stream, and the first audio mixer assembles a first processed audio stream at a preset time interval to obtain a first assembled audio stream. The mixing engine clock-synchronizes the first assembled video stream and the first assembled audio stream and then encapsulates them into first media stream data. The video compositer and audio mixer created in the media stream pipeline improve the continuity and smoothness of a single audio and video stream.

[0140] Figure 10 A flowchart illustrating the multi-channel audio and video stream mixing method provided in the embodiments of this application. Figure 7 ,like Figure 10 As shown, in step S103 above, the processed audio and video stream data are mixed based on the integrated unit in each media stream pipeline and the mixing engine to generate mixed audio and video stream data, and the mixed audio and video stream data is sent to the SFU and each terminal device respectively, including:

[0141] S701. The media switches in each media stream pipeline detect the transmission continuity and data integrity of each media stream data respectively, and obtain the detection status value of each media stream data.

[0142] Optionally, continue to refer to Figure 3 Media switches in each media streaming pipeline The transmission continuity and data integrity of each media stream are checked, and the detection status value of each media stream is determined based on the detection results. The detection status value includes a first detection status value, a second detection status value, and a third detection status value.

[0143] For example, the first detection state value, the second detection state value, and the third detection state value can be respectively Figure 3The diagram shows green, yellow, and red status values. A green status value indicates that the media stream data transmission is continuous and complete, and can be transmitted normally to the integration unit. A yellow status value indicates that the media stream data has packet loss or frame loss alarms, requiring packet reassembly. After the media stream data is reassembled, the status value updates to green, and it can be transmitted normally to the integration unit. A red status value indicates that the media stream data is abnormal or erroneous, and needs to be retrieved again or repaired.

[0144] S702, The integrated unit receives the media stream data to be mixed from each media stream with a detection status value of the first detection status value, performs clock synchronization and merging processing on each media stream data to be mixed, generates mixed audio and video stream data, and sends the mixed audio and video stream data to the media output device in the mixing engine.

[0145] Optionally, the integration unit monitors the detection status value of the media switch in each media stream pipeline in real time, and only receives the media stream data to be mixed when the detection status value is the first detection status value (i.e., the green status value). If the detection status value of the media switch is the second or third detection status value (i.e., the yellow or red status value), the integration unit notifies the corresponding media stream pipeline to report an error and repair it via a message, prompting the corresponding media stream pipeline to re-pull the stream or repair the stream data until the detection status value of the media switch changes back to the first detection status value.

[0146] The integration unit synchronizes the clock of each received media stream to be mixed, ensuring that the timestamps of each media stream to be mixed from multiple terminal devices are accurately aligned. The integration unit then merges each media stream to be mixed using a thread within the integration unit to obtain complete mixed audio and video stream data, and sends the mixed audio and video stream data to the media output in the mixing engine.

[0147] S703: The media output device sends the mixed audio and video stream data to the SFU and each terminal device respectively.

[0148] Optionally, the media output device is used for distributing the mixed audio and video stream data. The media output device sends the mixed audio and video stream data to the SFU and each terminal device respectively, so that the SFU can record and store the mixed audio and video stream data, and each terminal device can render the mixed audio and video stream data locally.

[0149] In this embodiment, media switches in each media stream pipeline detect the transmission continuity and data integrity of each media stream data to obtain a detection status value for each media stream data. The integration unit receives each media stream data to be mixed with a detection status value of the first detection status value, performs clock synchronization and merging processing on each media stream data to be mixed, generates mixed audio and video stream data, and sends the mixed audio and video stream data to the media output in the mixing engine. The media output sends the mixed audio and video stream data to the SFU and each terminal device respectively. Before mixing by the integration unit, the transmission continuity and data integrity of each media stream data are detected to ensure the quality and stability of the mixed data and ensure the data quality of the mixed audio and video stream data.

[0150] As an optional implementation, the SFU has an external memory.

[0151] Optionally, refer to Figure 1 The external memory of the SFU is used to store the mixed audio and video stream data recorded by the SFU.

[0152] As an optional implementation, in step S104 above, the SFU records and stores the mixed audio and video stream data, including:

[0153] SFU records the mixed audio and video stream data, obtains the recorded mixed audio and video stream data, and saves the recorded mixed audio and video stream data in the memory.

[0154] Optionally, SFU pulls mixed audio and video stream data from the media output of the mixing engine and records the pulled mixed audio and video stream data in real time to obtain recorded mixed audio and video stream data.

[0155] SFU streams the recorded mixed audio and video data to an external storage device, eliminating the need for third-party cloud recording, thus reducing the cost of mixed recording and increasing its flexibility.

[0156] In this embodiment, the SFU is externally connected to a memory. The SFU records the mixed audio and video stream data, obtains the recorded mixed audio and video stream data, and saves the recorded mixed audio and video stream data in the memory. This eliminates the need for third-party cloud recording, reducing the cost of mixed recording and improving its flexibility.

[0157] This application provides a multi-channel audio and video stream mixing system, which includes an SFU, a mixing engine, and multiple terminal devices. The SFU and the mixing engine are both communicatively connected to each terminal device, and the SFU is communicatively connected to the mixing engine.

[0158] The multi-channel audio and video stream mixing system is used to perform the steps of the multi-channel audio and video stream mixing method described in the foregoing embodiments.

[0159] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0160] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0161] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for mixing multiple audio and video streams, characterized in that, An application is made in a multi-channel audio and video stream mixing system, the multi-channel audio and video stream mixing system including a selective forwarding unit (SFU), a mixing engine, and multiple terminal devices; the method includes: SFU acquires the audio and video stream data of each terminal device and the status parameters of each terminal device, and determines the processed audio and video stream data corresponding to each terminal device based on the status parameters of each terminal device and the audio and video stream data. The SFU generates multiple push messages based on the processed audio and video stream data corresponding to each terminal device, and sends each push message to the mixing engine. Each push message encapsulates metadata of the processed audio and video stream data corresponding to the terminal device. The metadata includes fields such as the identifier of the terminal device and the resolution and / or frame rate of the video stream in the processed audio and video stream data corresponding to the terminal device. Before pulling data, the mixing engine determines whether to pull the processed audio and video stream data corresponding to the terminal device to which the push message belongs from the SFU and to create a media stream pipeline based on the resolution and / or frame rate in the push message. The mixing engine creates an independent media stream pipeline for each terminal device based on each push message. It pulls the processed audio and video stream data corresponding to each terminal device from the SFU and inputs it into the media stream pipeline. Based on each media stream pipeline and the integration unit in the mixing engine, it mixes the processed audio and video stream data to generate mixed audio and video stream data, and sends the mixed audio and video stream data to the SFU and each terminal device respectively. The media stream pipeline is equipped with a stream filter, a video compositer, an audio mixer, and a media switch. The media stream pipeline is used for the transmission and secondary filtering of the processed audio and video stream data corresponding to the terminal devices. The stream filter is used for audio and video separation of single-channel mixed audio and video streams and for filtering out media impurities. The media switch is used to detect the transmission continuity and data integrity of each media stream data. The SFU records and stores the mixed audio and video stream data, and each of the terminal devices renders the mixed audio and video stream data.

2. The method according to claim 1, characterized in that, The step of determining the processed audio and video stream data corresponding to each terminal device based on the status parameters of each terminal device and the audio and video stream data includes: The SFU performs transcoding and single-channel mixing processing on the audio and video stream data of each terminal device according to the status parameters of each terminal device, and determines the processed audio and video stream data corresponding to each terminal device.

3. The method according to claim 2, characterized in that, The SFU performs transcoding and single-channel mixing processing on the audio and video stream data of each terminal device according to the status parameters of each terminal device, and determines the processed audio and video stream data corresponding to each terminal device, including: The SFU decodes the audio and video stream data of the first terminal device to obtain the first decoded video stream data and the first decoded audio stream data, wherein the first terminal device is any terminal device in the multi-channel audio and video stream mixing system. The SFU determines whether to adjust the resolution and / or frame rate of the first decoded video stream data based on the status parameters of the first terminal device. If so, the SFU will reduce the resolution and / or frame rate of the first decoded video stream data to a preset resolution and / or frame rate to obtain the first adjusted video stream data. After clock synchronization of the first adjusted video stream data and the first decoded audio stream data, the SFU will encapsulate them into the first processed audio and video stream data, wherein the first processed audio and video stream data is the processed audio and video stream data corresponding to the first terminal device. Otherwise, the SFU performs clock synchronization on the first decoded video stream data and the first decoded audio stream data, and then encapsulates them into the first processed audio and video stream data.

4. The method according to claim 1, characterized in that, The SFU generates multiple push messages based on the processed audio and video stream data corresponding to each terminal device, including: The SFU generates metadata for the processed audio and video stream data corresponding to each terminal device based on the identifier of each terminal device and the resolution and / or frame rate of the video stream in the processed audio and video stream data corresponding to each terminal device. The SFU generates each push message based on the metadata of the processed audio and video stream data corresponding to each terminal device.

5. The method according to claim 3, characterized in that, The mixing engine creates an independent media stream pipeline for each terminal device based on each push message, pulls the processed audio and video stream data corresponding to each terminal device from the SFU and inputs it into the media stream pipeline, including: The mixing engine determines, based on the first push message, whether the resolution and / or frame rate of the video stream in the first processed audio and video stream data is greater than the preset resolution and / or preset frame rate. If so, the mixing engine creates a first media stream pipeline corresponding to the first terminal device according to the first push message, and pulls the first processed audio and video stream data from the SFU and inputs it into the first media stream pipeline; Otherwise, the mixing engine discards the first push message.

6. The method according to claim 5, characterized in that, The mixing engine creates a first media stream pipeline corresponding to the first terminal device based on the first push message, and pulls the first processed audio and video stream data from the SFU and inputs it into the first media stream pipeline, including: The mixing engine creates a first media stream pipeline corresponding to the first terminal device based on the identifier of the first terminal device in the first push message; The mixing engine generates a first stream filter in the first media stream pipeline, and uses a coroutine mode to cyclically pull the first processed audio and video stream data from the SFU and input it into the first media stream pipeline. The first stream filter separates the first processed audio stream and the first processed video stream from the first processed audio and video stream data.

7. The method according to claim 6, characterized in that, Before mixing the processed audio and video stream data based on the integrated unit in each media stream pipeline and the mixing engine, the process includes: The mixing engine creates a first video compositer and a first audio mixer in the first media stream pipeline; The first video compositer assembles the first processed video streams at a preset time interval to obtain the first assembled video stream, and the first audio mixer assembles the first processed audio streams at a preset time interval to obtain the first assembled audio stream. The mixing engine clock-synchronizes the first assembled video stream and the first assembled audio stream and then encapsulates them into first media stream data.

8. The method according to claim 7, characterized in that, The process of mixing processed audio and video streams based on the media stream pipelines and the integrated unit in the mixing engine to generate mixed audio and video stream data, and then sending the mixed audio and video stream data to the SFU and each of the terminal devices, includes: The media switches in each media stream pipeline detect the transmission continuity and data integrity of each media stream data to obtain the detection status value of each media stream data; The integration unit receives each streaming media stream data to be mixed with a detection status value of the first detection status value, performs clock synchronization and merging processing on each streaming media stream data to be mixed, generates the mixed audio and video stream data, and sends the mixed audio and video stream data to the media output device in the mixing engine; The media output device sends the mixed audio and video stream data to the SFU and each of the terminal devices respectively.

9. The method according to claim 1, characterized in that, The SFU is externally connected to a memory. The SFU records and stores the mixed audio and video stream data, including: The SFU records the mixed audio and video stream data to obtain recorded mixed audio and video stream data, and saves the recorded mixed audio and video stream data in the memory.

10. A multi-channel audio and video stream mixing system, characterized in that, The multi-channel audio and video stream mixing system includes a selective forwarding unit (SFU), a mixing engine, and multiple terminal devices. The SFU and the mixing engine are both communicatively connected to each of the terminal devices, and the SFU is communicatively connected to the mixing engine. The multi-channel audio and video stream mixing system is used to perform the steps of the multi-channel audio and video stream mixing method according to any one of claims 1-9.

Images