Network audio and video signal multicast output method for reducing time delay by reducing network bandwidth

Abstract

The invention relates to a network audio and video signal multicast output method for reducing time delay by reducing network bandwidth, which comprises the following steps of: when a broadband code rate in a multicast network meets a preset condition, triggering an audio and video compression mechanism; the mechanism comprises the following steps: separating a video stream in an input audio and video signal, and carrying out foreground and background segmentation processing on a video frame in the video stream; compressing the foreground motion area by adopting a dynamic video coding algorithm to obtain a foreground compressed stream; compressing the static background image by adopting a static image coding algorithm to obtain a background compressed image; packaging the foreground compressed stream and the background compressed image into a composite video data packet; and distributing the composite video data packet to a client through a multicast protocol. According to the method, the compression strategy can be adaptively adjusted based on the real-time network state, and on the premise of ensuring the basic audio and video quality, the data bandwidth occupation in the network audio and video signal multicast transmission process is effectively reduced, and the end-to-end transmission delay is remarkably reduced.

Description

Technical Field

[0001] This invention relates to the field of multicast transmission technology, and more specifically to a method for multicast output of network audio and video signals that reduces latency by lowering network bandwidth. Background Technology

[0002] With the rapid development of internet technology and the continuous enrichment of application scenarios, the transmission of network audio and video signals has become an indispensable technical support for information exchange, entertainment interaction, and remote collaboration. Especially in scenarios such as education, conferences, live streaming, and remote monitoring, multicast transmission technology for network audio and video signals has been widely used because it can efficiently send the same data to multiple receivers, significantly reducing server load and network bandwidth consumption.

[0003] However, the transmission quality of network audio and video signals, especially end-to-end transmission latency, has always been a key factor affecting user experience. Excessive latency can lead to problems such as audio and video desynchronization, sluggish interactive response, and screen stuttering, and in severe cases, it can even render real-time interactive applications (such as video conferencing and online games) unusable.

[0004] Existing multicast transmission schemes for network audio and video signals typically employ high coding bitrates or add redundant data (such as forward error correction codes) to cope with network jitter and packet loss in order to ensure a certain transmission quality in complex and variable network environments. While these measures improve transmission robustness to some extent, they also lead to higher network bandwidth consumption. When network bandwidth is limited or congestion occurs, high-bandwidth data streams are more likely to cause queuing delays at routing nodes, and the probability of packet loss increases accordingly. Lost packets may need to be retransmitted (although multicast generally does not guarantee reliable transmission, the receiver introduces additional overhead in processing lost data), or the receiver may need to perform complex error masking processing, all of which further exacerbate end-to-end transmission latency.

[0005] Furthermore, while some existing solutions employ audio and video compression coding techniques to reduce the amount of raw data, their compression strategies are often fixed or fail to dynamically adjust according to real-time network conditions. This means that the solutions may not fully utilize available bandwidth to improve quality when network conditions are good, while failing to reduce data volume in time to adapt to bandwidth limitations when network conditions deteriorate, resulting in unsatisfactory latency control.

[0006] In conclusion, how to solve the above-mentioned technical problems is an urgent issue that needs to be addressed by those skilled in the art. Summary of the Invention

[0007] In view of this, the present invention provides a method for reducing latency in network audio and video signal multicast output by reducing network bandwidth. The method aims to adaptively adjust the compression strategy based on the real-time network status, thereby effectively reducing the data bandwidth occupation during network audio and video signal multicast transmission while ensuring basic audio and video quality, thus significantly reducing end-to-end transmission latency.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: A method for reducing latency in network audio and video signal multicast output by lowering network bandwidth includes: Real-time monitoring of the bandwidth bitrate in the multicast network; when preset conditions are met, an audio / video compression mechanism is triggered; the audio / video compression mechanism includes: Separate the video stream from the input audio and video signals, and perform foreground and background segmentation processing on the video frames in the video stream; The moving foreground region is compressed using a dynamic video coding algorithm to obtain a compressed foreground stream; the static background image is compressed using a static image coding algorithm to obtain a compressed background image. The foreground compressed stream and the background compressed image are encapsulated into a composite video data packet; the composite video data packet is then distributed to the client via a multicast protocol.

[0009] Preferably, when preset conditions are met, the audio and video compression mechanism is triggered, including: When the current bandwidth bitrate is detected to drop to the first preset threshold, the video compression mechanism is triggered; Alternatively, when the bit rate reduction between the current broadband bit rate and the broadband bit rate at the previous detection time reaches a second preset threshold, the video compression mechanism is triggered.

[0010] Preferably, the foreground and background segmentation processing of video frames in the video stream includes: The video processing unit is divided into N consecutive video frames. The latest background image is retrieved as a reference background based on time priority, where the latest image is the background image that was updated most recently; The foreground region of each frame is extracted based on a motion analysis algorithm, and the coordinates of the minimum bounding rectangle corresponding to the foreground region are calculated. The foreground YUV stream is generated by cropping the foreground region based on the coordinates of the minimum bounding rectangle, and the corresponding region of the reference background is cropped simultaneously.

[0011] Preferably, the dynamic video encoding algorithm is the H.264 or H.265 standard, and the compression process includes: The foreground YUV stream and the corresponding reference background are segmented and packaged into continuous foreground video units; Add header data to each unit, including the start timestamp, frame rate, coordinates, and media stream identifier.

[0012] Preferably, the still image encoding algorithm is the JPEG standard, and the compression process includes: Background images are classified into category A or category B based on the degree of background change; category A is updated in a low-frequency manner, and category B is updated in a high-frequency manner. Class B background images are compressed and encrypted in real time for transmission, while Class A background images are transmitted in batches during low-bandwidth periods.

[0013] Preferably, the background image is classified into category A or category B based on the degree of background change, including: Calculate the pixel difference between the current frame and the reference background, and segment the background image based on the difference value.

[0014] Preferably, network bandwidth, latency, and image quality loss are monitored in real time using formulas. Optimize the compression parameters in the dynamic video encoding algorithm and the static image encoding algorithm; where D represents latency and B represents network bandwidth. Indicates image quality loss. , and These are weighting coefficients, and the priority of each parameter is determined based on the needs of the application scenario.

[0015] Preferably, the composite video data packets are distributed to the client via a multicast protocol, including: Assign a unique media stream identifier to each client; Dynamically adjust the transmission priority of the foreground stream and background image based on the client's network status: When the bandwidth is below the third preset threshold, only the foreground compressed stream is transmitted, and the client reuses the locally cached background image; When bandwidth is restored, a new, compressed background image will be resent.

[0016] Preferably, it further includes client-side synthesis processing, including: Decrypt and decode the foreground compressed stream and the background compressed image; The foreground frame is superimposed onto the corresponding coordinate region of the background image to generate a complete video frame; When no foreground stream is received, the latest background image is continuously played. The latest background image is the background image that was most recently and validly received by the client and has not expired. Its update rule is as follows: When a Class B background is received, the old background is immediately overwritten. When a Class A background is received, the old background is overwritten only when the timestamp is updated.

[0017] Preferably, real-time monitoring of the broadband bitrate in a multicast network includes: Real-time collection of end-to-end latency, packet loss rate, jitter, and available bandwidth; The real-time bitrate is determined using the following formula:

[0018] In the formula, , , These are adaptive coefficients.

[0019] As can be seen from the above technical solutions, the present invention discloses a method for multicast output of network audio and video signals by reducing network bandwidth to reduce latency, which significantly improves the dynamic adaptability and coding efficiency of audio and video transmission, achieves more stable and higher quality video transmission under complex network conditions, and reduces bandwidth usage and latency.

[0020] Compared with existing technologies, the beneficial effects include: By separating the video stream into dynamic foreground and static background, and employing different compression strategies for each, encoding efficiency and bandwidth utilization are significantly improved. Real-time bandwidth monitoring and dynamic adjustment of transmitted content effectively reduce transmission pressure in low-bandwidth environments, ensuring smooth transmission of core content. Independent processing and transmission of the foreground and background, combined with intelligent client-side synthesis, not only enhances video quality but also strengthens system robustness and user experience, especially in scenarios with significant network fluctuations, providing more stable and reliable audio and video services.

[0021] By optimizing the compression parameters of dynamic video coding algorithms and static image coding algorithms, it is possible to reduce the amount of data while ensuring video quality, effectively cope with network fluctuations, ensure transmission stability, and improve video transmission quality. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0023] Figure 1 This is a flowchart of the audio and video compression mechanism of the present invention. Detailed Implementation

[0024] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0025] To address the shortcomings of existing technologies in terms of transmission latency, this application provides a novel technical solution to optimize the multicast output of network audio and video signals, adapting to various application scenarios with higher requirements for low latency. Specifically, this embodiment discloses a method for multicast output of network audio and video signals that reduces latency by lowering network bandwidth, significantly improving the system's adaptability in complex network environments. Ultimately, this provides users with a smoother and clearer video experience, while also making multicast output more efficient in terms of bandwidth usage and resource consumption.

[0026] In one embodiment, the multicast output method of this application includes: First, monitor the broadband bitrate in the multicast network in real time, and trigger the audio and video compression mechanism when preset conditions are met; in one embodiment, meeting the preset conditions includes: When the current bandwidth bitrate is detected to drop to the first preset threshold, the video compression mechanism is triggered; Alternatively, when the bit rate reduction between the current broadband bit rate and the broadband bit rate at the previous detection time reaches a second preset threshold, the video compression mechanism is triggered.

[0027] This application triggers a video compression mechanism by monitoring the broadband bitrate, enabling it to respond to broadband changes in real time and reduce latency.

[0028] In this embodiment, real-time monitoring of the broadband bitrate in the multicast network includes: Real-time collection of end-to-end latency, packet loss rate, jitter, and available bandwidth; The real-time bitrate is determined using the following formula:

[0029] In the formula, , , These are adaptive coefficients.

[0030] Among them, end-to-end delay is the total bidirectional time from the sending end to the receiving end's acknowledgment return, packet loss rate is the ratio of the difference between the number of sent packets and the number of received packets, jitter is the degree of fluctuation in the time interval between data packet arrivals, and available bandwidth is the maximum sustainable throughput of the network path.

[0031] Second, the audio and video compression mechanism of this invention, such as Figure 1 Specifically, it includes: S1. Separate the video stream from the input audio and video signal, and perform foreground and background segmentation processing on the video frames in the video stream; S2. The foreground moving region is compressed using a dynamic video coding algorithm to obtain the foreground compressed stream; the static background image is compressed using a static image coding algorithm to obtain the background compressed image. S3. Encapsulate the foreground compressed stream and the background compressed image into a composite video data packet; distribute the composite video data packet to the client via a multicast protocol.

[0032] In this embodiment, step S1 involves performing foreground and background segmentation processing on the video frames in the video stream, including: The latest background image is obtained as a reference background based on time priority, where the latest image is the background image that was updated most recently; specifically, it is a background image of category B or category A (refer to the background image compression process).

[0033] The motion analysis algorithm is used to extract the foreground region of each frame and calculate the coordinates of the minimum bounding rectangle corresponding to the foreground region; the motion analysis algorithm is a set of multimodal algorithms used to detect pixel changes between video frames and separate moving foregrounds; Then, based on the coordinates of the minimum bounding rectangle, the foreground region is cropped to generate a foreground YUV stream, and the corresponding region of the reference background is cropped simultaneously.

[0034] In one embodiment, in step S2, the dynamic video coding algorithm is preferably the H.264 or H.265 standard, and the compression process includes: The foreground YUV stream and the corresponding reference background are segmented and packaged into continuous foreground video units; Each unit is then given a header containing start timestamp, frame rate, coordinates, and media stream identifier to generate a foreground compressed stream. Furthermore, the still image encoding algorithm is preferably based on the JPEG standard, and the compression process includes: Background images are classified into category A or category B based on the degree of background change; category A is updated in a low-frequency manner, and category B is updated in a high-frequency manner. Class B background images are compressed and encrypted in real time for transmission, while Class A background images are transmitted in batches during low-bandwidth periods.

[0035] Based on the degree of background variation, background images are classified into category A or category B, including: Calculate the pixel difference between the current frame and the reference background, and segment the background image based on the difference value.

[0036] As a preferred option, a pixel difference threshold can be set. When the pixel difference value is greater than the pixel difference threshold, it is set as a Class A background, and otherwise it is set as a Class B background.

[0037] In one embodiment, step S3 includes: encapsulating the foreground compressed stream and the background compressed image into a composite video data packet, and distributing the composite video data packet to the client via a multicast protocol. When distributing clients, each client is assigned a unique media stream identifier; Then, the transmission priority of the foreground stream and background image is dynamically adjusted based on the client's network status: Specifically, when the bandwidth is lower than the third preset threshold, only the foreground compressed stream is transmitted, and the client reuses the locally cached background image; when the bandwidth is restored, the updated background compressed image is retransmitted.

[0038] Furthermore, this embodiment also includes client-side synthesis processing, including: Decrypt and decode the foreground compressed stream and the background compressed image; The foreground frame is superimposed onto the corresponding coordinate region of the background image to generate a complete video frame; When no foreground stream is received, the latest background image is continuously played. The latest background image is the background image that was most recently and validly received by the client and has not expired. Its update rule is as follows: When a Class B background is received, the old background is immediately overwritten. When a Class A background is received, the old background is overwritten only when the timestamp is updated.

[0039] To further optimize the above technical solution, this application preferably involves real-time monitoring of network bandwidth, latency, and image quality loss, and using formulas... Optimize the compression parameters in the dynamic video encoding algorithm and the static image encoding algorithm; where D represents latency and B represents network bandwidth. Indicates image quality loss. , and These are the weighting coefficients.

[0040] In this embodiment, the priority of each parameter is determined according to the requirements of the application scenario, such as: For cloud gaming: Increase the weight of α, but avoid excessively low image quality (γ cannot be zero). For example, use edge computing to reduce latency.

[0041] Video-on-demand: Increase the full redness of γ, but control the buffering delay (α moderately) to avoid user waiting. For example, adaptive streaming (DASH / HLS).

[0042] When bandwidth is limited: increase the weight of β, use compression techniques (such as B-frame reduction), or dynamically adjust the resolution.

[0043] By optimizing the compression parameters of dynamic video coding algorithms and static image coding algorithms, it is possible to reduce the amount of data while ensuring video quality, effectively cope with network fluctuations, ensure transmission stability, and improve video transmission quality.

[0044] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0045] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for multicast output of network audio and video signals to reduce latency by lowering network bandwidth, characterized in that, include: Real-time monitoring of the broadband bitrate in the multicast network; when preset conditions are met, triggering the audio and video compression mechanism. The audio and video compression mechanism includes: Separate the video stream from the input audio and video signals, and perform foreground and background segmentation processing on the video frames in the video stream; The moving foreground region is compressed using a dynamic video coding algorithm to obtain a compressed foreground stream; the static background image is compressed using a static image coding algorithm to obtain a compressed background image. The foreground compressed stream and the background compressed image are encapsulated into a composite video data packet; the composite video data packet is then distributed to the client via a multicast protocol.

2. The network audio and video signal multicast output method according to claim 1, characterized in that, When preset conditions are met, the audio and video compression mechanism is triggered, including: When the current bandwidth bitrate is detected to drop to the first preset threshold, the video compression mechanism is triggered; Alternatively, when the bit rate reduction between the current broadband bit rate and the broadband bit rate at the previous detection time reaches a second preset threshold, the video compression mechanism is triggered.

3. The network audio and video signal multicast output method according to claim 1, characterized in that, Perform foreground and background segmentation processing on video frames in the video stream, including: The video processing unit is divided into N consecutive video frames. The latest background image is retrieved as a reference background based on time priority, where the latest image is the background image that was updated most recently; The foreground region of each frame is extracted based on a motion analysis algorithm, and the coordinates of the minimum bounding rectangle corresponding to the foreground region are calculated. The foreground YUV stream is generated by cropping the foreground region based on the coordinates of the minimum bounding rectangle, and the corresponding region of the reference background is cropped simultaneously.

4. The network audio and video signal multicast output method according to claim 3, characterized in that, The dynamic video encoding algorithm is the H.264 or H.265 standard, and the compression process includes: The foreground YUV stream and the corresponding reference background are segmented and packaged into continuous foreground video units; Add header data to each unit, including the start timestamp, frame rate, coordinates, and media stream identifier.

5. The network audio and video signal multicast output method according to claim 3, characterized in that, The static image encoding algorithm is based on the JPEG standard, and the compression process includes: Background images are classified into category A or category B based on the degree of background change; category A consists of low-frequency updated background images, and category B consists of high-frequency updated background images. Class B background images are compressed and encrypted in real time before transmission, while Class A background images are transmitted in batches when the bandwidth is below the preset limit.

6. The network audio and video signal multicast output method according to claim 5, characterized in that, Background images are classified into category A or category B based on the degree of background variation, including: Calculate the pixel difference between the current frame and the reference background, and segment the background image based on the difference value.

7. The network audio and video signal multicast output method according to claim 1, characterized in that, Real-time monitoring of network bandwidth, latency, and image quality loss, using formulas Optimize the compression parameters in the dynamic video encoding algorithm and the static image encoding algorithm; where D represents latency and B represents network bandwidth. Indicates image quality loss. , and These are the weighting coefficients.

8. The network audio and video signal multicast output method according to claim 1, characterized in that, Multicast protocols are used to distribute composite video data packets to clients, including: Assign a unique media stream identifier to each client; Dynamically adjust the transmission priority of the foreground stream and background image based on the client's network status: When the bandwidth is below the third preset threshold, only the foreground compressed stream is transmitted, and the client reuses the locally cached background image; When bandwidth is restored, a new, compressed background image will be resent.

9. The network audio and video signal multicast output method according to claim 5, characterized in that, It also includes client-side compositing processing, including: Decrypt and decode the foreground compressed stream and the background compressed image; The foreground frame is superimposed onto the corresponding coordinate region of the background image to generate a complete video frame; When no foreground stream is received, the latest background image is continuously played. The latest background image is the background image that was most recently and validly received by the client and has not expired. Its update rule is as follows: When a Class B background is received, the old background is immediately overwritten. When a Class A background is received, the old background is overwritten only when the timestamp is updated.

10. The network audio and video signal multicast output method according to claim 1, characterized in that, Real-time monitoring of broadband bitrate in multicast networks includes: Real-time collection of end-to-end latency, packet loss rate, jitter, and available bandwidth; The real-time bitrate is determined using the following formula: In the formula, , , These are adaptive coefficients.

Images