Double-line channel audio and video transmission optimization method and system

Abstract

The invention discloses a double-line channel audio and video transmission optimization method and a double-line channel audio and video transmission optimization system. The method relates to the field of audio and video transmission control, and dynamically determines bandwidth requirements and performs resource allocation by analyzing data variation amplitudes of a main stream channel and an auxiliary stream channel. The main stream channel pays attention to content changes of subtitles, labels and the like, and the auxiliary stream channel pays attention to parameters of video playing frame rate, resolution and the like; meanwhile, a classifier combined by a full connection layer and a softmax activation function is adopted to identify dynamic and static contents in the video so as to further optimize bandwidth allocation. The system comprises a server, a client, a main stream channel and an auxiliary stream channel, the server receives audio and video data, packages the audio and video data according to an optimization method and transmits the audio and video data through different channels, and the client receives, decompresses and plays the audio and video data. According to the scheme, an effective bandwidth management strategy and an intelligent scheduling mechanism can be adopted, the audio and video transmission quality and real-time performance are improved, participants can enjoy clearer and smoother audio and video experience, and the stable transmission quality can be kept even in a complex network environment.

Description

Technical Field

[0001] The present invention relates to the technical field of audio and video transmission, and in particular to a dual-line channel audio and video transmission optimization method and system. Background Art

[0002] In the digital age, with the rapid popularization and development of multimedia networks, real-time audio and video data transmission has become an indispensable part of people's lives and work. With the surge in applications such as video conferencing, online education, and real-time audio and video entertainment, higher requirements have been put forward for efficient, stable, and secure audio and video transmission. Real-time transmission of audio and video data in multimedia networks faces a series of technical challenges, including bandwidth limitations, delays, packet loss, codec complexity, and security requirements. However, in the case of limited bandwidth, the transmission of audio and video data may be restricted, which in turn causes problems such as image quality degradation and audio distortion. In order to solve the challenge of bandwidth limitations, it is necessary to adopt effective bandwidth management strategies and intelligent scheduling mechanisms to improve bandwidth utilization and the stability and quality of real-time transmission of audio and video data. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a dual-line channel audio and video transmission optimization method and system, which adopts an effective bandwidth management strategy and an intelligent scheduling mechanism to improve bandwidth utilization and the stability and quality of real-time transmission of audio and video data.

[0004] The objective of the present invention is achieved through the following technical solutions: Receive a first data stream of a mainstream channel and a second data stream of a secondary stream channel; Automatically identify a first change amplitude of a first data stream of a mainstream channel, and determine a first bandwidth requirement for audio and video transmission of a two-line channel according to the first change amplitude; Analyze a second change amplitude of the second data stream of the auxiliary stream channel, and determine a second bandwidth requirement for dual-line channel audio and video transmission according to the second change amplitude; Dynamically determine the total bandwidth requirements of the main stream channel and auxiliary stream channel and allocate bandwidth resources; The optimized data packets are transmitted back to the client through the main stream channel and auxiliary stream channel.

[0005] By adopting the above technical solution, the system can automatically identify the change range of the first data stream (such as the frequency of subtitle content update, label change, etc.) in the auxiliary stream channel (such as the data stream for subtitles and tags), and accurately determine the bandwidth required for the auxiliary stream channel (first bandwidth requirement). At the same time, the change range of the second data stream (such as video frame rate, audio bit rate, etc.) in the mainstream channel (such as the data stream for audio and video) is analyzed to determine the bandwidth required for the mainstream channel (second bandwidth requirement). Combining the two, the system can determine the total bandwidth requirement and dynamically adjust the bandwidth resource allocation according to actual needs to ensure that key data (such as dynamic video content) obtains sufficient bandwidth while reducing unnecessary bandwidth waste; secondly, by decomposing multimedia data into different data streams (mainstream channels and auxiliary stream channels) and optimizing them separately, the solution can significantly improve the overall transmission efficiency of multimedia content. The mainstream channel focuses on transmitting high-quality audio and video data, while the auxiliary stream channel focuses on transmitting auxiliary information (such as subtitles, tags). The two complement each other and jointly improve the user experience.

[0006] As a further improvement of the present technical solution, the first variation range includes at least one of the video playback frame rate, resolution, bit rate, audio sampling rate, sampling bit number and number of channels.

[0007] As a further improvement of the technical solution, the second change range includes at least one of the subtitle content update frequency, label change, the time point when the content changes, and the recording time.

[0008] By adopting the above technical solutions, the functions of automatically identifying the change amplitude of the first data stream of the mainstream channel and analyzing the change amplitude of the second data stream of the auxiliary channel are integrated into the system, reducing the need for manual intervention and improving the intelligence and automation level of the system. This not only reduces operating costs, but also improves the real-time and accuracy of data processing; through the analysis of the above parameters, the system can generate detailed data reports to provide data-driven decision support for content creators, platform operators, etc. For example, adjust the content distribution strategy according to data such as video playback frame rate and resolution, optimize the subtitle translation process according to the subtitle update frequency, etc.; for applications that require real-time transmission (such as video conferencing, online live broadcast, etc.), the system can ensure the synchronous transmission of audio and video data with subtitles, tags and other information, reduce delays and packet loss, and improve real-time performance.

[0009] As a further improvement of the technical solution, the dynamically determining bandwidth requirement includes: A classifier that combines a fully connected layer and a softmax activation function is designed, the feature map is input into the classifier, and the probability of each pixel or area belonging to dynamic content or static content is output; by setting a standard threshold, areas with a probability higher than the standard threshold are judged as dynamic content, and areas with a probability lower than the standard threshold are judged as static content.

[0010] As a further improvement of the technical solution, the classifier satisfies the following relationship: Among them: The output of the fully connected layer is z, where zi represents the original prediction value of the i-th category, and the Softmax function converts zi into probability pi.

[0011] By adopting the above technical solutions and designing a classifier that combines a fully connected layer with a softmax activation function, the system can efficiently and accurately classify each pixel or area in the video or image, and distinguish which are dynamic content (such as moving people, vehicles, etc.) and which are static content (such as background, stationary objects, etc.). This classification capability is crucial for fields such as video transmission, storage optimization, and content analysis; secondly, after identifying dynamic and static content, the system can dynamically determine bandwidth requirements based on the characteristics of these contents. Generally, dynamic content requires higher bandwidth to ensure the smoothness and quality of transmission, while static content can appropriately reduce bandwidth requirements while ensuring basic visibility. This intelligent bandwidth allocation strategy helps to effectively reduce network bandwidth consumption and transmission costs without sacrificing user experience.

[0012] As a further improvement of the present technical solution, the feature map includes key information in the image or video frame, such as at least one of edge, texture, and shape.

[0013] By adopting the above technical solutions, since the feature map reduces the dimension and complexity of the input data, the classifier can converge faster when processing the feature map, thereby improving classification efficiency. This is particularly important for application scenarios that process video streams or large-scale image data in real time; secondly, the key information extracted from the feature map provides the classifier with rich classification basis, enabling the classifier to more accurately identify dynamic and static content. This enhanced classification performance helps to improve the performance of the overall system and user experience; thirdly, since the feature map can contain different types of key information (such as edges, textures, shapes, etc.), the classifier can be flexibly applied to different application scenarios; finally, whether it is video surveillance that needs to identify moving objects or an autonomous driving system that needs to analyze dynamic scenes, the requirements can be met by adjusting the feature extraction method and classifier parameters.

[0014] As a further improvement of the technical solution, the total bandwidth requirement also includes adding a certain redundant bandwidth on the basis of .

[0015] By adopting the above technical solutions, the introduction of redundant bandwidth provides the system with a certain buffer space to cope with sudden traffic or network fluctuations. This helps to reduce transmission interruptions or delays caused by insufficient bandwidth, thereby enhancing the stability and reliability of the system. By ensuring that the actual bandwidth demand contains a certain amount of redundancy, the system can better cope with changes in user behavior or network conditions and maintain smooth transmission of audio and video content, subtitles, tags and other information. This helps to improve the user's viewing experience and reduce dissatisfaction caused by problems such as freezes and delays. As the business develops and user needs grow, the system may need to process more data or support higher transmission quality. The introduction of redundant bandwidth reserves expansion space for the system, making it easier for the system to adapt to future changes in demand without large-scale transformation or upgrades. During network transmission, data transmission errors or losses may occur due to various unforeseen reasons (such as network failures, equipment failures, etc.). The existence of redundant bandwidth can alleviate the impact of these problems to a certain extent, improve the system's fault tolerance, and ensure the integrity and accuracy of key data.

[0016] A two-line channel audio and video transmission system, characterized in that the system comprises a server, a client and a communication channel, wherein the communication channel comprises a mainstream channel and an auxiliary stream channel, the server receives from the client an instruction for transmitting the content of audio and video data through the mainstream channel and the auxiliary stream channel, wherein the mainstream channel and the auxiliary stream channel correspond to different types of communication interfaces; the server is used to receive the audio and video data from the client, and to package the information according to the mainstream channel data or the auxiliary stream channel, and then send the packaged data to the client through different communication channels; The client is responsible for receiving the packaged data from the server, and decompressing and restoring the data according to the data type. The decompressed audio and video data can be played on the client.

[0017] By adopting the above technical solutions, the system separates audio and video data into the mainstream channel and the auxiliary channel, and the system can optimize the processing according to the characteristics of different communication interfaces. For example, the mainstream channel may focus on the transmission of high-quality audio and video data, while the auxiliary channel is used to transmit low-latency, high-frequency updated auxiliary information (such as subtitles, bullet screens, control signals, etc.). This separation process reduces data congestion on a single channel, thereby significantly improving the overall transmission efficiency. Secondly, since the system supports data transmission through different types of communication interfaces, it can flexibly configure the mainstream channel and auxiliary channel according to actual scenarios and needs. For example, in an environment with poor network conditions, the bandwidth usage of the auxiliary channel can be reduced to ensure high-quality audio and video transmission of the mainstream channel. In addition, with the advancement of technology and the emergence of new application scenarios, the system can easily add new communication interfaces and channel types to meet future development needs; thirdly, since the system can select appropriate communication interfaces and channels for transmission according to different data types, this helps to reduce unnecessary bandwidth waste and hardware investment. For example, for auxiliary information that does not require high real-time performance, a low-cost communication interface can be selected for transmission, thereby reducing the overall system construction and operation costs; finally, the system is suitable for a variety of audio and video transmission application scenarios, including but not limited to video conferencing, online education, telemedicine, webcasting, etc. By flexibly configuring the mainstream channel and auxiliary channel, the system can meet the diverse needs of different scenarios and provide users with customized audio and video transmission solutions.

[0018] In summary, the present application includes at least one of the following beneficial technical effects: 1) By automatically identifying the change range of data streams in the mainstream channel and auxiliary stream channel (such as subtitle update frequency, video playback frame rate, resolution, etc.), the bandwidth requirements of the two-line channel are dynamically determined and optimized. This dynamic bandwidth allocation strategy can flexibly adjust bandwidth resources according to the characteristics of real-time content, ensure the smoothness and quality of audio and video transmission, and effectively avoid bandwidth waste.

[0019] 2) A classifier based on a combination of a fully connected layer and a softmax activation function is introduced to intelligently classify each pixel or region as dynamic content or static content by analyzing feature maps (containing key information in images or video frames). This intelligent classification technology not only improves the accuracy and efficiency of content recognition, but also provides strong support for subsequent bandwidth allocation and data processing.

[0020] 3) The dual-channel (mainstream channel and auxiliary channel) is used to transmit audio and video data in parallel. Through reasonable division of labor (such as the main channel transmits the main audio and video content, and the auxiliary channel transmits subtitles, tags and other auxiliary information) and collaborative work, efficient transmission and rapid response of audio and video data are achieved. This dual-channel parallel transmission mode not only improves the reliability and stability of data transmission, but also provides users with a richer audio-visual experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of the dual-line channel audio and video transmission process of the present invention.

[0022] Figure 2 It is a schematic diagram of the connection relationship between the client and the server of the present invention. DETAILED DESCRIPTION

[0023] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0024] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0025] Example 1 Please refer to Figure 1 and Figure 2 In a video conference scenario, the server needs to transmit high-definition video streams, real-time subtitles, and tag information to multiple clients. In order to optimize transmission efficiency and improve user experience, dual-line channels (mainstream channels and auxiliary stream channels) are used to transmit audio, video, and auxiliary data. The mainstream channel is used to transmit high-definition video and audio data, while the auxiliary stream channel is responsible for transmitting auxiliary information such as subtitles and tags.

[0026] Implementation steps S100 receives a first data stream of a main stream channel 21 and a second data stream of a secondary stream channel 22; The server 30 receives a high-definition video stream from a video source as the first data stream of the mainstream channel 21, which contains key information such as video playback frame rate (such as 30fps), resolution (such as 1080p) and bit rate (such as 5Mbps).

[0027] S200 automatically identifies a first change amplitude of a first data stream of a mainstream channel 21, and determines a first bandwidth requirement for dual-line channel audio and video transmission according to the first change amplitude; For the first data stream of the mainstream channel 21, the server 30 analyzes changes in the video playback frame rate, resolution and bit rate, specifically including at least one of the following parameters.

[0028] 1) Video playback frame rate: Standard video frame rates are usually 24fps or 30fps, but in high-definition video or game live broadcasts, the frame rate may reach 60fps or higher; assuming a video with a resolution of 1080p (1920x1080) at 30fps, if the bit rate is 5Mbps, then the bandwidth required is roughly 5Mbps. But if the frame rate is increased to 60fps, while the resolution and bit rate remain unchanged, then the bandwidth required will theoretically double, because the number of images that need to be transmitted has doubled. However, the actual bit rate may be adjusted according to the change in frame rate to maintain similar video quality.

[0029] 2) Resolution: The higher the resolution, the more pixels there are in the image, so more bandwidth is needed to transmit this data; 720p (1280x720) video resolution is lower than 1080p (1920x1080), so at the same bit rate, 720p video requires less bandwidth. Assuming the bit rate is 5Mbps, the bandwidth required for 720p video will be lower than that for 1080p video. The specific value depends on the coding efficiency and compression algorithm, but it can generally be considered that the higher the resolution, the greater the bandwidth required.

[0030] 3) Bitrate: Bitrate directly determines the transmission rate of video or audio data. The higher the bitrate, the greater the bandwidth required. A 1080p video at a 2Mbps bitrate requires significantly less bandwidth than a 10Mbps bitrate. The choice of bitrate depends on the video content, the viewer's network conditions, and the desired video quality. When bandwidth is limited, lowering the bitrate can reduce transmission delays and buffering time, but may sacrifice some video quality.

[0031] 4) Audio sampling rate: The higher the sampling rate, the higher the degree of restoration of the audio signal, but the greater the bandwidth required. The audio sampling rate of CD quality is 44.1kHz, while high-fidelity audio (such as DVD-Audio) may use a sampling rate of 96kHz or 192kHz. At the same bit depth and number of channels, audio files with high sampling rates will require more bandwidth to transmit.

[0032] 5) The number of bits per sample determines the dynamic range and accuracy of the audio signal, but it has a relatively small impact on bandwidth because the number of bits per sample is fixed. CD-quality audio typically uses 16 bits per sample, while professional audio productions may use 24 or 32 bits. Although a higher number of bits per sample provides better audio quality, the increase in bandwidth requirements is not significant because the extra bits per sample only increase the amount of data by a tiny fraction.

[0033] 6) Number of channels: The number of audio channels determines the directionality and stereoscopic sense of the sound. Stereo (2-channel) audio provides audio signals in the left and right channels, while surround sound (such as 5.1 channels or 7.1 channels) provides audio signals in multiple directions to enhance the immersiveness and spatial sense of the sound. When making movie or game sound effects, different numbers of channels are selected as needed, and this change also falls into the category of the first range of variation.

[0034] Changes in these parameters will affect the transmission bandwidth requirements and playback quality of audio and video data. Therefore, when designing and optimizing audio and video transmission systems, these changes need to be fully considered.

[0035] S300 analyzes a second change amplitude of the second data stream of the auxiliary stream channel 22, and determines a second bandwidth requirement for dual-line channel audio and video transmission according to the second change amplitude; The server 30 also receives real-time subtitles and tag information as a second data stream of the auxiliary stream channel 22, which includes subtitle content update frequency (such as once per second), tag changes (such as speaker name changes), etc.

[0036] For example, in a remote video conference, the main stream channel is responsible for transmitting high-definition video and audio streams, while the auxiliary stream channel 22 is used to transmit subtitles and conference-related dynamic tags (such as conference topic, speaker name, etc.). As the conference progresses, the subtitles and tags will change, and these changes will affect the bandwidth requirements of the auxiliary stream channel 22.

[0037] Analysis of the change range of the second data stream of auxiliary stream channel 22 1) Subtitle update frequency: During the conference, the server 30 monitors the data stream on the auxiliary stream channel 22 and finds that the subtitle content is updated once a second. This means that new subtitle data needs to be transmitted every second.

[0038] Since subtitles usually contain a small amount of text, the amount of data updated each time is relatively small.

[0039] 2) Label changes: During the meeting, the server 30 also detected that the tag had changed three times. These changes may include changes in the meeting theme, changes in the speaker's name, etc.

[0040] Tags change more sparsely than the continuous updates of subtitles, but each change may contain more information, depending on the content and format of the tag.

[0041] 3) Subtitle content update frequency: During the live broadcast, the teacher may update the subtitles on the screen in real time based on students' questions or discussion hotspots. For example, the teacher updates the subtitles every two minutes to emphasize the key points of the current lecture or answer students' questions. Assume that the subtitles are updated every two minutes, that is, every 120 seconds.

[0042] 4) Time when the content changes and the time of recording: The server records and transmits key points in time when content changes so that viewers can review or jump to sections of interest later.

[0043] For example, when the live broadcast is 25 minutes in, the teacher starts to explain a key case, and this time point is recorded and marked as "Case explanation starts". At 50 minutes, the case explanation ends, and this time point is also recorded accordingly.

[0044] Determine the bandwidth requirement of auxiliary stream channel 22 1) Based on subtitle update frequency: Assume that the average amount of data to be transmitted for each subtitle update is 1KB (this is just an example value and the actual situation may be different).

[0045] Considering that it is updated once per second, the bandwidth requirement generated by the subtitle update is 1KB / s=0.0009765625Mbps (but usually we will round up or determine it based on more precise calculations, here is a simplified explanation).

[0046] 2) Based on label changes: Assume that the amount of data that needs to be transmitted for each label change is 10KB (this is also an example value). Since only three changes occurred during the meeting, we can evenly distribute it to the entire meeting duration. If the meeting lasts for 1 hour (3600 seconds), the average time taken for each change is 3600 seconds / 3=1200 seconds. Therefore, the average bandwidth requirement generated by label changes is (10KB*3) / 3600 seconds≈0.00833Mbps (this is also simplified here).

[0047] 3) Comprehensive bandwidth requirements: The bandwidth requirements of subtitles and tags are added together to get the total bandwidth required by the auxiliary stream channel 22. However, considering that the data volume of subtitles and tags is relatively small, and the actual bandwidth requirements are also affected by various factors such as coding efficiency and network conditions, we directly give an estimated value here, and estimate that the bandwidth required by the auxiliary stream channel 22 is 0.5Mbps. This value is obtained based on the above analysis and taking into account a certain margin.

[0048] In this embodiment, the server 30 estimates the bandwidth required for the channel by analyzing the update frequency of the subtitles and the change of the labels on the auxiliary stream channel 22, providing a basic analysis framework and idea.

[0049] S400 dynamically determines the total bandwidth requirement of the main stream channel 21 and the auxiliary stream channel 22 and performs bandwidth resource allocation; This solution builds a neural network module that integrates a fully connected layer and a Softmax activation function, which is designed to take the preprocessed feature map as input and convert it into a probabilistic output to determine whether each pixel or region belongs to dynamic content. In this process, the fully connected layer plays a core role, which is responsible for mapping the input features into a series of unnormalized prediction values ​​(i.e., logits), which correspond to different categories (specifically dynamic and possible other categories in this scenario); This combination of fully connected layers + Softmax is common in neural network architectures that handle multi-classification problems, such as image or text classification tasks. The fully connected layer is responsible for the integration and extraction of features to generate preliminary classification clues, while the Softmax layer converts these clues into intuitive probabilistic forms to facilitate subsequent classification decisions and result interpretation. The entire process constitutes a complete mapping from features to probabilistic outputs, and is a key component in many complex classification tasks.

[0050] S500 transmits the optimized data packet back to the client through the main stream channel 21 and the auxiliary stream channel 22.

[0051] The server 30 optimizes the data packets of the mainstream channel 21 and the auxiliary channel 22 according to the allocated bandwidth resources, such as adjusting the data packet size, compression algorithm, etc., to reduce transmission delay and improve transmission efficiency. The optimized data packets are respectively transmitted back to the client through the mainstream channel 21 and the auxiliary channel 22. After receiving the data packets, the client decompresses and restores them according to the data type, and finally plays the high-definition video on the user interface and displays real-time subtitles and tag information.

[0052] By automatically identifying the change range of the first data stream (such as the frequency of subtitle content updates, label changes, etc.) in the auxiliary stream channel 22 (such as the data stream for subtitles and tags), the system can accurately determine the bandwidth required by the auxiliary stream channel 22 (the first bandwidth requirement). At the same time, the change range of the second data stream (such as the video frame rate, audio bit rate, etc.) in the mainstream channel 21 (such as the data stream for audio and video) is analyzed to determine the bandwidth required by the mainstream channel 21 (the second bandwidth requirement). Combining the two, the system can determine the total bandwidth requirement and dynamically adjust the bandwidth resource allocation according to actual needs to ensure that key data (such as dynamic video content) obtains sufficient bandwidth while reducing unnecessary bandwidth waste; secondly, by decomposing multimedia data into different data streams (mainstream channel 21 and auxiliary stream channel 22) and optimizing them separately, the solution can significantly improve the overall transmission efficiency of multimedia content. The mainstream channel 21 focuses on transmitting high-quality audio and video data, while the auxiliary stream channel 22 focuses on transmitting auxiliary information (such as subtitles, tags). The two complement each other and jointly improve the user experience.

[0053] Example 2 In real-time audio and video transmission systems, such as video conferencing, remote monitoring, or live streaming services, the bandwidth requirements for dynamic content (such as fast-moving objects, changing scenes) and static content (such as backgrounds, static images) are significantly different. In order to more effectively utilize limited network resources, we designed a deep learning-based dynamic content recognition system that can automatically distinguish between dynamic and static areas in the video and dynamically adjust the transmission bandwidth for different areas accordingly.

[0054] System Architecture 1) Video capture and preprocessing The system first captures the original audio and video data. For the video part, necessary preprocessing is performed, such as scaling and denoising, to make it suitable for subsequent processing.

[0055] The video frames are decomposed into a series of feature maps, which can be original pixel values, color histograms, edge detection results, etc., or high-level features extracted by convolutional neural networks (CNN).

[0056] 2) Dynamic content recognition module Fully connected layer and Softmax activation function: This scheme designs a neural network module with a fully connected layer. The input of this module is the preprocessed feature map. After the fully connected layer, the Softmax activation function is applied to convert the original prediction value (logits) of each pixel or region into a probability value. Specifically, for the i-th pixel or region, the probability of it belonging to dynamic content is: Among them: The output of the fully connected layer is z, where zi represents the original prediction value of the i-th category, and the Softmax function converts zi into probability pi.

[0057] In multi-category classification tasks, such as image classification, text classification, etc., the fully connected layer and Softmax Activation functions are usually used together in the last layer of the network.

[0058] The fully connected layers are responsible for converting the input features into “logits”, while the Softmax activation function converts these “logits” into probability distributions for classification decisions.

[0059] The usual model structure is: the output of the fully connected layer is used as the input of the Softmax layer, and the output of the Softmax layer is used as the final prediction result of the model.

[0060] The feature map is input into the classifier, and the feature map contains key information in the image or video frame, such as at least one of edge, texture, and shape.

[0061] Since feature maps reduce the dimensionality and complexity of input data, the classifier can converge faster when processing feature maps, improving classification efficiency. This is particularly important for application scenarios that process video streams or large-scale image data in real time; the key information extracted from the feature map provides the classifier with rich classification basis, allowing the classifier to more accurately identify dynamic content and static content. This enhanced classification performance helps improve the overall system performance and user experience; since feature maps can contain different types of key information (such as edges, textures, shapes, etc.), the classifier can be flexibly applied to different application scenarios, whether it is video surveillance that needs to identify moving objects, or autonomous driving systems that need to analyze dynamic scenes, the requirements can be met by adjusting the feature extraction method and classifier parameters.

[0062] 3) Threshold determination In a video frame, each pixel or more commonly each small block (such as an 8x8 or 16x16 pixel block) may be classified as dynamic or static. This classification is usually based on the degree of change or complexity of the content in the area. Using deep learning models (such as convolutional neural networks CNN with fully connected layers and Softmax activation functions), it is possible to automatically learn and predict the probability of each area belonging to dynamic or static content. In statistics and machine learning, 0.5 is often used as the default threshold for binary classification problems because it represents a state of equilibrium, assuming that the prior probabilities of positive and negative classes (here, dynamic content and static content) are equal.

[0063] However, in practical applications, this threshold may need to be adjusted according to specific tasks. Depending on the characteristics of the video content, the requirements of the application scenario, and the performance goals of the system, the threshold may need to be adjusted to achieve the best results. For example, if the video contains a lot of subtle dynamic changes, the threshold may need to be set slightly lower to capture these changes more sensitively. On the contrary, if the video background is relatively stable and the focus is mainly on the movements of the characters, the threshold can be set slightly higher to reduce the misjudgment of the static background.

[0064] 4) Bandwidth optimization and transmission For areas that are determined to be dynamic content, the system should use more advanced encoding algorithms and higher quantization parameters (QP) to reduce information loss during compression and ensure that the motion, details, and changes in the video can be clearly presented. For example, a more sophisticated prediction model, more encoding bits, and a smaller quantization step size can be used.

[0065] In addition, for scenes with rich dynamic content, you can also consider increasing the frame rate to display fast motion more smoothly and reduce motion blur and frame skipping.

[0066] For static content areas, the system can reduce the encoding quality, use simpler encoding modes and larger quantization steps, thereby reducing the amount of encoded data. Although this will lose image details to a certain extent, since the content of these areas does not change much, this loss is visually acceptable.

[0067] For static areas that remain unchanged for multiple consecutive frames, a longer GOP (Group of Pictures) length can be used, that is, the number of key frames in these areas is reduced, and the amount of data can be further reduced through inter-frame prediction.

[0068] Based on the results of dynamic content recognition, the system can adjust the transmission bandwidth of different areas in real time. For dynamic content areas, more bandwidth resources are allocated to ensure timely data transmission and high-quality decoding effects; while for static content areas, bandwidth allocation can be appropriately reduced and the saved bandwidth can be used for other parts that require higher bandwidth transmission, such as audio or higher priority video streams.

[0069] In the case of limited bandwidth, the system can also prioritize content according to its importance and urgency. For example, in a video conference, the speaker's face and gestures are often more important than the background or the content of a fixed camera view, so they can be given higher bandwidth priority.

[0070] It is understandable that the system should continuously monitor network conditions, including key indicators such as bandwidth utilization, latency, and packet loss rate. These indicators will directly affect the quality of audio and video transmission and user experience. Based on real-time feedback from network conditions, the system should be able to adaptively adjust parameters such as encoding quality, frame rate, and bandwidth allocation. For example, when the network is congested, the system can reduce the overall encoding quality or frame rate to reduce the amount of data transmission; when the network condition improves, the encoding quality and frame rate can be increased to restore a better visual experience.

[0071] Through the implementation of the above strategies, the audio and video transmission system can more intelligently respond to transmission requirements in different scenarios, achieve dynamic optimization of bandwidth and encoding quality, and thus improve overall transmission efficiency and user experience.

[0072] 5) Audio and video synchronization and playback The client 30 receives the audio and video data packets after optimized transmission, and performs decoding and synchronization processing.

[0073] During playback, ensure that the timestamps of the video frames and audio signals are aligned to ensure synchronous playback of audio and video.

[0074] In video conferencing systems, this bandwidth optimization method can significantly improve the user experience. For example, during a meeting, when participants' facial expressions and gestures become dynamic content, the system will prioritize the transmission quality of these areas to ensure that other participants can clearly see the expressions and emotional exchanges. For static content such as background or static PPT presentations, the transmission quality can be appropriately reduced, thereby reducing network bandwidth usage without sacrificing key information.

[0075] By introducing dynamic content recognition technology based on deep learning, the audio and video transmission system can allocate network resources more intelligently and achieve efficient use of bandwidth. This can not only improve transmission efficiency, reduce network delay and packet loss, but also improve user experience and make audio and video communication smoother and more realistic.

[0076] Example 3 Please refer to Figure 1 and Figure 2 On an online education platform, teachers need to impart knowledge to remote students through video conferencing. In order to provide high-quality audio and video transmission experience, the platform adopts a two-channel audio and video transmission system. The system includes a server, a client, and a main channel and auxiliary channel for transmitting video, audio, and auxiliary information (such as PPT, whiteboard annotations, etc.).

[0077] System composition and workflow 1) Server 30 Hardware and interfaces: The server 30 is equipped with a high-performance CPU, GPU and multiple communication interfaces (such as Ethernet interface, USB interface, etc.) to support different types of data transmission requirements. The main stream channel 21 and the auxiliary stream channel 22 respectively realize parallel data transmission through different communication interfaces.

[0078] 2) Data reception and processing: The server 30 receives audio and video data streams and instructions for auxiliary information such as PPT and whiteboard annotations from the client 10. These instructions inform the server 30 which content should be transmitted through the main stream channel 21 (such as high-definition video and audio) and which content should be transmitted through the auxiliary stream channel 22 (such as PPT slides, dynamic annotations, etc.).

[0079] 3) Information Packaging and Transmission: The server 30 performs intelligent packaging processing according to the content and type of the received data. For the mainstream channel 21, the server 30 encodes and compresses the high-definition video and audio data according to the network conditions, and packages them into data packets suitable for network transmission. For the auxiliary stream channel 22, the server 30 packages auxiliary information such as PPT and whiteboard annotations with a lower data volume to ensure that this information can be synchronized to the client 10 in a timely manner.

[0080] communication channel 1) Mainstream Channel 21: Uses high-bandwidth, low-latency communication protocols (such as UDP protocol with TCP confirmation mechanism) to ensure smooth transmission of high-definition video and audio data. Mainstream Channel 21 is implemented through an Ethernet interface and uses existing network infrastructure to provide a stable transmission environment.

[0081] 2) Auxiliary stream channel 22: uses a more flexible communication protocol (such as WebSocket or HTTP / 2's Server Push function) to transmit auxiliary information such as PPT and whiteboard annotations in the form of smaller data packets. Auxiliary stream channel 22 may be implemented through a USB interface or other auxiliary communication interface to meet the transmission requirements of different types of data.

[0082] Client 10 1) Data reception and decompression: After receiving the packaged data from the server 30 through the network, the client 10 decompresses and restores it according to the data type. For the data packets of the mainstream channel 21, the client 10 uses the corresponding decoder to decompress the video and audio data into a playable format; for the data packets of the auxiliary stream channel 22, the client 10 restores the auxiliary information such as PPT and whiteboard annotations to the user interface.

[0083] 2) Playback and synchronization: The decompressed audio and video data is played synchronously in the player of the client 10, ensuring real-time synchronization of video, audio, and auxiliary information such as PPT and whiteboard annotations. The client 10 also has a buffering mechanism to deal with network delays or packet loss and other issues to ensure the continuity of user experience.

[0084] During the online education process, the client 10 can receive and display audio and video data and subtitle tag data in real time to ensure the efficient conduct of the meeting. The server 30 dynamically adjusts bandwidth resources according to the real-time monitored data changes to ensure the stability and efficiency of data transmission.

[0085] Dynamically adjust bandwidth: The server 30 dynamically adjusts bandwidth allocation of the main stream channel 21 and the auxiliary stream channel 22 according to the change range of the audio and video data and the subtitle tag data to ensure efficient data transmission.

[0086] Prioritize the transmission of dynamic data: The server 30 prioritizes the transmission of audio and video data with large changes to ensure smooth playback of key scenes. For subtitles and tag data with small changes, the transmission volume is reduced to release bandwidth resources and improve overall transmission efficiency.

[0087] The dual-channel audio and video transmission optimization system can ensure the efficient transmission of audio and video data and subtitle tag data, and improve the user experience of remote conferences. Audio and video data are transmitted through the mainstream channel 21 to ensure the smoothness of the conference screen and the clarity of the voice signal; subtitle and tag data are transmitted through the auxiliary stream channel 22 to ensure the real-time and accuracy of subtitle and tag information.

[0088] Secondly, the system dynamically adjusts bandwidth allocation according to the change range of audio and video data and subtitle tag data to avoid wasting bandwidth resources and improve transmission efficiency. It prioritizes the transmission of audio and video data with large changes to ensure smooth playback of key scenes; it reduces the transmission volume of subtitle and tag data with small changes to release bandwidth resources and improve overall transmission efficiency.

[0089] Finally, by giving priority to the transmission of dynamic data and reducing the amount of static data transmitted, network congestion can be reduced, and the stability and reliability of the system can be improved. By dynamically adjusting bandwidth resources, the stability and efficiency of data transmission can be ensured, and transmission delays and data loss caused by insufficient bandwidth can be avoided.

[0090] The dual-channel audio and video transmission optimization system is not only suitable for remote conferencing, but can also be widely used in online education, telemedicine, video live broadcast and other fields. By optimizing bandwidth utilization and improving data transmission efficiency, the system can significantly improve user experience and has broad application prospects and market value.

[0091] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A two-line channel audio and video transmission optimization method, the method comprising: Receive a first data stream of a mainstream channel and a second data stream of a secondary stream channel; Automatically identify a first change amplitude of a first data stream of a mainstream channel, and determine a first bandwidth requirement for audio and video transmission of a two-line channel according to the first change amplitude; Analyze a second change amplitude of the second data stream of the auxiliary stream channel, and determine a second bandwidth requirement for dual-line channel audio and video transmission according to the second change amplitude; Dynamically determine the total bandwidth requirements of the main stream channel and auxiliary stream channel and allocate bandwidth resources; The optimized data packets are transmitted back to the client through the main stream channel and auxiliary stream channel.

2. A two-line channel audio and video transmission optimization method according to claim 1, characterized in that: The first variation range includes at least one of the video playback frame rate, resolution, bit rate, audio sampling rate, sampling bit number and channel number.

3. A two-line channel audio and video transmission optimization method according to claim 1, characterized in that: The second change range includes at least one of a subtitle content update frequency, a label change, a time point when the content changes, and a recording time.

4. A two-line channel audio and video transmission optimization method according to claim 1, characterized in that: The dynamically determining bandwidth requirement includes: A classifier that combines a fully connected layer and a softmax activation function is designed, the feature map is input into the classifier, and the probability of each pixel or area belonging to dynamic content or static content is output; by setting a standard threshold, areas with a probability higher than the standard threshold are judged as dynamic content, and areas with a probability lower than the standard threshold are judged as static content.

5. A two-line channel audio and video transmission optimization method according to claim 4, characterized in that: The classifier satisfies the following relationship: Among them: The output of the fully connected layer is z, where zi represents the original prediction value of the i-th category, and the Softmax function converts zi into probability pi.

6. A two-line channel audio and video transmission optimization method according to claim 4, It is characterized in that The feature map contains key information in the image or video frame, such as at least one of edge, texture, and shape.

7. According to the two-line channel audio and video transmission optimization method described in claim 1, the total bandwidth requirement also includes adding a certain redundant bandwidth on the basis of.

8. A two-line channel audio and video transmission system, characterized in that: The system includes a server, a client, and a communication channel, wherein the communication channel includes a mainstream channel and a secondary stream channel, and the server receives an indication of the mainstream channel and the secondary stream channel for transmitting the content of the video and audio data from the client, wherein the mainstream channel and the secondary stream channel correspond to different types of communication interfaces; The server is used to receive audio and video data from the client, and package the information according to the main stream channel data or the auxiliary stream channel, and then send the packaged data to the client through different communication channels; The client is responsible for receiving the packaged data from the server, and decompressing and restoring the data according to the data type, and the decompressed audio and video data is played on the client.

9. A two-line channel audio and video transmission system, characterized in that: The system includes the optimization method for dual-line channel audio and video transmission as described in any one of claims 1-7.

Images