Network transmission optimization method and apparatus, storage medium, electronic device, and program product
By identifying and counting keyframe information in live broadcast and streaming data streams, determining network optimization requirements, and using bandwidth adjustment and bit rate adaptive algorithms, the problem of poor network quality in live broadcast and streaming media applications is solved, and the user experience is improved and data transmission stability.
Patent Information
- Application Number
- PCT/CN2024/090785
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-29
- Filing Date
- 2024-04-30
- Publication Date
- 2024-12-05
AI Technical Summary
Live streaming and streaming applications have high requirements for real-time and bandwidth, but the impact of network quality leads to limitations in stability and user experience, and a method that can optimize the network in a timely and reasonable manner is needed to improve network quality.
By receiving data flow, identifying and counting keyframe information, determining whether network optimization processing is required for data receiver based on statistical information, and using network optimization methods such as bandwidth adjustment and bit rate adaptive algorithms to ensure the stability and fluency of data transmission .
It effectively improves network transmission quality, reduces lag and screen dissipation in video playback, and ensures a smooth experience on the user side, especially in scenarios where network quality decreases or data transmission pressure increases.
Smart Images

Figure CN2024090785_05122024_PF_FP_ABST
Abstract
Description
Network transmission optimization method, device, storage medium, electronic device and program product
[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 29, 2023, with application number 202310621502.5 and application name “Network transmission optimization method, device, computer-readable medium and electronic device”. Technical Field
[0002] The present application relates to the field of computer and communication technology, and more specifically, to a network transmission optimization method, device, storage medium, electronic device, and program product.
[0003] Background of the Invention
[0004] Live streaming and streaming applications have become an integral part of daily life. These applications have high real-time requirements and consume a lot of bandwidth. However, network quality can significantly impact the stability and user experience of these applications. Specifically, users have increasing demands for live streaming and streaming applications, demanding a smoother, clearer, and more stable viewing experience. Furthermore, the content of live streaming and streaming applications is becoming increasingly diverse, requiring higher-speed, low-latency network support.
[0005] Therefore, how to optimize the network in a timely and reasonable manner to improve network quality is a technical problem that needs to be solved urgently.
[0006] Summary of the Invention
[0007] The embodiments of the present application provide a network transmission optimization method, device, computer-readable medium, electronic device and program product, which can promptly and reasonably determine whether network optimization processing is required, and then take corresponding network optimization measures in a timely manner when network optimization processing is required, which is conducive to improving network transmission quality.
[0008] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.
[0009] According to one aspect of an embodiment of the present application, a network transmission optimization method is provided, including:
[0010] Receive the data stream sent by the data sender that needs to be transmitted to the data receiver;
[0011] identifying key frames contained in the data stream;
[0012] Performing statistics on the key frames to obtain key frame statistical information; and,
[0013] When it is determined according to the key frame statistical information that network optimization processing needs to be performed on the data receiver, a network optimization operation is performed on the data receiver.
[0014] According to one aspect of an embodiment of the present application, a network transmission optimization device is provided, including:
[0015] A receiving unit configured to receive a data stream sent by a data sender and to be transmitted to a data receiver;
[0016] an identification unit configured to identify key frames contained in the data stream;
[0017] a statistical unit configured to perform statistics on the key frames to obtain key frame statistical information; and
[0018] The processing unit is configured to perform a network optimization operation for the data receiver when it is determined according to the key frame statistical information that network optimization processing needs to be performed for the data receiver.
[0019] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the network transmission optimization method as described in the above embodiment is implemented.
[0020] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more computer programs, wherein when the one or more computer programs are executed by the one or more processors, the electronic device implements the network transmission optimization method as described in the above embodiments.
[0021] According to one aspect of an embodiment of the present application, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium. A processor of an electronic device reads and executes the computer program from the computer-readable storage medium, causing the electronic device to perform the network transmission optimization method provided in the various optional embodiments described above.
[0022] BRIEF DESCRIPTION OF THE DRAWINGS
[0023] FIG1 is a schematic diagram showing an exemplary application scenario in which the technical solutions of the embodiments of the present application can be applied;
[0024] FIG2 is a schematic diagram showing an exemplary application scenario in which the technical solution of the embodiments of the present application can be applied;
[0025] FIG3 is a schematic diagram showing an exemplary application scenario in which the technical solution of the embodiments of the present application can be applied;
[0026] FIG4 shows a flow chart of a network transmission optimization method according to an embodiment of the present application;
[0027] FIG5 shows a flow chart of a network transmission optimization method according to an embodiment of the present application;
[0028] FIG6 shows a schematic diagram of a network optimization process according to an embodiment of the present application;
[0029] FIG7 shows a schematic diagram of functions included in a UPF according to an embodiment of the present application;
[0030] FIG8 is a schematic diagram illustrating the functions of a UPF according to an embodiment of the present application;
[0031] FIG9 shows a schematic diagram of a network testing environment according to an embodiment of the present application;
[0032] FIG10 shows a schematic diagram of a network testing process according to an embodiment of the present application;
[0033] FIG11 shows a block diagram of a network transmission optimization device according to an embodiment of the present application;
[0034] FIG12 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present application.
[0035] Implementation Method
[0036] Example embodiments will now be described in a more complete manner with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to these examples; rather, these embodiments are provided to make this application more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art.
[0037] In addition, the features, structures or characteristics described in the present application may be combined in one or more embodiments in any suitable manner. In the following description, there are many specific details so that the embodiments of the present application can be fully understood. However, it will be appreciated by those skilled in the art that when implementing the technical solution of the present application, it is not necessary to use all the detailed features in the embodiments, one or more specific details may be omitted, or other methods, elements, devices, steps, etc. may be adopted.
[0038] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0039] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.
[0040] It should be noted that the term "plurality" used in this document refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. The character " / " generally indicates an "or" relationship between the associated objects.
[0041] First of all, it should be noted that this application can display a prompt interface or pop-up window before collecting relevant data (such as the data flow transmitted from the data sender to the data receiver) and during the process of collecting relevant data. The prompt interface or pop-up window is used to prompt the user that relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining relevant data after obtaining the user's confirmation operation on the prompt interface or pop-up window. Otherwise (that is, when the user's confirmation operation on the prompt interface or pop-up window is not obtained), the step of obtaining relevant data is terminated, that is, the relevant data is not obtained. In other words, in the specific implementation of this application, when it involves data flow and other related data, when the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions.
[0042] Next, the technical terms that appear in the examples of this application are explained:
[0043] 5GC, or 5G Core, is the core component of the 5G communication system. It handles data transmission and control within the mobile communication network, providing higher data rates, lower latency, and increased network capacity to support more users and devices. The 5G core network architecture adopts a layered design, consisting of a user plane and a control plane. The user plane handles data transmission, including encryption, decryption, and transmission; the control plane handles network control and management, including user authentication, session management, and resource allocation.
[0044] Video push and pull streaming, push streaming refers to the process of transmitting video from the source (usually a camera or video capture device) to the streaming server; pull streaming refers to the process of obtaining video from the streaming server and displaying it on the terminal device (usually a computer or mobile device).
[0045] UPF, or User Plane Function, refers to the network function responsible for handling user data transmission in 5G networks. UPF distributes user data traffic to different network nodes to facilitate various service requirements within the network, such as providing low-latency, high-bandwidth network services. This enables flexible network configuration, efficient data transmission and management, and improved network quality and user experience for various business scenarios. Application scenarios for UPF include enhanced mobile broadband (eMBB), the Internet of Things (IoT), and smart manufacturing (Industry 4.0).
[0046] eMBB is a fundamental service requirement in 5G networks, encompassing scenarios such as high-definition video, virtual reality (VR), and augmented reality (AR). The UPF in SDN (Software Defined Network) transmits user data to appropriate network nodes to quickly respond to mobile users' service needs and achieve low-latency, high-bandwidth data transmission.
[0047] IoT refers to connecting various devices and objects through the internet to collect and transmit data. The UPF in SDN enables flexible network configuration, distributing data transmission from various devices and objects to different network nodes, and achieving efficient data transmission and management.
[0048] Smart manufacturing refers to the use of the internet and digital technologies to digitize and intelligentize the manufacturing industry, including the collection and transmission of data from various devices and systems. The UPF within SDN can manage and optimize the networks of various devices and systems involved in smart manufacturing, improving manufacturing efficiency and product quality.
[0049] RTP, or Real-Time Transport Control Protocol, can be used to transmit voice, video, and other multimedia data. The RTP protocol is very suitable for real-time multimedia applications such as live video, Internet telephony, and video conferencing.
[0050] H.264, also known as the Advanced Video Coding (AVC) protocol, is a video codec standard for compressing digital video. The H.264 codec utilizes advanced compression technology to compress video to smaller file sizes while maintaining high quality. During the H.264 encoding process, video frames are divided into multiple small blocks, each of which is encoded into an independent data unit called a NALU (Network Abstraction Layer Unit). The NALU is the basic unit of the H.264 bitstream and contains both header information and the actual video data. The H.264 bitstream is comprised of two layers: the Video Coding Layer (VCL) and the Network Abstraction Layer (NAL). The VCL layer is responsible for compressing and encoding the video, while the NAL layer packages the data generated by the VCL layer into NALUs and adds header information for easier transmission and decoding.
[0051] Video frames are a common concept in video coding and are primarily categorized as I-frames, B-frames, and P-frames. I-frames, also known as independent frames or keyframes, can be independently encoded without relying on information from other frames. Therefore, I-frames carry the largest amount of data, providing better image quality and helping to restore the complete structure of the video image during video coding. B-frames, known as bidirectional frames, use the preceding I or P-frame and the following P-frame as reference information for decoding. B-frames have smaller data volumes than both I-frames and P-frames, improving video compression. P-frames, known as predicted frames, rely on information from the preceding keyframe for decoding. The primary function of P-frames is to improve video compression efficiency and reduce the size of transmitted data.
[0052] The VPP framework, or Vector Packet Processing framework, is an extensible, open-source framework that provides production-quality switch / router functionality. VPP offers two key features: framework extensibility and mature switching / routing capabilities. The VPP platform is designed to address the high latency associated with traditional scalar processing methods for processing individual packets. To improve network scalability, VPP employs a vector processing approach that can process multiple packets simultaneously, reducing latency. VPP can function as a network switch or router, with common use cases including data centers, cloud computing, edge computing, and the Internet of Things. Furthermore, VPP's flexibility and scalability open up new possibilities for network security, virtualized networks, and other areas.
[0053] DPDK, or Data Plane Development Kit, is an open-source data plane development toolkit that provides a set of libraries and drivers to help developers quickly build high-performance data plane applications. DPDK is primarily used in areas such as Network Functions Virtualization (NFV) and Software-Defined Networking (SDN), accelerating packet processing and forwarding, improving the performance and throughput of network applications.
[0054] An adaptive bitrate algorithm dynamically adjusts the video bitrate based on factors such as network conditions and device performance to ensure smooth video playback. ABR (Adaptive Bitrate Streaming) is a common adaptive bitrate algorithm. ABR is a video streaming transmission algorithm based on bitrate adaptation. It continuously monitors factors such as network conditions and device performance to dynamically adjust the video bitrate to ensure smooth video playback. The core concept of the ABR algorithm is to switch between different bitrates to adapt to varying network conditions. Specifically, when network conditions are good, the ABR algorithm selects a higher bitrate for transmission to improve video clarity; when network conditions are poor, the ABR algorithm selects a lower bitrate to ensure smooth video playback. In practical applications, ABR algorithms typically use heuristic strategies to select bitrates, such as buffer-based ABR and rate-based ABR. The Buffer-based ABR algorithm selects the bitrate based on the current buffer filling situation to avoid buffer overflow or rapid buffer drop; the Rate-based ABR algorithm selects the bitrate based on factors such as the current network conditions and device performance to ensure smooth video playback.
[0055] Next, let's explain the application environment of the embodiments of this application. The technical solutions of the embodiments of this application can be applied to scenarios with high real-time and bandwidth requirements, such as live broadcast and streaming applications. However, due to the influence of network quality, the stability of these applications and user experience will also be greatly affected. Specifically, users have increasingly higher demands for live broadcast and streaming applications, requiring a smoother, clearer, and more stable viewing experience. In addition, the content of live broadcast and streaming applications is becoming more and more diverse, requiring higher-speed, low-latency network support.
[0056] In response to the above problems, the embodiment of the present application proposes a new network transmission optimization solution, which can identify and count the data streams in the data transmission channel between the data sender and the data receiver to determine the transmission status of key frames that play a decisive role in the recovery of the data stream, and then promptly determine whether network optimization processing is needed, ensuring that the data receiver can smoothly receive and recover the data stream transmitted by the data sender, which is conducive to improving the quality of network transmission.
[0057] Specifically, in an application scenario of the present application, as shown in Figure 1, a remote driving system based on a 5G network is used. The camera on the vehicle side 101 transmits the video stream data through the MEC (Mobile Edge Computing) gateway 1 and the 5G private network 102. After reaching the cloud 103, the corresponding data can be processed by the signaling server, media server and Web server, and transmitted to the user side 104 such as the simulation cockpit, WPF (Windows Presentation Foundation) interface, video display window, Web large screen, etc. through data channels and media channels for display and playback.
[0058] Based on the technical solution of the embodiment of the present application, in the 5GC 105 of the 5G private network, the UPF can identify specific data (such as key frame data) in the video stream through capability exposure technology, and can cache and analyze the data to determine the current network quality. When it is determined that the network quality is poor, it can combine user bandwidth allocation or bit rate adaptation algorithm to perform network linkage optimization processing, thereby optimizing the data transmission process and ensuring that the video playback on the user side does not experience freezes, screen distortion, etc.
[0059] In addition, UPF can also handle data packet loss through capability exposure technology, such as discarding key frames in the video stream, and then simulate poor network conditions to verify the user end's ability to cope with weak network environments.
[0060] Figure 2 shows a schematic diagram of the technical solution of an embodiment of the present application applied to a vehicle-road cooperative system. Roadside equipment 201, such as millimeter-wave radar, lidar, and cameras, collects roadside images and other data. This data undergoes roadside processing 202 and is then transmitted to a V2X (Vehicle to Everything) service 203. This data is then transmitted via a vehicle-road cooperative platform 204 and the 5G core network to end users 205 (such as mini-programs, map applications, in-vehicle devices, and apps).
[0061] Similarly, in this application scenario, network linkage optimization can also be used to ensure that the transmitted data can be effectively transmitted to the end user. That is, the UPF in the 5G system can identify specific data (such as key frame data) in the transmitted data through capability exposure technology, and can cache and analyze data to determine the current network quality. When it is determined that the network quality is poor, network linkage optimization can be combined with user bandwidth allocation or bit rate adaptive algorithms to perform network linkage optimization.
[0062] FIG3 shows a schematic diagram of the application of the technical solution of an embodiment of the present application to a cloud gaming scenario. Cloud gaming, also known as gaming on demand, is an online gaming technology based on cloud computing technology. Cloud gaming technology enables thin clients with relatively limited graphics processing and data computing capabilities to run high-quality games. In a cloud gaming scenario, the game is not run on the player's game terminal, but on a cloud server, which renders the game scene into a video and audio stream and transmits it to the player's game terminal over the network. The player's game terminal does not need to have powerful graphics computing and data processing capabilities, but only needs to have basic streaming media playback capabilities and the ability to obtain player input commands and send them to the cloud server.
[0063] As shown in Figure 3, in the cloud gaming scenario, the cloud gaming client 301 communicates with the edge cloud gaming server 303 in the edge cloud platform 302 through LTE (Long Term Evolution) / NR (New Radio, i.e., new wireless, such as the wireless access network of the 5G system), and after the edge cloud platform 302 accesses EPC (Evolved Packet Core, i.e., 4G core network) / 5GC, it communicates collaboratively with the Internet application in the cloud. After applying the technical solution of the embodiment of the present application, the business data transmission of the cloud gaming scenario can be optimized for data transmission in scenarios where the network quality decreases or the data increases significantly in a short period of time, so as to ensure that the terminal user will not experience problems such as freezes and data corruption when receiving and rendering data.
[0064] It can be seen that the technical solution of the embodiment of the present application can optimize the network in any scenario, so as to timely and reasonably determine whether network optimization is needed, and then take corresponding network optimization measures in time when network optimization is needed, which is conducive to improving network transmission quality.
[0065] It should be noted that in the embodiments of the present application, the devices used by the end users may be, but are not limited to, smartphones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, vehicle-mounted terminals, smart TVs, aircraft, etc. The server mentioned in the embodiments of the present application may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.
[0066] The following is a detailed description of the implementation details of the technical solution of the embodiment of the present application:
[0067] FIG4 shows a flowchart of a network transmission optimization method according to an embodiment of the present application. The network transmission optimization method can be performed by a network device, such as an access network device (such as a base station), a core network device (such as a UPF), or an MEC server connected to the UPF. Referring to FIG4 , the network transmission optimization method includes at least steps S410 to S440, which are described in detail as follows:
[0068] In step S410 , a data stream sent by a data sender and to be transmitted to a data receiver is received.
[0069] In some optional embodiments, the data sender and data receiver are the sender and receiver of the data. For example, the data sender can be a terminal device running a game client, and the data receiver can be a game server. In this case, the game data sent by the terminal device is transmitted to the game server after passing through the access network and the core network. Alternatively, the data sender can also be the game server, and the data receiver can be a terminal device running a game client. In this case, the game data sent by the game server is transmitted to the terminal device after passing through the core network and the access network.
[0070] Since data transmission between the data sender and the data receiver needs to pass through the access network and the core network, the network device on the transmission path between the data sender and the data receiver can receive the data stream sent by the data sender that needs to be transmitted to the data receiver. For example, the UPF can receive the data stream sent by the data sender that needs to be transmitted to the data receiver, or the base station can receive the data stream sent by the data sender that needs to be transmitted to the data receiver. In other embodiments of the present application, other devices can also receive the data stream sent by the data sender that needs to be transmitted to the data receiver. For example, the MEC server connected to the UPF can receive the data stream sent by the data sender that needs to be transmitted to the data receiver.
[0071] In step S420 , key frames included in the data stream are identified.
[0072] In some optional embodiments, the type of the data frame included in the data stream may be identified based on the characteristics of the data frame, that is, whether the data frame is a key frame.
[0073] Specifically, when identifying key frames, data packets in the data stream can be captured, and features of the captured data packets can be extracted. The features of the captured data packets are then compared with features of key frame data packets to identify key frames contained in the data stream.
[0074] Optionally, at least one of the following data packet features can be compared to identify key frames contained in the data stream: the protocol field in the Internet Protocol (IP) header, the version field (i.e., the Version field) in the Real-time Transport Control Protocol (RTP) layer, the padding field (i.e., the Padding field) in the RTP layer, the extension field (i.e., the Extension field) in the RTP layer, the marker field (i.e., the Maker field) in the RTP layer, and the network abstraction layer unit type field (i.e., the Nal_unit_type field).
[0075] The reason why the key frames contained in the data stream can be identified through the above fields in the embodiment of the present application is mainly because when the key frames are transmitted based on the UDP (User Datagram Protocol) protocol, the protocol field value in the IP header of the key frame is 17, the Version field in the RTP layer is RFC 1889 Version, the Padding field, Extension field and Maker field are all False; and the Nal_unit_type of the H.264 layer is equal to 5, so the key frames contained in the data stream can be identified based on the above fields.
[0076] In some optional embodiments, when a key frame is split into multiple data packets for transmission, data packets belonging to the same key frame can be identified based on the values of designated fields contained in the captured data packets; wherein the designated fields include a start bit (such as start_bit) and an end bit (such as end_bit) in the coding protocol layer; the start bit of the first data packet in the multiple data packets is a first value, and the end bit is a second value; the start bit of the last data packet in the multiple data packets is a second value, and the end bit is the first value; the start bit and end bit of all data packets in the multiple data packets other than the first and last data packets are both the second value. Optionally, the first value can be 1, and the second value can be 0.
[0077] In some optional embodiments, after identifying the key frames contained in the data stream, the identified key frames can be stored; wherein, when the key frames are split into multiple data packets for transmission, the RTP timestamp (RTP.Timestamp) of the key frames is used as the key and the multiple data packets are used as the values, and the identified key frames are stored in the form of key-value pairs.
[0078] Alternatively, the identified key frames may be stored in a hash table. In this case, the size of the storage space set for the hash table may be received, and then, based on the size of the storage space, the most recently received key frames may be stored in a queue. That is, in the embodiment of the present application, newly received key frames are stored in a queue. However, due to the limited size of the storage space, previously received key frames may be deleted to ensure that the number of key frames stored in the queue matches the set storage space size.
[0079] In step S430, statistics are collected on the key frames included in the data stream to obtain key frame statistical information.
[0080] In some optional embodiments, the number of key frames to be transmitted currently or the growth rate of key frames to be transmitted per unit time may be counted as key frame statistical information.
[0081] In step S440 , when it is determined according to the key frame statistical information that network optimization processing needs to be performed on the data receiver, the network optimization operation for the data receiver is performed.
[0082] In some optional embodiments, if it is determined based on the key frame statistical information that the number of key frames currently to be transmitted is greater than or equal to a first number threshold, it is determined that network optimization processing is required for the data receiver. This is because if the number of key frames to be transmitted is large, it may indicate that the network is congested, and therefore it may be determined that network optimization processing is required.
[0083] In some optional embodiments, if the growth rate of key frames to be transmitted per unit time is determined to be greater than or equal to a set growth rate threshold based on key frame statistics, network optimization processing is determined to be necessary for the data receiver. This is because a large growth rate of key frames to be transmitted per unit time indicates that the video bit rate corresponding to the data stream is high or the video screen switching frequency is high, resulting in increased transmission pressure, and therefore, it can be determined that network optimization processing is necessary.
[0084] In some optional embodiments, if the growth rate of key frames to be transmitted per unit time is determined to be greater than or equal to a set growth rate threshold based on key frame statistical information, and the length of the group of pictures of the data stream is less than or equal to a set length threshold, then it is determined that network optimization processing is required for the data receiver. This is because if the growth rate of key frames to be transmitted per unit time is large and the group of pictures (GOP) is long, it indicates that the video bit rate corresponding to the data stream is high, resulting in increased transmission pressure, and therefore it can be determined that network optimization processing is required.
[0085] In some optional embodiments, if the number of key frame retransmission requests received per unit time is greater than or equal to a second quantity threshold, it may also indicate that network congestion has occurred, resulting in a large number of packet losses. Therefore, it can also be determined that network optimization processing is required for the data receiver.
[0086] In some optional embodiments, when performing network optimization processing on the data receiver, the transmission bit rate of the data stream may be reduced, or the network bandwidth for the data receiver may be increased.
[0087] In some optional embodiments, network optimization processing can also be adaptively performed through machine learning. For example, when it is determined that network optimization processing is required for the data recipient, the network characteristics between the data sender and the data recipient and the business characteristics corresponding to the data flow are obtained, and then the input parameters of a pre-trained network optimization model are generated based on the network characteristics and business characteristics. The pre-trained network optimization model is trained using network feature samples and business feature samples; the input parameters are then input into the pre-trained network optimization model, and the network optimization parameters are output to perform network optimization processing on the data recipient based on the network optimization parameters.
[0088] Optionally, the network characteristics between the data sender and the data receiver may be network status parameters, such as packet loss rate, RSSI (Received Signal Strength Indicator), RSRP (Reference Signal Receiving Power), etc. The service characteristics corresponding to the data stream may include the service's bandwidth requirements, the service's latency requirements, etc. The network optimization model may be trained using a CNN (Convolutional Neural Network) model, an RNN (Recurrent Neural Network) model, etc.
[0089] Optionally, when generating the input parameters of the pre-trained network optimization model based on the network characteristics and the service characteristics, the network characteristics and the service characteristics may be cascaded to obtain the input parameters of the network optimization model.
[0090] In some optional embodiments, after identifying key frames contained in a data stream, the technical solution of the embodiments of the present application may further discard a predetermined number of key frames according to a set discard policy, and then count the number of received key frame retransmission requests to determine the network quality simulated by discarding the key frames based on the counted number of key frame retransmission requests. In other words, in this embodiment, network quality can be simulated by discarding key frames.
[0091] Optionally, a network simulation configuration item may be set for the network simulation function, and then the network simulation function may be turned on or off based on the network simulation configuration item. Specifically, a trigger operation for the network simulation configuration item may be received. If the trigger operation is to turn on the network simulation function, the step of discarding a predetermined number of key frames according to the set discard policy is executed; if the trigger operation is to turn off the network simulation function, the process of discarding key frames is stopped.
[0092] Based on the technical solution of the embodiment shown in Figure 4, it can be seen that the present application can timely and reasonably determine whether network optimization processing is needed by identifying and counting key frames in the data stream, and then take corresponding network optimization measures in a timely manner when network optimization processing is needed, which is conducive to improving network transmission quality.
[0093] The following takes the UPF-based capability exposure technology for network optimization processing as an example to describe in detail the implementation details of the technical solution of the embodiment of the present application:
[0094] As shown in FIG5 , in 501 , it is assumed that the current network is congested, resulting in a decrease in network quality and packet loss in the transmission data stream, which results in a decrease in the video playback quality of the client, such as freezes and screen distortion.
[0095] In 502, after introducing the UPF capability opening function in the embodiment of the present application, UPF will identify the video key frames, count the number of key frames and the number of key frame retransmission requests, and then determine whether a network quality degradation problem has occurred based on the number of key frames and the number of key frame retransmission requests obtained by counting.
[0096] There are two situations when making judgments:
[0097] 1. If the number of key frame re-requests per unit time increases, it means that the network quality has indeed deteriorated, resulting in more network packet loss;
[0098] 2. If the number of key frames per unit time increases sharply and the GOP is small, it means that the user's demand for bandwidth is increasing. At this time, further decisions need to be made to optimize the network and avoid network congestion.
[0099] In step 503, after determining that network optimization is required, network optimization can be performed through a fast linkage mechanism. In the embodiment of the present application, the following two optimization methods can be used:
[0100] 1. By increasing the user's bandwidth through UPF or base stations, even if the current user's demand for key frame data transmission increases sharply, the smooth transmission of data can be guaranteed by adjusting the network bandwidth;
[0101] 2. Optimize with adaptive bitrate algorithms, such as the ABR algorithm. Even if network congestion or other quality degradation issues occur, the bitrate can be reduced for transmission to avoid lag during playback on user terminals.
[0102] After the network optimization process is performed, in step 504 , the optimization can be fed back to the user terminal to improve the video playback quality of the terminal.
[0103] It should be noted that the key frame in the embodiment of the present application can be an I frame or an IDR (Instantaneous Decoder Refresh) frame. The following description will continue with the example of the key frame being an I frame.
[0104] FIG6 shows a schematic diagram of a network optimization process according to an embodiment of the present application. Specifically,
[0105] In step 601, relevant functions can be developed in the UPF network element in the 5G core network to identify, count, and cache I frames, as well as analyze the amount of I frame data per unit time and I frame loss. This allows the current network quality to be determined in step 602. For example, an increase in the number of I frame re-requests indicates severe I frame loss; a small GOP of streaming data indicates a high frame rate, which can lead to severe video freezes and artifacts.
[0106] In 603, when UPF detects that the current network quality has deteriorated and is insufficient to cope with the sharply increased I-frame data transmission volume or alleviate the problem of severe I-frame loss, it can immediately start optimization measures in 604. For example, to alleviate the severe I-frame loss, it can reduce the bit rate based on the ABR algorithm, that is, increase the GOP length, or increase the allocated network bandwidth, etc., thereby optimizing the video playback quality of the client.
[0107] Based on the technical solution of the embodiment of the present application, the UPF in the embodiment of the present application may have the following functional modules:
[0108] 1. I frame data detection function;
[0109] 2. I-frame data statistics function;
[0110] 3. I-frame data storage function;
[0111] 4. I-frame data packet loss function is used to simulate network quality through packet loss.
[0112] Optionally, the development of UPF can be assisted by DPDK technology and VPP framework, so it is only necessary to concentrate these functional modules into the node Node in VPP. In the embodiment of the present application, these functions can be integrated in both the uplink and downlink Nodes of UPF, so that I frame data can be fully (i.e., uplink and downlink) identified, analyzed and cached. Specifically, the functions implemented by the UPF functional module are shown in Figure 7, which includes the four functions mentioned above. The following is a detailed description of the functions defined by the UPF in the embodiment of the present application in conjunction with Figure 8:
[0113] The I-frame data detection function is used to identify I-frames in the data stream, that is, it includes the I-frame identification function 801 shown in FIG. 8 .
[0114] Among them, the I frame identification function 801 can be based on the packet capture tool (such as wireshark) to perform packet capture and message analysis to obtain the characteristics of the I frame data packet. After sufficient packet capture comparison experiments and reference to the RFC standard, the embodiment of the present application determined that the characteristics of the I frame data packet transmitted based on the UDP protocol are: the protocol field value in the IP header is 17, the Version field of the RTP layer is RFC 1889Version, and the Padding field, Extension field and Maker field are all False. And the Nal_unit_type of the H.264 layer of the I frame is equal to 5, indicating that the message is an encoded strip of an IDR image. Therefore, based on the above characteristics, the characteristics of the I frame data are obtained. When the code is implemented, the corresponding identification function can be developed in the UPF upstream and downstream Nodes to store the IDR data passing through the UPF in the hash table.
[0115] It should be noted that the I frame data detection function can also include the I frame head and tail packet determination function 802 shown in Figure 8. This is because an I frame may be split into multiple IP data packets during network transmission, so it is necessary to determine an I frame by identifying the IP data packets at the beginning and end of the frame. Similarly, in the embodiment of the present application, this function can be developed based on wireshark packet capture analysis and RFC standard documents. The characteristics of the I frame head message are: the start_bit field of the H.264 layer is 1; the characteristics of the I frame tail message are: the end_bit field of the H.264 layer is 1; the characteristics of the I frame middle message are: the start_bit and end_bit of the H.264 layer are both 0.
[0116] The I-frame data statistics function counts the amount of I-frame data per unit time and the number of I-frame retransmission requests received, so as to facilitate subsequent analysis of whether network optimization processing is needed.
[0117] The I frame data storage function 803 is to cache the identified I frame in the UPF into a hash table for storage. The RTP.Timestamp of the I frame is used as the key of the hash table, and the message array (i.e., multiple IP data packets obtained by splitting the I frame) is used as the value of the hash table. At the same time, in order to reduce memory overhead, the function of removing the oldest I frame data can be implemented based on the queue. In addition, the global variable I Frame_hash_size, that is, the hash table size for storing I frames, can also be set through the vat command and API (Application Programming Interface). The hash_get() in the code implementation is used to read the I frame data in the hash table; hash_set() is used to store data in the hash table; hash_unset_mem_free() is used to release the older I frame data stored in the hash table.
[0118] The purpose of developing the I frame discard function 804 is to simulate a weak network environment, that is, to actively drop packets in the UPF to simulate the packet loss phenomenon in a real network environment. For example, the value of DROP_NUM (this parameter is a pre-set parameter) is used to determine from which I frame the packet loss starts. After the packet is lost, the video playback of the client will be seriously affected, and the screen will be blurred and stuck. Then the client will actively send an I frame data re-request message to re-request the I frame data to resume video playback. Therefore, UPF can also confirm the quality of the network by counting these I frame re-request messages. Optionally, this function can be enabled and disabled through configuration items.
[0119] Based on the various functional modules in the UPF capability exposure shown in Figures 7 and 8, the technical solution of the embodiment of the present application can realize real-time statistics and analysis of I-frame data. At the same time, after the analysis shows that the current network quality has declined, the network can be optimized in a linked manner, such as increasing the bandwidth or optimizing and feedback-adjusting through the ABR bitrate adaptive algorithm to ensure the smoothness of client video playback.
[0120] In one embodiment of the present application, a schematic diagram of the network test environment in an embodiment of the present application is shown in Figure 9. To simulate streaming media transmission scenarios such as live broadcasts, the present embodiment can use ffmpeg (an open source computer program that can be used to record, convert digital audio and video, and convert them into streams) and ffplay (an audio and video media player provided by ffmpeg) tools to push and pull video streams.
[0121] Specifically, as shown in Figure 9, the streaming data is pushed by the streaming end 901 over the (R)AN (Radio Access Network). The streaming data is transmitted through the 5G core network 902. Therefore, all transmitted I information is counted and analyzed in real time by the capability exposure function in the UPF. The streaming data then flows out of the UPF's N6 port and is sent to the streaming server 903. The streaming end 904 then pulls the streaming data from the streaming server 903 to the local device for playback.
[0122] Based on the test environment shown in Figure 9, as shown in Figure 10, the network environment test scenario according to the embodiment of the present application includes a normal network scenario 1001, a weak network simulation scenario 1002 and a linkage optimization mechanism scenario 1003 in the embodiment of the present application.
[0123] Among them, in the ordinary network scenario 1001, streaming media data can be transmitted based on UDP and TCP (Transmission Control Protocol) respectively, that is, the UPF real-time statistical analysis and linkage optimization mechanism in the embodiment of the present application is not introduced, and then the situation of the client pulling the stream to play the video is observed.
[0124] Weak network simulation scenario 1002 simulates the transmission of streaming media data based on the UDP protocol in an environment with relatively congested network and high packet loss rate. However, the UPF real-time statistical analysis and linkage optimization mechanism in the embodiments of this application are not introduced. To create a weak network environment, the embodiments of this application simulate network packet loss based on the I-frame discard function in the UPF capability exposure technology, making it impossible for the client to properly receive the transmitted streaming media data.
[0125] In scenario 1003 of the linkage optimization mechanism, the UPF real-time statistical analysis and linkage optimization mechanism proposed in the embodiments of this application are introduced in a weak network environment, and then streaming media data is transmitted based on the UDP protocol. During data transmission, the UPF will count, analyze, and cache all I-frame data passing through the 5G core network. If it is determined that the current network has a large demand for I-frame transmission or that the number of I-frame re-request messages sent by the client increases, that is, if the network is unable to cope with the demand, the linkage optimization mechanism will be activated to optimize the data transmission.
[0126] Optimization methods can be divided into two aspects:
[0127] 1. Increase bandwidth. Specifically, if the UPF detects an increase in the amount of I-frame data being transmitted, or an increase in the number of I-frame re-request messages sent by the client, network congestion may have occurred, resulting in packet loss. Therefore, the bandwidth allocated to the user can be increased to ensure data transmission quality.
[0128] 2. If UPF statistics detect that there are a large number of I frames per unit time, it may be that the video bit rate played by the client is high or the screen switching frequency is high. Therefore, the amount of I frame data transmitted can be reduced by reducing the video bit rate.
[0129] After testing, we found that under normal network conditions, client playback performance was good when streaming media data was transmitted using the TCP protocol. However, when using the UDP protocol for streaming media data transmission, client playback experienced artifacts and lags. Experiments have shown that this phenomenon is caused by network packet loss due to the inherent reliability of the UDP protocol. In most applications, UDP is used for data transmission, given its superior transmission efficiency and latency compared to TCP.
[0130] To ensure the usability of the technical solutions of the embodiments of the present application, the embodiments of the present application were also tested mainly based on the UDP protocol. In a weak network environment, due to the continuous network packet loss of I frames, the client playback was extremely poor, with almost no complete picture. However, in the third test environment, that is, after the introduction of the UPF real-time statistical analysis and linkage optimization mechanism, data transmission clearly has adaptive capabilities. Even in a weak network environment, the network can be adjusted by real-time analysis of the amount of I frame data transmitted in the network per unit time, ensuring the quality of client video playback.
[0131] In summary, the technical solution of the embodiment of the present application is aimed at multimedia data transmission scenarios based on the 5G core network, and proposes a data transmission solution based on UPF capability exposure technology and network linkage feedback mechanism, aiming to ensure the smooth transmission of streaming media data, improve the video playback quality of the client, and more effectively support real-time digital twins, 5G remote driving, and industrial Internet products and projects. Compared with existing technologies, the advantages of the embodiment of the present application are reduced hardware cost overhead, and for the first time, the UPF capability exposure technology and I-frame data statistics, recognition and caching functions are combined. The proposed whole set of mechanisms has certain reference and application value in the field of 5G communication networks.
[0132] At the same time, the architecture proposed in the embodiment of the present application can be divided into two modules: statistical analysis and optimization feedback. Both modules have high scalability. In the statistical analysis function module of UPF, the I frames of streaming media data are mainly identified, statistically analyzed and cached, and the module can be further expanded to statistical analysis of other data (such as B frames or P frames).
[0133] Similarly, the technical solution of the embodiment of the present application adopts a solution of increasing user bandwidth and optimizing based on a rate adaptive algorithm in the optimization feedback step. However, in other embodiments of the present application, reinforcement learning or dynamic adaptive methods can also be introduced for further optimization. In addition, in other embodiments of the present application, congestion avoidance and control algorithms can also be introduced to reduce the probability of network congestion, and a more robust routing mechanism can also be introduced to ensure that the user end can smoothly send and receive data.
[0134] The following describes an embodiment of the device of the present application, which can be used to execute the network transmission optimization method in the above embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the network transmission optimization method in the above embodiment of the present application.
[0135] Figure 11 shows a block diagram of a network transmission optimization device according to an embodiment of the present application. The network transmission optimization device can be applied to network equipment, which can be, for example, an access network device (such as a base station), a core network device (such as a UPF), or a MEC server connected to the UPF, etc.
[0136] 11 , a network transmission optimization device 1100 according to an embodiment of the present application includes: a receiving unit 1102 , an identification unit 1104 , a statistics unit 1106 and a processing unit 1108 .
[0137] The receiving unit 1102 is configured to receive a data stream sent by a data sender and to be transmitted to a data receiver;
[0138] an identification unit 1104 configured to identify key frames contained in the data stream;
[0139] The statistics unit 1106 is configured to perform statistics on the key frames to obtain key frame statistics information; and
[0140] The processing unit 1108 is configured to perform a network optimization operation for the data receiver when it is determined according to the key frame statistical information that network optimization processing needs to be performed for the data receiver.
[0141] In some embodiments of the present application, based on the aforementioned solution, the processing unit 1108 is configured to determine that network optimization processing needs to be performed on the data receiver according to at least one of the following methods:
[0142] When it is determined according to the key frame statistical information that the number of key frames currently to be transmitted is greater than or equal to a first number threshold, determining that network optimization processing needs to be performed on the data receiver;
[0143] When it is determined according to the key frame statistical information that the growth rate of the key frames to be transmitted per unit time is greater than or equal to a set growth rate threshold, determining that network optimization processing needs to be performed on the data receiver;
[0144] When it is determined based on the key frame statistical information that the growth rate of the key frames to be transmitted per unit time is greater than or equal to the set growth rate threshold, and the picture group length of the data stream is less than or equal to the set length threshold, it is determined that network optimization processing is required for the data recipient.
[0145] In some embodiments of the present application, based on the aforementioned scheme, the processing unit 1108 is further configured to: when the number of key frame retransmission requests received per unit time is greater than or equal to a second quantity threshold, determine that network optimization processing needs to be performed on the data recipient.
[0146] In some embodiments of the present application, based on the aforementioned scheme, the identification unit 1104 is configured to: capture data packets in the data stream and extract features of the captured data packets; compare the features of the captured data packets with features of key frame data packets to identify the key frames contained in the data stream.
[0147] In some embodiments of the present application, based on the aforementioned scheme, the identification unit 1104 is configured to: compare at least one of the following data packet features to identify the key frames contained in the data stream: the protocol field in the Internet Protocol IP header, the version field in the Real-time Transport Control Protocol RTP layer, the padding field in the RTP layer, the extension field in the RTP layer, the tag field in the RTP layer, and the network abstraction layer unit type field.
[0148] In some embodiments of the present application, based on the aforementioned solution, the identification unit 1104 is further configured to: when the key frame is split into multiple data packets for transmission, identify the data packets belonging to the same key frame according to the values of the designated fields contained in the captured data packets;
[0149] Wherein, the designated field includes a start bit and an end bit in the coding protocol layer;
[0150] The start bit of the first data packet in the plurality of data packets is the first value, and the end bit of the first data packet is the second value;
[0151] The start bit of the last data packet in the multiple data packets is the second value, and the end bit of the last data packet is the first value;
[0152] The start bit and the end bit in the other data packets except the first data packet and the last data packet in the multiple data packets are both the second value.
[0153] In some embodiments of the present application, based on the aforementioned scheme, the network transmission optimization device 1100 also includes: a storage unit, configured to store the identified key frame after identifying the key frame contained in the data stream; wherein, when the key frame is split into multiple data packets for transmission, the real-time transport control protocol RTP timestamp of the key frame is used as the key, and the multiple data packets are used as the value, and the identified key frame is stored in the form of a key-value pair.
[0154] In some embodiments of the present application, based on the aforementioned scheme, the storage unit is configured to store the identified key frames through a hash table; and is configured to receive the size of the storage space set for the hash table; and according to the size of the storage space, store the latest received key frames in a queue.
[0155] In some embodiments of the present application, based on the aforementioned scheme, the network transmission optimization device 1100 also includes: a discarding unit, configured to discard a predetermined number of key frames according to a set discarding strategy after identifying the key frames contained in the data stream; the statistical unit 1106 is also configured to count the number of key frame retransmission requests received; the processing unit 1108 is also configured to determine the network quality simulated by discarding key frames based on the number of key frame retransmission requests obtained by statistics.
[0156] In some embodiments of the present application, based on the aforementioned scheme, the receiving unit 1102 is further configured to: when the trigger operation is used to turn on the network simulation function, execute the step of discarding a predetermined number of key frames according to the set discard strategy; when the trigger operation is used to turn off the network simulation function, stop the process of discarding the key frames.
[0157] In some embodiments of the present application, based on the aforementioned scheme, the processing unit 1108 performs network optimization operations for the data recipient, including at least one of the following network optimization methods: reducing the transmission bit rate of the data stream and increasing the network bandwidth for the data recipient.
[0158] In some embodiments of the present application, based on the aforementioned scheme, the processing unit 1108 is further configured to: when determining that network optimization processing needs to be performed on the data recipient, obtain the network characteristics between the data sender and the data recipient and the business characteristics corresponding to the data flow; generate input parameters of a pre-trained network optimization model based on the network characteristics and the business characteristics, and the pre-trained network optimization model is obtained by training through network feature samples and business feature samples; input the input parameters into the pre-trained network optimization model and output network optimization parameters; and perform network optimization processing on the data recipient based on the network optimization parameters.
[0159] FIG12 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present application.
[0160] It should be noted that the computer system 1200 of the electronic device shown in FIG12 is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present application.
[0161] As shown in Figure 12, the computer system 1200 may include a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1202 or the program loaded from the storage part 1208 into the random access memory (RAM) 1203, such as executing the method described in the above embodiment. Various programs and data required for system operation are also stored in the RAM 1203. The CPU 1201, ROM 1202 and RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.
[0162] The following components can be connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, and the like; an output section 1207 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 1208 including a hard disk; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. Removable media 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1210 as needed, so that computer programs read from the removable media can be installed in the storage section 1208 as needed.
[0163] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program is used to perform the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1209, and / or installed from a removable medium 1211. When the computer program is executed by the central processing unit (CPU) 1201, the various functions defined in the system of the present application are performed.
[0164] It should be noted that the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a computer program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.
[0165] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and a computer program.
[0166] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.
[0167] As another aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiments, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more computer programs, and when the one or more computer programs are executed by the electronic device, the electronic device implements the method described in the above embodiments.
[0168] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.
[0169] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable an electronic device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.
[0170] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.
[0171] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A network transmission optimization method, performed by an electronic device, comprising: Receive the data stream sent by the data sender and needs to be transmitted to the data receiver; identifying key frames contained in the data stream; Performing statistics on the key frames to obtain key frame statistical information; and, When it is determined according to the key frame statistical information that network optimization processing needs to be performed on the data receiver, a network optimization operation is performed on the data receiver.
2. The network transmission optimization method according to claim 1, further comprising: Determining that network optimization processing needs to be performed on the data receiver according to at least one of the following methods: When it is determined according to the key frame statistical information that the number of key frames currently to be transmitted is greater than or equal to a first number threshold, determining that network optimization processing needs to be performed on the data receiving party; When it is determined according to the key frame statistical information that the growth rate of the key frames to be transmitted within a unit time is greater than or equal to a set growth rate threshold, it is determined that network optimization processing needs to be performed on the data receiving party; When it is determined according to the key frame statistical information that the growth rate of the key frames to be transmitted per unit time is greater than or equal to the set growth rate threshold, and the picture group length of the data stream is less than or equal to the set length threshold, it is determined that network optimization processing needs to be performed on the data receiver.
3. The network transmission optimization method according to claim 1, further comprising: When the number of key frame retransmission requests received within a unit time is greater than or equal to a second number threshold, it is determined that network optimization processing needs to be performed on the data receiver.
4. The network transmission optimization method according to claim 1, wherein: The identifying key frames included in the data stream includes: Capturing data packets in the data stream and extracting features of the captured data packets; The features of the captured data packets are compared with the features of the key frame data packets to identify the key frames contained in the data stream.
5. The network transmission optimization method according to claim 4, wherein: The step of comparing the features of the captured data packets with the features of the key frame data packets to identify the key frames contained in the data stream includes: Compare at least one of the following data packet features to identify key frames included in the data stream: The protocol field in the Internet Protocol IP header, the version field in the Real-time Transport Control Protocol RTP layer, the padding field in the RTP layer, the extension field in the RTP layer, the tag field in the RTP layer, and the network abstraction layer unit type field.
6. The network transmission optimization method according to claim 1, wherein: When the key frame is split into multiple data packets for transmission, the method further includes: According to the value of the specified field contained in the captured data packet, the data packets belonging to the same key frame are identified; wherein, The specified field includes a start bit and an end bit in the coding protocol layer; The start bit of the first data packet among the multiple data packets is the first value, and the end bit is the second value; The start bit of the last data packet among the multiple data packets is the second value, and the end bit of the last data packet is the first value; The start bit and the end bit in the other data packets among the multiple data packets except the first data packet and the last data packet are both the second value.
7. The network transmission optimization method according to claim 1, further comprising: After identifying the key frames included in the data stream, storing the identified key frames; When the key frame is split into multiple data packets for transmission, the real-time transport control protocol RTP timestamp of the key frame is used as a key and the multiple data packets are used as values to store the identified key frame in a key-value pair manner.
8. The network transmission optimization method according to claim 7, wherein: The identified key frames are stored in a hash table; The network transmission optimization method further includes: Receiving a size of a storage space set for the hash table; According to the size of the storage space, the most recently received key frame is stored in a queue.
9. The network transmission optimization method according to claim 1, further comprising: After identifying the key frames included in the data stream, discarding a predetermined number of key frames according to a set discarding strategy; Count the number of key frame retransmission requests received; The network quality simulated by discarding the key frames is determined according to the number of key frame retransmission requests obtained by statistics.
10. The network transmission optimization method according to claim 9, further comprising: Receive trigger actions for network simulation configuration items; When the trigger operation is used to start the network simulation function, the step of discarding a predetermined number of key frames according to the set discarding strategy is executed; When the trigger operation is used to turn off the network simulation function, the process of discarding the key frame is stopped.
11. The network transmission optimization method according to any one of claims 1 to 10, wherein: The performing of the network optimization operation for the data receiving party includes at least one of the following network optimization methods: The transmission bit rate of the data stream is reduced, and the network bandwidth for the data receiver is increased.
12. The network transmission optimization method according to any one of claims 1 to 10, further comprising: When it is determined that network optimization processing needs to be performed on the data receiver, obtaining network characteristics between the data sender and the data receiver and service characteristics corresponding to the data flow; Generating input parameters of a pre-trained network optimization model according to the network characteristics and the service characteristics, wherein the pre-trained network optimization model is trained by using network characteristic samples and service characteristic samples; Inputting the input parameters into the pre-trained network optimization model, and outputting the network optimization parameters; Perform network optimization processing on the data receiver based on the network optimization parameters.
13. A network transmission optimization device, comprising: A receiving unit, configured to receive a data stream sent by a data sender and to be transmitted to a data receiver; an identification unit configured to identify key frames contained in the data stream; A statistical unit, configured to perform statistics on the key frames to obtain key frame statistical information; and, The processing unit is configured to perform a network optimization operation for the data receiver when it is determined according to the key frame statistical information that network optimization processing needs to be performed for the data receiver.
14. The network transmission optimization device according to claim 13, wherein: The processing unit is further configured to determine that network optimization processing needs to be performed on the data receiver according to at least one of the following methods: When it is determined according to the key frame statistical information that the number of key frames currently to be transmitted is greater than or equal to a first number threshold, determining that network optimization processing needs to be performed on the data receiving party; When it is determined according to the key frame statistical information that the growth rate of the key frames to be transmitted within a unit time is greater than or equal to a set growth rate threshold, it is determined that network optimization processing needs to be performed on the data receiving party; When it is determined according to the key frame statistical information that the growth rate of the key frames to be transmitted per unit time is greater than or equal to the set growth rate threshold, and the picture group length of the data stream is less than or equal to the set length threshold, it is determined that network optimization processing needs to be performed on the data receiver.
15. The network transmission optimization device according to claim 13, wherein: The processing unit is further configured to determine that network optimization processing needs to be performed on the data receiver when the number of key frame retransmission requests received within a unit time is greater than or equal to a second number threshold.
16. The network transmission optimization device according to claim 13, wherein: The identification unit is configured to capture data packets in the data stream and extract features of the captured data packets; and compare the features of the captured data packets with features of key frame data packets to identify key frames contained in the data stream.
17. The network transmission optimization device according to claim 13, wherein: When the key frame is split into multiple data packets for transmission, the identification unit is further configured to identify data packets belonging to the same key frame according to the value of a specified field contained in the captured data packets; wherein, The specified field includes a start bit and an end bit in the coding protocol layer; The start bit of the first data packet among the multiple data packets is the first value, and the end bit is the second value; The start bit of the last data packet among the multiple data packets is the second value, and the end bit of the last data packet is the first value; The start bit and the end bit in the other data packets among the multiple data packets except the first data packet and the last data packet are both the second value.
18. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the network transmission optimization method according to any one of claims 1 to 12.
19. An electronic device comprising: one or more processors; A memory for storing one or more computer programs, which, when executed by the one or more processors, enables the electronic device to implement the network transmission optimization method as described in any one of claims 1 to 12.
20. A computer program product, comprising a computer program, wherein the computer program is stored in a computer-readable storage medium, and a processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the network transmission optimization method as described in any one of claims 1 to 12.
Citation Information
Patent Citations
Code rate adjustment method and device, equipment, medium and program product
CN114501083A
Video transmission system
CN115086779A
Video optimization method, playing device, gateway device, storage medium and system
CN115623246A
Network transmission optimization method and device, computer readable medium and electronic equipment
CN116962179A
Cited By
Cache adjustment method, electronic equipment, chip system and storage medium
CN122293624A
End-cloud collaborative stream processing-based defective commodity visual quality inspection and storage interception method
CN122414997A
