Audio and video code rate adjusting method and device, program product and electronic equipment

Abstract

The invention discloses an audio and video code rate adjusting method and device, a program product and electronic equipment, and relates to the field of financial science and technology or other related fields, and the code rate adjusting method comprises the steps: collecting performance data and network data of target equipment, obtaining audio and video original stream data, and transmitting the audio and video original stream data to a server; the method comprises the steps of constructing multi-modal data based on performance data, network data and audio and video original stream data, determining a reference code rate based on the multi-modal data, determining target encoder parameters of a target encoder based on the reference code rate and the multi-modal data, and determining the target encoder parameters of the target encoder based on the target encoder parameters. And performing code rate adjustment on the audio and video original stream data by using the target encoder to obtain target audio and video stream data. According to the invention, the technical problem that the code rate of audio and video cannot be accurately adjusted in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of financial technology, and more specifically, to a method, apparatus, program product, and electronic device for adjusting the bitrate of audio and video. Background Technology

[0002] With the development of fintech, users have increasingly higher demands for financial services. To help users better select financial services, audio and video content about financial services can be played on the device.

[0003] In the audio and video field, current dynamic bitrate adaptive technology mainly relies on fixed rules or thresholds, periodically detecting network bandwidth and switching between preset bitrate levels (such as 500Kbps, 1Mbps, 2Mbps). This technology typically only considers the complexity of the video content, ignoring audio characteristics and performance differences between different terminal devices, resulting in an inability to accurately adjust the audio and video bitrate.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a method, apparatus, program product, and electronic device for adjusting the bitrate of audio and video, so as to at least solve the technical problem in the related art that the bitrate of audio and video cannot be accurately adjusted.

[0006] According to one aspect of the present invention, a method for adjusting the bitrate of audio and video is provided, comprising: collecting performance data and network data of a target device, and obtaining raw audio and video stream data; constructing multimodal data based on the performance data, network data, and raw audio and video stream data; determining a reference bitrate based on the multimodal data, and determining target encoder parameters of a target encoder based on the reference bitrate and the multimodal data; and adjusting the bitrate of the raw audio and video stream data using the target encoder based on the target encoder parameters to obtain target audio and video stream data, wherein the target audio and video stream data is used for playback on the target device.

[0007] Further, the steps of collecting performance data and network data of the target device include: collecting network data used by the target device through a preset probe to obtain network data, wherein the network data includes at least one of the following: bandwidth fluctuation data, latency data, and packet loss data; and collecting performance data of the target device through a preset interface, wherein the preset interface is used to monitor the device status of the target device.

[0008] Furthermore, the steps for constructing multimodal data based on performance data, network data, and raw audio and video stream data include: determining device status indicators based on performance data, wherein the device status indicators include at least one of the following: processor utilization, battery level, and screen resolution; determining network indicators based on network data, wherein the network indicators include at least one of the following: bandwidth fluctuation rate, latency, and packet loss rate; splitting the raw audio and video stream data to obtain video streams and audio; processing the video stream to obtain video motion vectors; processing the audio to obtain audio spectrum entropy values; and constructing multimodal data based on device status indicators, network indicators, video motion vectors, and audio spectrum entropy values.

[0009] Furthermore, the steps for determining the baseline bitrate based on multimodal data include: processing the multimodal data using a Long Short-Term Memory (LSTM) network to obtain the network state for the next time period; determining the video quality based on the video motion vectors in the multimodal data, and determining the audio quality based on the audio spectral entropy value in the multimodal data; determining the dynamic compensation factor based on the device status indicators in the multimodal data; determining the network bandwidth for the next time period based on the network state; determining the device differentiation coefficient based on the device type of the target device; and determining the baseline bitrate based on the video quality, audio quality, dynamic compensation factor, network bandwidth, and device differentiation coefficient.

[0010] Furthermore, the step of determining the target encoder parameters based on the reference bitrate and multimodal data includes: determining the motion vector magnitude based on the video motion vectors in the multimodal data; determining the target encoder indicated by the reference bitrate, wherein each encoder corresponds to a bitrate range; and determining the target encoder parameters based on the encoder type of the target encoder, the reference bitrate, the motion vector magnitude, and the multimodal data, wherein the target encoder parameters include: quantization parameter values ​​or adaptive quantization values.

[0011] Further, the step of determining the target encoder parameters based on the encoder type, reference bit rate, motion vector magnitude, and multimodal data of the target encoder includes: when the encoder type is a first type, determining a first initial quantization parameter value based on a first preset parameter, reference bit rate, and video motion vector, and adjusting the first initial quantization parameter value based on the motion vector magnitude and battery level in the multimodal data to obtain a first target quantization parameter value; when the encoder type is a second type, determining a second initial quantization parameter value based on a second preset parameter, reference bit rate, and video motion vector, and adjusting the second initial quantization parameter value based on the motion vector magnitude and battery level in the multimodal data to obtain a second target quantization parameter value; when the encoder type is a third type, determining a target adaptive quantization value based on a third preset parameter and audio spectral entropy value in the multimodal data.

[0012] Furthermore, the step of adjusting the bitrate of the raw audio and video stream data using the target encoder based on the target encoder parameters to obtain the target audio and video stream data includes: starting the target encoder based on the target encoder parameters; determining multiple playback frames in the raw audio and video stream data and identifying the frame type of each playback frame; when the frame type is the first frame type, allocating the highest bitrate to the playback frame through the target encoder, where the highest bitrate is the highest value in the bitrate range corresponding to the target encoder; when the frame type is the second frame type, adjusting the bitrate of the playback frame based on the bitrate range through the target encoder; identifying the audio scene of each playback frame, and adjusting the audio bitrate of the playback frame based on the audio scene to obtain the target audio and video stream data.

[0013] According to another aspect of the present invention, an audio / video bitrate adjustment apparatus is also provided, comprising: an acquisition unit for acquiring performance data and network data of a target device, and acquiring raw audio / video stream data; a construction unit for constructing multimodal data based on the performance data, network data, and raw audio / video stream data; a determination unit for determining a reference bitrate based on the multimodal data, and determining target encoder parameters of a target encoder based on the reference bitrate and the multimodal data; and an adjustment unit for adjusting the bitrate of the raw audio / video stream data using a target encoder based on the target encoder parameters to obtain target audio / video stream data, wherein the target audio / video stream data is used for playback on a target device.

[0014] Furthermore, the acquisition unit includes: a first acquisition module, used to acquire network data used by the target device through a preset probe, wherein the network data includes at least one of the following: bandwidth fluctuation data, latency data, and packet loss data; and a second acquisition module, used to acquire performance data of the target device through a preset interface, wherein the preset interface is used to monitor the device status of the target device.

[0015] Further, the construction unit includes: a first determining module, used to determine device status indicators based on performance data, wherein the device status indicators include at least one of the following: processor utilization, battery level, and screen resolution; a second determining module, used to determine network indicators based on network data, wherein the network indicators include at least one of the following: bandwidth fluctuation rate, latency, and packet loss rate; a first splitting module, used to split the raw audio and video stream data to obtain video streams and audio; a first processing module, used to process the video stream to obtain video motion vectors; a second processing module, used to process the audio to obtain audio spectrum entropy values; and a first construction module, used to construct multimodal data based on device status indicators, network indicators, video motion vectors, and audio spectrum entropy values.

[0016] Furthermore, the determining unit includes: a third processing module, used to process the multimodal data using a long short-term memory network to obtain the network state for the next time period; a third determining module, used to determine the video quality based on the video motion vectors in the multimodal data, and to determine the audio quality based on the audio spectral entropy value in the multimodal data; a fourth determining module, used to determine the dynamic compensation factor based on the device status indicators in the multimodal data; a fifth determining module, used to determine the network bandwidth for the next time period based on the network state; a sixth determining module, used to determine the device differentiation coefficient based on the device type of the target device; and a seventh determining module, used to determine the base bitrate based on the video quality, audio quality, dynamic compensation factor, network bandwidth, and device differentiation coefficient.

[0017] Furthermore, the determining unit also includes: an eighth determining module, used to determine the motion vector magnitude based on the video motion vector in the multimodal data; a ninth determining module, used to determine the target encoder indicated by the reference bitrate, wherein each encoder corresponds to a bitrate range; and a tenth determining module, used to determine the target encoder parameters based on the encoder type of the target encoder, the reference bitrate, the motion vector magnitude, and the multimodal data, wherein the target encoder parameters include: quantization parameter values ​​or adaptive quantization values.

[0018] Furthermore, the tenth determining module includes: a first determining submodule, used to determine a first initial quantization parameter value based on a first preset parameter, a reference bit rate, and a video motion vector when the encoder type is a first type, and to adjust the first initial quantization parameter value based on the motion vector magnitude and the battery level in the multimodal data to obtain a first target quantization parameter value; a second determining submodule, used to determine a second initial quantization parameter value based on a second preset parameter, a reference bit rate, and a video motion vector when the encoder type is a second type, and to adjust the second initial quantization parameter value based on the motion vector magnitude and the battery level in the multimodal data to obtain a second target quantization parameter value; and a third determining submodule, used to determine a target adaptive quantization value based on a third preset parameter and an audio spectral entropy value in the multimodal data when the encoder type is a third type.

[0019] Furthermore, the adjustment unit includes: a first startup module, used to start the target encoder based on the target encoder parameters; an eleventh determination module, used to determine multiple playback frames in the original audio and video stream data and identify the frame type of each playback frame; a first allocation module, used to allocate the highest bitrate to the playback frame through the target encoder when the frame type is the first frame type, wherein the highest bitrate is the highest value in the bitrate range corresponding to the target encoder; a first adjustment module, used to adjust the bitrate of the playback frame based on the bitrate range through the target encoder when the frame type is the second frame type; and a second adjustment module, used to identify the audio scene of each playback frame and adjust the audio bitrate of the playback frame based on the audio scene to obtain the target audio and video stream data.

[0020] According to another aspect of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described audio and video bitrate adjustment methods.

[0021] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement any of the above-described audio and video bitrate adjustment methods.

[0022] In this invention, performance data and network data of the target device are collected, and raw audio and video stream data are obtained. Based on the performance data, network data, and raw audio and video stream data, multimodal data is constructed. Based on the multimodal data, a base bitrate is determined. Based on the base bitrate and multimodal data, the target encoder parameters of the target encoder are determined. Based on the target encoder parameters, the target encoder is used to adjust the bitrate of the raw audio and video stream data to obtain the target audio and video stream data. This solves the technical problem in related technologies that the bitrate of audio and video cannot be accurately adjusted.

[0023] This invention employs real-time acquisition of target device performance data and network data. By integrating raw audio and video stream data to construct multimodal data packets, it achieves accurate assessment of the current playback environment, thereby realizing the technical effect of intelligent dynamic adjustment of audio and video bitrates. This solves the technical problems of response lag and inability to consider the complex relationships between network, content, and device characteristics under fixed-rule bitrate adjustments. Specifically, it collects indicators such as terminal GPU (Graphics Processing Unit) utilization and network bandwidth fluctuations. Then, it uses deep learning algorithms to analyze video motion intensity and audio spectral complexity to form multimodal data. Based on this data, it quickly predicts network conditions and content requirements, calculates a baseline bitrate, and then automatically converts the target encoder's parameter settings according to the baseline bitrate and multimodal data. This guides the streaming media control module to perform real-time encoding optimization of the raw audio and video streams, ensuring smooth playback of high-quality audio and video in various environments, effectively improving user experience and system efficiency. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0025] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing an audio / video bitrate adjustment method is shown.

[0026] Figure 2 This is a flowchart of an audio / video bitrate adjustment method according to Embodiment 1 of the present invention;

[0027] Figure 3 This is a schematic diagram of the AI-based audio and video dynamic bitrate adaptive system structure according to Embodiment 1 of the present invention;

[0028] Figure 4 This is a schematic diagram of the data acquisition module acquisition process according to Embodiment 1 of the present invention;

[0029] Figure 5 This is a schematic diagram of the processing flow of the AI ​​decision engine module according to Embodiment 1 of the present invention;

[0030] Figure 6 This is a schematic diagram of the cross-codec adapter module processing flow according to Embodiment 1 of the present invention;

[0031] Figure 7 This is a schematic diagram of the streaming media control module processing flow according to Embodiment 1 of the present invention;

[0032] Figure 8This is a schematic diagram of an optional audio / video bitrate adjustment device according to an embodiment of the present invention;

[0033] Figure 9 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] To facilitate understanding of the present invention by those skilled in the art, some terms or nouns involved in the various embodiments of the present invention are explained below:

[0037] Dynamic bitrate adaptive: A closed-loop control system that automatically adjusts audio and video encoding parameters based on real-time changes in network bandwidth, content complexity, and terminal performance.

[0038] It should be noted that all relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) collected and involved in this invention are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data comply with the relevant laws, regulations, and standards of the relevant regions, necessary confidentiality measures have been taken, and it does not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse. For example, this system has an interface with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface. The relevant information is obtained only after receiving consent from the aforementioned user or organization. If the user chooses to refuse, the process proceeds to an expert decision-making process.

[0039] This invention proposes an AI (Artificial Intelligence)-based dynamic bitrate adaptive system for audio and video, which can eliminate the decision-making disconnect between network status, content features, and device performance, achieve joint optimization of multimodal data, and compress the bitrate adjustment response latency to within 200ms, adapting to the high-fluctuation scenarios of 5G networks. Furthermore, by constructing an automatic parameter conversion mechanism across codecs, the cost of multi-format compatibility can be reduced.

[0040] The present invention will now be described in detail with reference to various embodiments.

[0041] Example 1

[0042] According to an embodiment of this application, an embodiment of an audio / video bitrate adjustment method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0043] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing an audio / video bitrate adjustment method is shown. Figure 1 As shown, computer terminal 10 (or mobile device) may include one or more ( Figure 1The processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions may also be included. In addition, it may include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera, wherein the network interface can be connected to wired and / or wireless networks. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0044] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0045] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the audio and video bitrate adjustment method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned audio and video bitrate adjustment method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0046] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0047] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0048] Under the aforementioned operating environment, this application provides the following: Figure 2 The method for adjusting the bitrate of audio and video is shown. Figure 2 This is a flowchart of the audio / video bitrate adjustment method according to Embodiment 1 of the present invention, as follows: Figure 2 As shown, the method includes the following steps:

[0049] Step S201: Collect performance data and network data of the target device, and obtain raw audio and video stream data.

[0050] In this embodiment of the invention, the status of the target device can be continuously monitored to obtain performance data, such as GPU utilization, battery level, and screen resolution. This data reveals the device's rendering capabilities and power consumption levels, forming the basis for subsequent optimization of video clarity and frame rate. Furthermore, network probe technology can be used to capture real-time network metrics at a preset frequency (e.g., 5Hz, i.e., once every 200 milliseconds), including bandwidth fluctuation, RTT (Round-Trip Time) latency, and packet loss rate. This information is crucial for timely adjustment of the transmission bitrate and prevention of stuttering.

[0051] In this embodiment of the invention, raw audio and video stream data from the source can be received, which provides raw material for the subsequent customization of encoding parameters.

[0052] Step S202: Construct multimodal data based on performance data, network data, and raw audio and video stream data.

[0053] In this embodiment of the invention, the collected data (performance data, network data, and raw audio and video stream data) can be fused to generate multi-dimensional decision inputs (i.e., multimodal data) that include network conditions, video content attributes, and device status. For example, the raw audio and video streams can be transformed into computable feature vectors using deep learning models from different fields (such as CNN (Convolutional Neural Network) for image analysis and FFT (Fast Fourier Transform) for audio feature extraction). These vectors are then combined with real-time data obtained from network probes and system monitoring and packaged into a multimodal data format that is easy for AI engines to analyze.

[0054] Step S203: Based on the multimodal data, determine the reference bit rate, and based on the reference bit rate and the multimodal data, determine the target encoder parameters of the target encoder.

[0055] In this embodiment of the invention, multimodal data can be processed by an AI decision engine to obtain a baseline bitrate. Here, the AI ​​decision engine includes: an LSTM (Long Short-Term Memory) network and a multi-objective optimization network.

[0056] In this embodiment of the invention, an LSTM (Long Short-Term Memory) network can be used to predict the network state and the complexity trend of audio and video content in the next 200ms, thereby initially setting an ideal transmission bitrate. This prediction mechanism can anticipate upcoming changes and reduce playback latency. Furthermore, the multi-objective optimization network can introduce dynamic compensation factors and device differentiation coefficients based on the initially set bitrate, considering the impact of motion intensity on video quality, the influence of audio spectrum on auditory experience, and the specific needs of the target device (such as the frame rate of VR (Virtual Reality) devices and the low power consumption preference of IoT (Internet of Things) devices). It calculates video quality scores, audio quality scores, and dynamic compensation factors based on device state, and combines this with the real-time available network bandwidth and device differentiation coefficients to output three-dimensional decision parameters suitable for the current situation: bitrate R, resolution, and frame rate. These three-dimensional decision parameters are then further transformed into control commands (encoder parameters) required by a specific encoder. For example, when calculating the QP (Quantization Parameter) value of an H.264 encoder (a video compression tool based on the H.264 standard), the intensity of motion vectors can be considered; while VP9 (a video coding format) adjusts its AQ-mode (Adaptive Quantization) based on the audio spectral entropy value. This ensures that the encoder operates with optimal parameters, maintaining good playback quality even under poor network conditions or sudden changes in content complexity.

[0057] Step S204: Based on the target encoder parameters, the target encoder is used to adjust the bitrate of the original audio and video stream data to obtain the target audio and video stream data, which is used for playback on the target device.

[0058] In this embodiment of the invention, the target encoder can prioritize the quality of I-frames (keyframes) in the raw audio and video stream data based on its parameters. For P-frames (predicted frames) and B-frames (bidirectional predicted frames) in the raw audio and video stream data, the bitrate is dynamically adjusted based on real-time motion intensity and network bandwidth to achieve optimal picture quality with limited network resources. Furthermore, the audio encoding bitrate is intelligently graded and controlled according to the importance of audio in different scenarios. In voice scenarios, to ensure clarity, the audio encoding bitrate is maintained at a low level (e.g., ≤64Kbps), while in action or live streaming scenarios, to ensure spatial awareness and an immersive experience, the audio encoding bitrate is significantly increased (e.g., ≥128Kbps). This not only allows for rapid response to network fluctuations but also enables the most suitable encoding decisions for different audio and video content and terminal device characteristics, improving the playback quality and user experience of the audio and video streams while reducing reliance on cloud computing resources, thus achieving an efficient and adaptive bitrate adjustment mechanism.

[0059] In summary, by acquiring real-time performance and network data from the target device and constructing multimodal data packets from raw audio and video streams, the current playback environment is accurately assessed. This enables intelligent dynamic adjustment of audio and video bitrates, resolving the technical issues of response lag and the inability to consider the complex relationships between network, content, and device characteristics under fixed-rule bitrate adjustments. Specifically, metrics such as terminal GPU utilization and network bandwidth fluctuations are collected. Then, deep learning algorithms are used to analyze video motion intensity and audio spectral complexity to form multimodal data. Based on this data, network conditions and content requirements are quickly predicted, and a baseline bitrate is calculated. Subsequently, based on the baseline bitrate and multimodal data, the parameter settings of the target encoder are automatically adjusted to guide the streaming media control module in real-time encoding optimization of the raw audio and video streams. This ensures smooth playback of high-quality audio and video in various environments, effectively improving user experience and system efficiency.

[0060] Figure 3 This is a schematic diagram of the AI-based audio / video dynamic bitrate adaptive system structure according to Embodiment 1 of the present invention, as shown below. Figure 3 As shown, it includes: a data acquisition module, an AI decision engine module, a cross-codec adapter module, and a streaming media control module. The data acquisition module is used to output multimodal data based on the performance feedback of the terminal device and network data. The AI ​​decision engine module is used to output three-dimensional decision parameters based on the received multimodal data. The cross-codec adapter module is used to output encoder-specific instructions based on the received three-dimensional decision parameters. The streaming media control module is used to generate real-time bitstream based on the received encoder-specific instructions to achieve playback on the terminal device.

[0061] To improve the accuracy of collecting performance data and network data from the target device, in the audio and video bitrate adjustment method provided in Embodiment 1 of this application, network data is obtained by collecting network data used by the target device through a preset probe. The network data includes at least one of the following: bandwidth fluctuation data, latency data, and packet loss data. Performance data of the target device is collected through a preset interface, wherein the preset interface is used to monitor the device status of the target device.

[0062] In this embodiment of the invention, a data acquisition module can continuously monitor the network environment of the target device using probe technology pre-installed at the network layer. Here, the probe collects bandwidth fluctuation data, latency data (i.e., RTT (Round Trip Time)), and packet loss data (i.e., the rate of data packet loss during network transmission) at a frequency of 5Hz (i.e., once every 200 milliseconds). These data are crucial for accurately assessing the real-time network status and serve as the input basis for subsequent intelligent bitrate adjustment algorithms, ensuring the smoothness and stability of audio and video stream transmission. For example, bandwidth fluctuation data reflects changes in available network resources. If a sudden drop in bandwidth is detected, the system needs to immediately reduce the transmission bitrate to avoid buffering delays or packet loss. Latency data is used to estimate the average time for data packets to travel from source to destination. An increase in RTT may indicate network congestion, in which case the system should proactively select a lower resolution and frame rate to reduce the amount of data. Packet loss data indicates network reliability. When the packet loss rate is too high, the encoding strategy needs to be adjusted immediately to ensure the complete transmission of key frames.

[0063] Device status is collected in real time through pre-defined interfaces on the target device, including but not limited to performance metrics such as GPU utilization, battery level, and screen resolution. These pre-defined interfaces connect to the device's underlying system API (Application Programming Interface) to obtain the most direct performance feedback with minimal system overhead. For example, GPU utilization reflects the device's graphics processing load; when GPU utilization exceeds 80%, the AI ​​decision engine will consider reducing video resolution or frame rate to alleviate rendering pressure and prevent overheating or stuttering. Battery level is a crucial indicator of device battery life; when the battery is below 20%, the system will appropriately reduce video quality to extend device usage time. Screen resolution is used to adjust the clarity of transmitted video according to the device's actual display needs; higher resolution devices will receive a clearer video stream.

[0064] In this embodiment, real-time collection of network data and terminal performance data provides the AI ​​decision engine with detailed and real-time environmental information. This enables the system to make rapid and accurate decisions regarding audio and video encoding parameter adjustments in different scenarios (such as network congestion, limited device resources, and changes in content complexity). This not only improves the playback quality and smoothness of audio and video streams but also optimizes the resource utilization efficiency of terminal devices, reduces cloud computing load, and provides users with a better viewing experience. Especially in high-bandwidth, low-latency network environments such as 5G, this strategy can fully utilize network resources, reduce rebuffering time, and ensure smooth playback of audio and video content on various devices, while simultaneously balancing content quality and device battery life.

[0065] To improve the accuracy of constructing multimodal data, in the audio and video bitrate adjustment method provided in Embodiment 1 of this application, device status indicators are determined based on performance data, wherein the device status indicators include at least one of the following: processor utilization, battery power, and screen resolution; network indicators are determined based on network data, wherein the network indicators include at least one of the following: bandwidth fluctuation rate, latency, and packet loss rate; the original audio and video stream data are split to obtain video streams and audio; the video stream is processed to obtain video motion vectors; the audio is processed to obtain audio spectrum entropy values; and multimodal data is constructed based on device status indicators, network indicators, video motion vectors, and audio spectrum entropy values.

[0066] In this embodiment of the invention, device status indicators can be determined based on the collected performance data, including but not limited to: processor (CPU) utilization, battery level, screen resolution, etc. These indicators collectively constitute the device status indicators. For example, processor utilization can be obtained through the operating system API, battery level can be read through the device hardware status API, and screen resolution can be obtained through the display management API. The determination of device status indicators is the basis for subsequent intelligent decision-making, ensuring that the system can make reasonable resource allocation based on real-time terminal performance. Furthermore, network indicators can be determined based on the collected network data, including: bandwidth fluctuation rate, latency, and packet loss rate, etc. Real-time monitoring of these indicators ensures that the system can quickly respond to changes in network conditions. For example, when network bandwidth suddenly drops or high latency occurs, the system can adjust the bitrate in a timely manner to adapt to the network environment and avoid interruption or stuttering of audio and video streams.

[0067] Then, the raw audio and video streams can be split into video and audio streams. For the video stream, a convolutional neural network (CNN) is used to analyze inter-frame motion and extract video motion vectors to reflect the dynamic range of the video content. For action-intensive scenes, the system tends to allocate a higher bitrate to maintain detail clarity. For the audio stream, a fast Fourier transform (FFT) is used to calculate the audio spectral entropy value to determine the complexity and frequency characteristics of the audio. For example, in explosion or music scenes, the system will prioritize audio quality.

[0068] Subsequently, the data aggregation module packages device status indicators, network indicators, video motion vectors, and audio spectrum entropy values ​​into multimodal data packets, with each data packet spaced 200ms apart. This multimodal data construction enables the AI ​​decision engine to comprehensively analyze the dynamic environment during audio and video transmission, considering device performance, network status, and content characteristics, thereby making more accurate and faster decisions. This real-time multimodal data analysis improves the stability and quality of audio and video transmission, adapts to various complex scenarios (such as the high volatility of 5G networks), and optimizes encoding parameters for different types of terminal devices.

[0069] Figure 4 This is a schematic diagram of the data acquisition module acquisition process according to Embodiment 1 of the present invention, as shown below. Figure 4 As shown, in the data acquisition module, the network probe samples network metrics such as bandwidth fluctuation rate, RTT latency, and packet loss rate at 5Hz. The video analyzer extracts motion vectors MV_x and MV_y (i.e., vector coordinates) for each frame using CNN. The audio analyzer calculates the audio spectrum entropy value using FFT. The device sensors monitor device status such as GPU utilization, battery level, and screen resolution. Then, through data aggregation, a 200ms data packet (including: network metrics, motion vectors, spectrum entropy value, and device status) can be output.

[0070] In this embodiment, a smart decision-making system supporting multimodal data is constructed by sensing the dynamic changes of devices and networks in real time, as well as the characteristics of audio and video content. This system can intelligently adjust the bitrate, resolution, and frame rate of audio and video encoding to adapt to different device performance and network environments without affecting user experience. Furthermore, through the analysis of motion vectors and spectral entropy values, encoding resources can be intelligently allocated according to the dynamics and complexity of the content, such as automatically increasing the bitrate in high-dynamic scenes and decreasing the bitrate in static or simple audio scenes to save bandwidth.

[0071] To improve the accuracy of determining the baseline bitrate, the audio and video bitrate adjustment method provided in Embodiment 1 of this application employs a Long Short-Term Memory (LSTM) network to process multimodal data and obtain the network state for the next time period; video quality is determined based on the video motion vectors in the multimodal data, and audio quality is determined based on the audio spectral entropy value in the multimodal data; a dynamic compensation factor is determined based on the device status indicators in the multimodal data; the network bandwidth for the next time period is determined based on the network state; a device differentiation coefficient is determined based on the device type of the target device; and the baseline bitrate is determined based on video quality, audio quality, dynamic compensation factor, network bandwidth, and device differentiation coefficient.

[0072] In this embodiment of the invention, multimodal data packets containing terminal performance feedback, raw network data, and audio / video content features are received. These data packets are continuously input into an LSTM network at 200ms intervals to predict the network state for the next time period. Here, LSTM is a type of recurrent neural network (RNN), which is particularly suitable for processing and predicting sequential data because it can remember long-term dependency information and solve the gradient vanishing problem common in RNNs. Through training, LSTM can identify trends in network state changes, such as bandwidth fluctuations, increased latency, or sudden changes in packet loss rate, and predict the network state for the next 200ms based on these trends, including bandwidth availability, latency, and packet loss.

[0073] In this embodiment of the invention, video quality can be determined based on video motion vectors in multimodal data. Motion vectors reflect the degree of change between video frames; high-intensity motion typically requires a higher bitrate to ensure detail clarity. Similarly, audio quality is determined based on audio spectral entropy values. The spectral entropy value reflects the complexity of the audio; highly complex audio requires a higher bitrate to maintain fidelity, especially to preserve spatial awareness and detail clarity. Furthermore, the dynamic compensation factor is determined based on device status indicators from multimodal data. For example, when GPU utilization exceeds 80%, it indicates that the device may not be able to handle high-resolution or high-frame-rate video smoothly. At this time, the dynamic compensation factor will be triggered, and the system will prioritize reducing the resolution or frame rate to alleviate the device's burden and avoid rendering latency. In addition, if the device's battery is too low, the dynamic compensation factor will also be adjusted to optimize power consumption and extend the device's usage time.

[0074] In this embodiment of the invention, the network bandwidth for the next time period can be calculated based on the network state predicted by the LSTM network for the next 200ms. Bandwidth prediction is a key factor in determining the coding bitrate, especially in 5G network environments where bandwidth fluctuates significantly. Accurately predicting bandwidth changes is crucial for maintaining stable transmission. Then, based on the type of target device, a device differentiation coefficient α is determined. For example, the α value for VR devices is set to 1.2, indicating that the system will allocate more resources to VR devices to ensure high frame rates and high-resolution video transmission, reducing motion sickness. For IoT devices, such as 1080P cameras, the α value is set to 0.8, and the system will be more inclined to optimize energy consumption and bandwidth utilization efficiency, as these devices typically do not require high-definition video transmission.

[0075] Then, based on video quality Audio quality Dynamic compensation factor Predicted network bandwidth In addition, the device differentiation coefficient α is used to determine the baseline bitrate through multi-objective optimization calculations. This baseline bitrate forms the basis for adjusting audio and video encoding parameters, comprehensively considering network conditions, device performance, and the complexity of the audio and video content to ensure the best audiovisual experience on any device. The multi-objective optimization calculation formula is as follows: For example, in high-speed motion scenarios, the system will adjust based on the motion vector. Increase video bitrate; in low power conditions, the system may reduce audio bitrate. The system adjusts video resolution to save power. Through this dynamic and intelligent baseline bitrate adjustment, the system can adapt to various complex scenarios, providing smooth and high-quality audio and video transmission while reducing encoding and network transmission costs.

[0076] Figure 5 This is a schematic diagram of the processing flow of the AI ​​decision engine module according to Embodiment 1 of the present invention, as shown below. Figure 5 As shown, multimodal data packets are input into the AI ​​decision engine module. The network state and content complexity trend in the next 200ms are predicted based on LSTM time series. Then, multi-objective optimization calculations are performed to obtain bitrate R, resolution, and frame rate, and output three-dimensional decision parameters.

[0077] In this embodiment, deep learning is used to predict network status and combine it with device performance and audio / video content characteristics to achieve intelligent optimization of audio / video encoding parameters. This not only improves the stability of audio / video transmission, but also provides a personalized and high-quality experience for different devices, reduces the switching cost of multi-format encoding, optimizes the computing load of terminals and the cloud, and ultimately improves the overall user satisfaction and system efficiency of audio / video services.

[0078] To improve the accuracy of determining the target encoder parameters, in the audio and video bitrate adjustment method provided in Embodiment 1 of this application, the motion vector magnitude is determined based on the video motion vector in the multimodal data; the target encoder indicating the reference bitrate is determined, wherein each encoder corresponds to a bitrate range; the target encoder parameters are determined based on the encoder type of the target encoder, the reference bitrate, the motion vector magnitude, and the multimodal data, wherein the target encoder parameters include: quantization parameter values ​​or adaptive quantization values.

[0079] In this embodiment of the invention, based on the motion vector information of the video stream contained in the multimodal data packet, a convolutional neural network (CNN) is used to analyze and extract pixel changes between video frames by comparing the previous and subsequent frames. This allows for the accurate calculation of the magnitude of the motion vector, i.e., the intensity of the motion vector. The magnitude of the motion vector reflects the degree of movement of objects in the video. High-intensity motion vectors mean that the video content is dynamic and rich, requiring a higher bitrate to maintain video quality.

[0080] Then, based on the current network conditions, device performance, and the requirements of the audio and video content, the most suitable target encoder is intelligently selected from H.264, H.265 (an efficient video coding format), or VP9 encoders, and its corresponding bitrate range is determined. For example, for high dynamic range video content, H.265 might be chosen as the target encoder because it can maintain good video quality even at high compression ratios. Each encoder has preset minimum and maximum bitrate limits to ensure encoding quality and efficiency. After determining the target encoder and its bitrate range, the encoder parameters, namely quantization parameter values ​​(QP values) or adaptive quantization mode (AQ-mode), are intelligently calculated based on the encoder type of the target encoder, the currently determined baseline bitrate, the motion vector modulus, and the comprehensive multimodal data.

[0081] In this embodiment, the system can intelligently adjust the parameters of different encoders based on the dynamic characteristics of the video content to adapt to constantly changing network conditions and device states. For example, when transmitting high dynamic range video streams on VR devices, the system automatically selects a suitable encoder and increases the QP value to compensate for quality loss caused by motion, while also considering the high frame rate requirements of the device and the limitations of network bandwidth. This intelligent adjustment based on multimodal data not only improves the stability and quality of audio and video transmission but also effectively reduces encoding and network transmission costs, enhancing the user experience. Especially in the complex environment of 5G networks, the system can quickly respond to bandwidth fluctuations and changes in device performance, ensuring the smoothness and fidelity of audio and video transmission.

[0082] To further improve the accuracy of determining the target encoder parameters, in the audio and video bitrate adjustment method provided in Embodiment 1 of this application, when the encoder type is a first type, a first initial quantization parameter value is determined based on a first preset parameter, a reference bitrate, and a video motion vector. The first initial quantization parameter value is then adjusted based on the motion vector magnitude and the battery level in the multimodal data to obtain a first target quantization parameter value. When the encoder type is a second type, a second initial quantization parameter value is determined based on a second preset parameter, a reference bitrate, and a video motion vector. The second initial quantization parameter value is then adjusted based on the motion vector magnitude and the battery level in the multimodal data to obtain a second target quantization parameter value. When the encoder type is a third type, a target adaptive quantization value is determined based on a third preset parameter and the audio spectral entropy value in the multimodal data.

[0083] In this embodiment of the invention, when the encoder type is determined to be the first type (H.264), the first initial quantization parameter value (QP) is calculated based on the first preset parameters, the base bitrate, and the video motion vector. Specifically, for the H.264 parameter mapping: the base QP value is calculated as: 0.35 × decision bitrate + 0.18 × (MV_x² + MV_y²). If the motion vector magnitude is >45, the QP value is decreased by 2; if the device battery level is <20%, the QP value is increased by 2. Here, MV_x and MV_y are the horizontal and vertical components of the motion vector, reflecting the dynamic range of the video content.

[0084] For the second type of encoder (H.265), a second initial quantization parameter value can be determined based on a second preset parameter, a base bitrate, and video motion vectors. This second initial quantization parameter value is then adjusted based on the motion vector magnitude and battery level in the multimodal data to obtain the second target quantization parameter value. Specifically, for H.265 parameter mapping: the base QP value is calculated as: 0.32 × decision bitrate + 0.15 × (MV_x² + MV_y²). If the motion vector magnitude is >50, the QP value is decreased by 3; if the device battery level is <20%, the QP value is increased by 2.

[0085] The third type encoder (VP9) employs a different adjustment strategy. Based on a third preset parameter and the audio spectral entropy value in the multimodal data, it can directly determine the target adaptive quantization value (AQ-mode), i.e., AQ-mode value = min(round(1.5 × audio spectral entropy value), 4). This process does not require direct association with video motion vectors but focuses more on audio features. By flexibly adjusting the quantization mode of audio encoding, it ensures optimal audio quality in different scenarios (such as music, dialogue, etc.) while controlling the overall consumption of encoding resources.

[0086] Figure 6This is a schematic diagram of the cross-codec adapter module processing flow according to Embodiment 1 of the present invention, as shown below. Figure 6 As shown, the input parameters are: 3D decision parameters, motion vectors (MV_x, MV_y), audio spectrum entropy value, and device battery level. For different encoder types (H.264, H.265, or VP9), different strategies are used to calculate encoder-specific parameters (QP value / AQ-mode). Specifically: When the encoder type is H.264, QP is calculated first, and then runtime compensation is performed. That is, it is first determined whether the motion vector magnitude is greater than 45. If so, QP is decremented by 2; otherwise, QP remains unchanged. Then, it is determined whether the battery is low. If so, QP is incremented by 2; otherwise, QP is maintained, and finally, QP is output. When the encoder type is H.265, QP is calculated first, and then runtime compensation is performed. That is, it is first determined whether the motion vector magnitude is greater than 50. If so, QP is decremented by 3; otherwise, QP remains unchanged. Then, it is determined whether the battery is low. If so, QP is incremented by 2; otherwise, QP is maintained, and finally, QP is output. When the encoder type is VP9, ​​AQ-mode is calculated and output. Then, the output values ​​are accelerated and converted to output encoder-specific parameters.

[0087] In this embodiment, based on the characteristics of different encoder types and the real-time requirements reflected in multimodal data, the quantization parameter values ​​or adaptive quantization values ​​are dynamically adjusted to achieve optimal audio and video encoding results. For visual content-based encoders such as H.264 and H.265, the system intelligently adjusts the initial quantization parameter values ​​through refined motion vector analysis combined with the terminal's battery status to achieve the goal of ensuring both video quality and device power consumption. For the VP9 encoder, the adjustment focuses more on audio characteristics. Through the analysis of audio spectrum entropy values, the system can dynamically optimize the audio encoding strategy to ensure a clear and realistic sound experience even in situations with complex and varied audio content. This not only significantly improves the efficiency and quality of audio and video transmission but also effectively reduces the power consumption of terminal devices, extends usage time, and enhances the user experience. Especially in high-dynamic video scenarios and complex audio environments, the system can respond quickly and automatically adjust encoding parameters, avoiding the manual configuration and response lag problems of traditional methods.

[0088] To improve the accuracy of determining the target audio and video stream data, the audio and video bitrate adjustment method provided in Embodiment 1 of this application involves: starting the target encoder based on the target encoder parameters; determining multiple playback frames in the original audio and video stream data and identifying the frame type of each playback frame; when the frame type is the first frame type, allocating the highest bitrate to the playback frame through the target encoder, wherein the highest bitrate is the highest value in the bitrate range corresponding to the target encoder; when the frame type is the second frame type, adjusting the bitrate of the playback frame based on the bitrate range through the target encoder; identifying the audio scene of each playback frame, and adjusting the audio bitrate of the playback frame based on the audio scene to obtain the target audio and video stream data.

[0089] In this embodiment of the invention, H.264, H.265, or VP9 is selected as the target encoder based on the target encoder parameters. Upon startup, calculated quantization parameter values ​​(QP value or AQ-mode) and other parameters are passed to the encoder to guide optimal encoding. The selection of the target encoder is based on a comprehensive analysis of multimodal data, including network bandwidth, device status, and audio / video content characteristics. Then, the raw audio / video stream data is received and decomposed into a series of playback frames. For the video stream, based on the characteristics of intra-frame prediction (I-frame), inter-frame prediction (P-frame), and bidirectional prediction (B-frame), the playback frames are divided into three types: I-frames (first frame type), P-frames, or B-frames (second frame type). I-frames are complete images independent of any other frames and typically have the highest quality requirements, thus requiring a higher bitrate. P-frames and B-frames, on the other hand, depend on the prediction of the previous or subsequent frames, and inter-frame redundancy can be utilized during encoding to achieve good video quality at a lower bitrate.

[0090] For I-frames (first frame type), the system assigns the highest bitrate to these frames for encoding via a target encoder. The highest bitrate is the maximum value within the bitrate range corresponding to the target encoder; for example, for an H.264 encoder, the highest bitrate might be set to 8 Mbps. This ensures the image quality of I-frames, as they cannot obtain information from adjacent frames and require a sufficient bitrate to encode complete image details, which is crucial for video coherence and clarity.

[0091] For P-frames or B-frames (second frame type), the system dynamically adjusts the bitrate of the playback frames based on the target encoder's bitrate range. The adjustment strategy takes into account network conditions, device performance, and content complexity to ensure efficient bandwidth utilization while maintaining video quality. For example, when network bandwidth is low, the system will appropriately reduce the bitrate of P-frames or B-frames to decrease the bandwidth required for transmission, while simultaneously using motion compensation technology to maintain video smoothness.

[0092] In addition to encoding video frames, the system also intelligently adjusts the audio bitrate of each playback frame based on the audio scene identified in the raw audio and video stream data. For example, if the playback frame is identified as primarily featuring dialogue, the system will reduce the audio bitrate to 64Kbps to prioritize speech clarity; while if the playback frame is identified as containing a large amount of dynamic sound effects (such as background music for motion scenes), the system will increase the audio bitrate to 128Kbps or higher to ensure high-quality transmission of spatial sound and detail.

[0093] Figure 7 This is a schematic diagram of the streaming media control module processing flow according to Embodiment 1 of the present invention, as shown below. Figure 7 As shown, based on encoder-specific parameters, the encoder is started, and then the frame type of the frames in the input raw audio and video stream is identified. If it is an I-frame, the highest bitrate is allocated; if it is a P / B frame, the bitrate is dynamically adjusted. For audio grading control: in voice scenes, the audio bitrate is less than or equal to 64Kbps (to preserve clarity); in action scenes, the audio bitrate is greater than or equal to 128Kbps (to preserve spatial sense). Finally, the real-time bitstream is output to the terminal device for playback.

[0094] Through the above implementation method, the system can intelligently identify different types of frames in the raw audio and video stream data, and dynamically adjust the encoding parameters of video and audio based on the characteristics of the frames, network conditions, and device status. This refined encoding strategy ensures high-quality audio and video transmission under any network conditions, while reducing bandwidth consumption and terminal computing burden. In particular, high bitrate encoding of I-frames ensures video clarity and coherence; dynamic bitrate adjustment of P-frames and B-frames optimizes bandwidth usage; and finally, intelligent adjustment of the audio bitrate based on the audio scene guarantees audio realism and immersion. Overall, this implementation method significantly improves the user experience of audio and video services and promotes further optimization and evolution of audio and video transmission technology in the 5G era.

[0095] The audio / video bitrate adjustment method provided in this application adopts a method of real-time collection of target device performance data and network data. By integrating raw audio and video stream data to construct multimodal data packets, it achieves the purpose of accurately assessing the current playback environment, thereby realizing the technical effect of intelligent dynamic adjustment of audio and video bitrate. This solves the technical problems of response lag and inability to take into account the complex relationship between network, content, and device characteristics under fixed-rule bitrate adjustment. Specifically, it can collect indicators such as terminal GPU utilization and network bandwidth fluctuation. Then, it uses deep learning algorithms to analyze video motion intensity and audio spectral complexity to form multimodal data. After that, based on this data, it quickly predicts network conditions and content requirements, calculates the baseline bitrate, and then automatically converts the parameter settings of the target encoder according to the baseline bitrate and multimodal data to guide the streaming media control module to perform real-time encoding optimization of the raw audio and video stream. This ensures that high-quality audio and video can be played smoothly in various environments, effectively improving user experience and system efficiency.

[0096] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0097] Example 2

[0098] This application also provides an audio / video bitrate adjustment device. It should be noted that the audio / video bitrate adjustment device of this application can be used to execute the audio / video bitrate adjustment method provided in this application. The audio / video bitrate adjustment device provided in this application is described below.

[0099] According to an embodiment of this application, an apparatus for implementing the above-described audio and video bitrate adjustment method is also provided. Figure 8 This is a schematic diagram of an optional audio / video bitrate adjustment device according to an embodiment of the present invention, such as... Figure 8 As shown, the bitrate adjustment device may include: a collection unit 80, a construction unit 81, a determination unit 82, and an adjustment unit 83.

[0100] The acquisition unit 80 is used to acquire performance data and network data of the target device, and to obtain raw audio and video stream data.

[0101] Construction unit 81 is used to construct multimodal data based on performance data, network data, and raw audio and video stream data;

[0102] The determining unit 82 is used to determine the reference code rate based on the multimodal data, and to determine the target encoder parameters of the target encoder based on the reference code rate and the multimodal data.

[0103] The adjustment unit 83 is used to adjust the bitrate of the raw audio and video stream data using the target encoder based on the target encoder parameters to obtain target audio and video stream data, wherein the target audio and video stream data is used for playback on the target device.

[0104] The audio / video bitrate adjustment device provided in this application adopts a method of real-time acquisition of target device performance data and network data. By integrating raw audio and video stream data to construct multimodal data packets, it achieves the purpose of accurately assessing the current playback environment, thereby realizing the technical effect of intelligent dynamic adjustment of audio and video bitrate. This solves the technical problems of response lag and inability to take into account the complex relationship between network, content, and device characteristics under fixed-rule bitrate adjustment. Specifically, it can collect indicators such as terminal GPU utilization and network bandwidth fluctuations. Then, it uses deep learning algorithms to analyze video motion intensity and audio spectral complexity to form multimodal data. After that, based on this data, it quickly predicts network conditions and content requirements, calculates the baseline bitrate, and then automatically converts the parameter settings of the target encoder according to the baseline bitrate and multimodal data to guide the streaming media control module to perform real-time encoding optimization of the raw audio and video stream. This ensures that high-quality audio and video can be played smoothly in various environments, effectively improving user experience and system efficiency.

[0105] Optionally, the acquisition unit includes: a first acquisition module, used to acquire network data used by the target device through a preset probe, wherein the network data includes at least one of the following: bandwidth fluctuation data, latency data, and packet loss data; and a second acquisition module, used to acquire performance data of the target device through a preset interface, wherein the preset interface is used to monitor the device status of the target device.

[0106] Optionally, the construction unit includes: a first determining module, used to determine device status indicators based on performance data, wherein the device status indicators include at least one of the following: processor utilization, battery power, and screen resolution; a second determining module, used to determine network indicators based on network data, wherein the network indicators include at least one of the following: bandwidth fluctuation rate, latency, and packet loss rate; a first splitting module, used to split the original audio and video stream data to obtain video streams and audio; a first processing module, used to process the video stream to obtain video motion vectors; a second processing module, used to process the audio to obtain audio spectrum entropy values; and a first construction module, used to construct multimodal data based on device status indicators, network indicators, video motion vectors, and audio spectrum entropy values.

[0107] Optionally, the determining unit includes: a third processing module, used to process the multimodal data using a long short-term memory network to obtain the network state for the next time period; a third determining module, used to determine the video quality based on the video motion vectors in the multimodal data, and to determine the audio quality based on the audio spectral entropy value in the multimodal data; a fourth determining module, used to determine the dynamic compensation factor based on the device status indicators in the multimodal data; a fifth determining module, used to determine the network bandwidth for the next time period based on the network state; a sixth determining module, used to determine the device differentiation coefficient based on the device type of the target device; and a seventh determining module, used to determine the base bitrate based on the video quality, audio quality, dynamic compensation factor, network bandwidth, and device differentiation coefficient.

[0108] Optionally, the determining unit further includes: an eighth determining module, used to determine the motion vector magnitude based on the video motion vector in the multimodal data; a ninth determining module, used to determine the target encoder indicated by the reference bitrate, wherein each encoder corresponds to a bitrate range; and a tenth determining module, used to determine the target encoder parameters based on the encoder type of the target encoder, the reference bitrate, the motion vector magnitude, and the multimodal data, wherein the target encoder parameters include: quantization parameter values ​​or adaptive quantization values.

[0109] Optionally, the tenth determining module includes: a first determining submodule, used to determine a first initial quantization parameter value based on a first preset parameter, a reference bit rate, and a video motion vector when the encoder type is a first type, and to adjust the first initial quantization parameter value based on the motion vector magnitude and the battery level in the multimodal data to obtain a first target quantization parameter value; a second determining submodule, used to determine a second initial quantization parameter value based on a second preset parameter, a reference bit rate, and a video motion vector when the encoder type is a second type, and to adjust the second initial quantization parameter value based on the motion vector magnitude and the battery level in the multimodal data to obtain a second target quantization parameter value; and a third determining submodule, used to determine a target adaptive quantization value based on a third preset parameter and an audio spectral entropy value in the multimodal data when the encoder type is a third type.

[0110] Optionally, the adjustment unit includes: a first startup module for starting the target encoder based on the target encoder parameters; an eleventh determination module for determining multiple playback frames in the original audio and video stream data and identifying the frame type of each playback frame; a first allocation module for allocating the highest bitrate to the playback frame through the target encoder when the frame type is the first frame type, wherein the highest bitrate is the highest value in the bitrate range corresponding to the target encoder; a first adjustment module for adjusting the bitrate of the playback frame based on the bitrate range through the target encoder when the frame type is the second frame type; and a second adjustment module for identifying the audio scene of each playback frame and adjusting the audio bitrate of the playback frame based on the audio scene to obtain the target audio and video stream data.

[0111] The aforementioned bitrate adjustment device may also include a processor and a memory. The aforementioned acquisition unit 80, construction unit 81, determination unit 82, adjustment unit 83, etc., are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0112] The aforementioned processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured. By adjusting the kernel parameters, the target encoder is used to adjust the bitrate of the raw audio and video stream data based on the target encoder parameters, thus obtaining the target audio and video stream data.

[0113] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0114] It should be noted that the acquisition unit 80, construction unit 81, determination unit 82, and adjustment unit 83 mentioned above correspond to steps S201 to S204 in Embodiment 1. The instances and application scenarios implemented by the above units and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above units can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0115] Example 3

[0116] Embodiments of this application may provide an electronic device. Figure 9 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Figure 9 As shown, the electronic device may include: one or more ( Figure 9 (Only one is shown) processor 902, memory 904, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0117] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the audio / video bitrate adjustment method and apparatus in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned audio / video bitrate adjustment method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0118] The processor can access information and applications stored in the memory via the transmission device to perform the following steps: acquiring performance data and network data of the target device, and obtaining raw audio and video stream data; constructing multimodal data based on the performance data, network data, and raw audio and video stream data; determining the base bitrate based on the multimodal data, and determining the target encoder parameters based on the base bitrate and multimodal data; adjusting the bitrate of the raw audio and video stream data using the target encoder based on the target encoder parameters to obtain target audio and video stream data, wherein the target audio and video stream data is used for playback on the target device.

[0119] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: collect network data used by the target device through a preset probe to obtain network data, wherein the network data includes at least one of the following: bandwidth fluctuation data, latency data, and packet loss data; collect the performance data of the target device through a preset interface, wherein the preset interface is used to monitor the device status of the target device.

[0120] The processor can access information and applications stored in memory via a transmission device to perform the following steps: Based on performance data, determine device status metrics, including at least one of the following: processor utilization, battery level, and screen resolution; based on network data, determine network metrics, including at least one of the following: bandwidth fluctuation, latency, and packet loss rate; split the raw audio and video stream data to obtain video and audio streams; process the video stream to obtain video motion vectors; process the audio to obtain audio spectral entropy values; and construct multimodal data based on device status metrics, network metrics, video motion vectors, and audio spectral entropy values.

[0121] The processor can access information and applications stored in memory via a transmission device to execute the following steps: process multimodal data using a Long Short-Term Memory (LSTM) network to obtain the network state for the next time period; determine video quality based on video motion vectors in the multimodal data, and determine audio quality based on audio spectral entropy values ​​in the multimodal data; determine a dynamic compensation factor based on device status indicators in the multimodal data; determine the network bandwidth for the next time period based on the network state; determine the device differentiation coefficient based on the device type of the target device; and determine the baseline bitrate based on video quality, audio quality, dynamic compensation factor, network bandwidth, and device differentiation coefficient.

[0122] The processor can access information and applications stored in the memory via a transmission device to perform the following steps: determining the motion vector magnitude based on the video motion vectors in the multimodal data; determining the target encoder indicated by the reference bitrate, wherein each encoder corresponds to a bitrate range; and determining the target encoder parameters based on the encoder type of the target encoder, the reference bitrate, the motion vector magnitude, and the multimodal data, wherein the target encoder parameters include quantization parameter values ​​or adaptive quantization values.

[0123] The processor can access information and applications stored in the memory via a transmission device to perform the following steps: When the encoder type is type one, determine a first initial quantization parameter value based on a first preset parameter, a reference bit rate, and a video motion vector; adjust the first initial quantization parameter value based on the motion vector magnitude and battery power in the multimodal data to obtain a first target quantization parameter value; when the encoder type is type two, determine a second initial quantization parameter value based on a second preset parameter, a reference bit rate, and a video motion vector; adjust the second initial quantization parameter value based on the motion vector magnitude and battery power in the multimodal data to obtain a second target quantization parameter value; when the encoder type is type three, determine a target adaptive quantization value based on a third preset parameter and the audio spectral entropy value in the multimodal data.

[0124] The processor can access information and applications stored in memory via the transmission device to perform the following steps: Start the target encoder based on the target encoder parameters; determine multiple playback frames in the raw audio / video stream data and identify the frame type of each playback frame; if the frame type is the first frame type, allocate the highest bitrate to the playback frame using the target encoder, where the highest bitrate is the highest value in the bitrate range corresponding to the target encoder; if the frame type is the second frame type, adjust the bitrate of the playback frame based on the bitrate range using the target encoder; identify the audio scene of each playback frame and adjust the audio bitrate of the playback frame based on the audio scene to obtain the target audio / video stream data.

[0125] This application provides a scheme for adjusting the bitrate of audio and video. By collecting target device performance data and network data in real time, and integrating raw audio and video stream data to construct multimodal data packets, it achieves accurate assessment of the current playback environment. This enables intelligent dynamic adjustment of the audio and video bitrate, solving the technical problems of response lag and inability to consider the complex relationships between network, content, and device characteristics under fixed-rule bitrate adjustment. Specifically, it collects indicators such as terminal GPU utilization and network bandwidth fluctuations. Then, it uses deep learning algorithms to analyze video motion intensity and audio spectral complexity to form multimodal data. Based on this data, it quickly predicts network conditions and content requirements, calculates a baseline bitrate, and then automatically converts the target encoder's parameter settings according to the baseline bitrate and multimodal data. This guides the streaming media control module to perform real-time encoding optimization of the raw audio and video streams, ensuring smooth playback of high-quality audio and video in various environments, effectively improving user experience and system efficiency.

[0126] Those skilled in the art will understand that Figure 9 The structure shown is for illustrative purposes only. Electronic devices can also be terminal devices such as smartphones, tablets, PDAs, and mobile internet devices (MIDs). Figure 9 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 9 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 9 The different configurations shown.

[0127] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0128] Example 4

[0129] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the audio / video bitrate adjustment method provided in Embodiment 1.

[0130] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0131] This application also provides a computer program product that, when executed on a data processing device, is adapted to perform steps of an audio / video bitrate adjustment method.

[0132] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0133] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0134] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0135] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0136] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0137] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0138] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for adjusting the bitrate of audio and video, characterized in that, include: Collect performance data and network data of the target device, and obtain raw audio and video stream data; Multimodal data is constructed based on the performance data, the network data, and the raw audio and video stream data; Based on the multimodal data, a baseline bitrate is determined, and based on the baseline bitrate and the multimodal data, the target encoder parameters of the target encoder are determined. Based on the target encoder parameters, the target encoder is used to adjust the bitrate of the original audio and video stream data to obtain target audio and video stream data, wherein the target audio and video stream data is used for playback on the target device.

2. The bitrate adjustment method according to claim 1, characterized in that, The steps for collecting performance data and network data from the target device include: The network data is obtained by collecting network data used by the target device through a preset probe, wherein the network data includes at least one of the following: bandwidth fluctuation data, latency data, and packet loss data; The performance data of the target device is collected through a preset interface, wherein the preset interface is used to monitor the device status of the target device.

3. The bitrate adjustment method according to claim 1, characterized in that, The steps for constructing multimodal data based on the performance data, the network data, and the raw audio and video stream data include: Based on the performance data, device status indicators are determined, wherein the device status indicators include at least one of the following: processor utilization, battery level, and screen resolution. Based on the network data, network metrics are determined, wherein the network metrics include at least one of the following: bandwidth fluctuation rate, latency, and packet loss rate. The raw audio and video stream data is split to obtain video stream and audio; The video stream is processed to obtain video motion vectors; The audio is processed to obtain the audio spectrum entropy value; The multimodal data is constructed based on the device status indicators, the network indicators, the video motion vectors, and the audio spectral entropy values.

4. The bitrate adjustment method according to claim 1, characterized in that, The step of determining the baseline bitrate based on the multimodal data includes: The multimodal data is processed using a long short-term memory network to obtain the network state for the next time period; Based on the video motion vectors in the multimodal data, the video quality is determined, and based on the audio spectral entropy value in the multimodal data, the audio quality is determined. Based on the equipment status indicators in the multimodal data, determine the dynamic compensation factor; Based on the network status, determine the network bandwidth for the next time period; Based on the equipment type of the target equipment, determine the equipment differentiation coefficient; The baseline bitrate is determined based on the video quality, the audio quality, the dynamic compensation factor, the network bandwidth, and the device differentiation coefficient.

5. The bitrate adjustment method according to claim 1, characterized in that, The step of determining the target encoder parameters based on the reference bit rate and the multimodal data includes: Based on the video motion vectors in the multimodal data, determine the motion vector magnitude; Determine the target encoder indicated by the reference bit rate, wherein each encoder corresponds to a bit rate range; Based on the encoder type of the target encoder, the reference bit rate, the motion vector magnitude, and the multimodal data, the parameters of the target encoder are determined, wherein the parameters of the target encoder include: quantization parameter values ​​or adaptive quantization values.

6. The bitrate adjustment method according to claim 5, characterized in that, The step of determining the parameters of the target encoder based on the encoder type, the reference bit rate, the motion vector magnitude, and the multimodal data includes: When the encoder type is the first type, a first initial quantization parameter value is determined based on the first preset parameter, the reference bit rate, and the video motion vector, and the first initial quantization parameter value is adjusted based on the motion vector magnitude and the battery power in the multimodal data to obtain a first target quantization parameter value; When the encoder type is the second type, a second initial quantization parameter value is determined based on the second preset parameter, the reference bit rate, and the video motion vector. The second initial quantization parameter value is then adjusted based on the motion vector magnitude and the battery power in the multimodal data to obtain a second target quantization parameter value. When the encoder type is the third type, the target adaptive quantization value is determined based on the third preset parameter and the audio spectrum entropy value in the multimodal data.

7. The bitrate adjustment method according to claim 1, characterized in that, Based on the target encoder parameters, the step of adjusting the bitrate of the raw audio and video stream data using the target encoder to obtain the target audio and video stream data includes: Based on the target encoder parameters, the target encoder is started; Determine multiple playback frames in the raw audio and video stream data, and identify the frame type of each playback frame; When the frame type is the first frame type, the highest bitrate is assigned to the playback frame by the target encoder, wherein the highest bitrate is the highest value in the bitrate range corresponding to the target encoder; When the frame type is the second frame type, the bitrate of the playback frame is adjusted by the target encoder based on the bitrate range; Identify the audio scene of each playback frame, and adjust the audio bitrate of the playback frame based on the audio scene to obtain the target audio and video stream data.

8. An audio / video bitrate adjustment device, characterized in that, include: The acquisition unit is used to acquire performance data and network data of the target device, and to obtain raw audio and video stream data; A construction unit is used to construct multimodal data based on the performance data, the network data, and the raw audio and video stream data; The determining unit is configured to determine a reference bitrate based on the multimodal data, and to determine the target encoder parameters of the target encoder based on the reference bitrate and the multimodal data. An adjustment unit is used to adjust the bitrate of the original audio and video stream data using the target encoder based on the target encoder parameters to obtain target audio and video stream data, wherein the target audio and video stream data is used for playback on the target device.

9. A computer program product, characterized in that, The method includes a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the audio / video bitrate adjustment method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, The device includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the audio / video bitrate adjustment method according to any one of claims 1 to 7.

Images