Audio and video optimization processing method and system for handheld terminal

Abstract

The application discloses an audio and video optimization processing method and system for a handheld terminal, and relates to the technical field of multimedia signal processing, which comprises the following steps: independently of an audio coding process, a lightweight neural network model is used to perform real-time analysis on original audio data in parallel, and a structured audio semantic descriptor is output; in response to a selected current video coding strategy mode, a set of operation parameters of a video encoder is dynamically reconstructed to code a synchronously collected video frame; if the current video coding strategy mode is a speech active mode, the coding frame rate is increased and a region of interest coding for a face region is started; if the current video coding strategy mode is a music dominant mode, the coding resolution is increased; and if the current video coding strategy mode is a silent listening mode, the coding frame rate and resolution are reduced. Through dynamic video coding strategy adjustment based on audio semantics, the application realizes accurate balance of video quality, smoothness and resource consumption.

Description

Technical Field

[0001] This invention relates to the field of multimedia signal processing technology, and in particular to an audio and video optimization processing method and system for handheld terminals. Background Technology

[0002] With the rapid development of mobile communication technology and the widespread adoption of handheld devices, audio and video collaborative processing technology plays a core role in applications such as real-time communication and streaming media playback. Significant progress has been made in related technological fields, including the optimization of audio coding standards and video coding frameworks, as well as the deployment and application of lightweight neural network models in embedded systems. These technologies have improved the compression efficiency and real-time analysis capabilities of audio and video data, meeting the demands of handheld devices for low-latency and high-energy-efficiency processing. Meanwhile, adaptive adjustment mechanisms for video coding parameters, such as dynamic frame rate control based on scene complexity or motion detection, have been validated in industrial practice, providing fundamental performance support for handheld devices.

[0003] However, when handling audio-video collaborative optimization, existing technologies often rely on simple features of the video content for video encoding parameter adjustments, lacking in-depth utilization of audio semantic information. This leads to a mismatch in resource allocation because video encoding strategies cannot accurately coordinate with audio event types in complex acoustic event scenarios. Specifically, in voice-dominated scenarios, video encoding may not prioritize increasing the frame rate to enhance smoothness; in music scenarios, resolution may not be specifically improved to preserve details; and in silent scenarios, resource consumption may not be effectively reduced, thus limiting overall efficiency optimization and increasing the computational burden and power consumption of handheld terminals. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides an audio and video optimization processing method for handheld terminals to solve the problem of dynamically adjusting video encoding parameters based on audio semantics to optimize resource allocation in the audio and video processing of handheld terminals.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides an audio and video optimization processing method for a handheld terminal, comprising: independently of the audio encoding process, using a lightweight neural network model in parallel to perform real-time analysis of the raw audio data and outputting a structured audio semantic descriptor; dynamically selecting a current video encoding strategy mode from a plurality of predefined video encoding strategy modes based on the acoustic event type and audio complexity score within the structured audio semantic descriptor; the plurality of video encoding strategy modes including a voice-active mode, a music-dominant mode, and a silent listening mode; responding to the selected current video encoding strategy mode, dynamically reconstructing the operating parameter group of the video encoder to encode synchronously acquired video frames; if the current video encoding strategy mode is a voice-active mode, increasing the encoding frame rate and initiating region of interest encoding for the face region; if the current video encoding strategy mode is a music-dominant mode, increasing the encoding resolution; and if the current video encoding strategy mode is a silent listening mode, decreasing the encoding frame rate and resolution.

[0008] As a preferred embodiment of the audio and video optimization processing method for handheld terminals described in this invention, the lightweight neural network model pre-training steps are as follows:

[0009] A training dataset is constructed by collecting raw audio samples containing various acoustic events. Each raw audio sample in the training dataset is labeled according to the corresponding acoustic event type.

[0010] Construct a neural network structure that includes convolutional layers and fully connected layers;

[0011] The neural network structure is iteratively trained using the labeled training dataset until convergence, and the trained neural network structure is deployed on a handheld terminal as a lightweight neural network model.

[0012] As a preferred embodiment of the audio and video optimization processing method for handheld terminals described in this invention, the specific steps for outputting a structured audio semantic descriptor are as follows:

[0013] The original audio data is segmented into segments according to fixed time windows to generate continuous audio frames;

[0014] The continuous audio frames are sequentially input into a lightweight neural network model for forward computation, and the output is a sequence of acoustic event type classifications for each audio frame.

[0015] Simultaneously, the frequency domain features of each audio frame are calculated and converted into an audio complexity score;

[0016] The acoustic event type classification sequence is combined with the audio complexity score to form a structured audio semantic descriptor.

[0017] As a preferred embodiment of the audio and video optimization processing method for handheld terminals described in this invention, the acoustic event types include speech activity types generated when human voices are uttered, music-dominant types generated by instruments and singing with continuous melodies, and silent listening types generated by environmental noise and silent gaps.

[0018] As a preferred embodiment of the audio and video optimization processing method for handheld terminals described in this invention, the audio complexity score calculation steps are as follows:

[0019] Short-time Fourier transform is performed on the audio frame to obtain spectral amplitude information, and the entropy value of the spectral amplitude information is calculated as a frequency domain feature.

[0020] Extract the zero-crossing rate of the audio frame in the temporal domain as a temporal feature;

[0021] The audio complexity score is generated by weighted fusion of frequency domain features and time domain features.

[0022] As a preferred embodiment of the audio and video optimization processing method for handheld terminals described in this invention, the voice active mode refers to an encoding configuration method that prioritizes video smoothness; the music-dominant mode refers to an encoding configuration method that prioritizes video clarity; and the silent listening mode refers to an encoding configuration method that prioritizes resource consumption.

[0023] As a preferred embodiment of the audio and video optimization processing method for handheld terminals described in this invention, the specific steps for dynamically selecting the current video encoding strategy mode from a plurality of predefined video encoding strategy modes are as follows:

[0024] Receive acoustic event types from structured audio semantic descriptors and identify the basic pattern categories corresponding to the acoustic event types;

[0025] Based on the audio complexity score in the structured audio semantic descriptor, the parameter configuration of the base mode category is adjusted, and the adjusted base mode category is determined as the current video coding strategy mode.

[0026] As a preferred embodiment of the audio and video optimization processing method for handheld terminals described in this invention, the operating parameter group of the dynamically reconstructed video encoder encodes the synchronously acquired video frames, and the specific steps are as follows:

[0027] Read the configuration instructions for the current video encoding strategy mode;

[0028] Configure the frame rate control parameters, resolution control parameters, and bitrate allocation parameters of the video encoder according to the configuration instructions;

[0029] The video frames are encoded using the set frame rate control parameters, resolution control parameters, and bitrate allocation parameters as a parameter group.

[0030] As a preferred embodiment of the audio and video optimization processing method for handheld terminals described in this invention, wherein: increasing the encoding frame rate and initiating region of interest encoding for the face region refers to increasing the frame rate control parameter of the video encoder and initiating a non-uniform bitrate allocation operation for the face region; increasing the encoding resolution refers to increasing the resolution control parameter of the video encoder; and decreasing the encoding frame rate and resolution refers to decreasing the frame rate control parameter and resolution control parameter of the video encoder.

[0031] "Raising up" means setting the value of the corresponding control parameter in the video encoder to a level higher than the current control parameter value; "lowering down" means setting the value of the corresponding control parameter in the video encoder to a level lower than the current control parameter value.

[0032] Secondly, this invention provides an audio and video optimization processing system for a handheld terminal, comprising an audio analysis module, a mode decision module, an encoding control module, and a parameter execution module. The audio analysis module is used to independently and in parallel employ a lightweight neural network model to perform real-time analysis of the raw audio data, outputting a structured audio semantic descriptor. The mode decision module is used to dynamically select the current video encoding strategy mode from multiple predefined video encoding strategy modes based on the acoustic event type and audio complexity score within the structured audio semantic descriptor. The multiple video encoding strategy modes include a voice-active mode, a music-dominated mode, and a silent listening mode. The encoding control module is used to dynamically reconstruct the operating parameter group of the video encoder to encode synchronously acquired video frames in response to the selected current video encoding strategy mode. The parameter execution module is used to increase the encoding frame rate and initiate region of interest encoding for the face region if the current video encoding strategy mode is a voice-active mode; increase the encoding resolution if the current video encoding strategy mode is a music-dominated mode; and decrease the encoding frame rate and resolution if the current video encoding strategy mode is a silent listening mode.

[0033] The beneficial effects of this invention are as follows: By analyzing acoustic event types and audio complexity scores, it intelligently matches predefined video encoding strategy patterns and drives adaptive reconstruction of parameter groups. This ensures that the video encoder prioritizes increasing the frame rate and initiating face region encoding in voice-active mode, significantly enhancing the smoothness of video calls and preserving subject details. In music-dominated mode, it increases resolution, effectively enhancing the visual clarity of instrument textures and stage backgrounds. In silent listening mode, it reduces frame rate and resolution, minimizing processor load and bandwidth usage. Thus, without relying on external input, it accurately balances video quality, smoothness, and resource consumption, improving the overall performance stability and user experience consistency of handheld terminals. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart of an audio and video optimization processing method for handheld terminals.

[0036] Figure 2 A flowchart for generating structured audio semantic descriptors.

[0037] Figure 3 Dynamically select a flowchart for the video encoding strategy mode.

[0038] Figure 4 Flowchart for dynamic reconstruction of video encoder runtime parameter groups. Detailed Implementation

[0039] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0040] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0041] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0042] Reference Figures 1-4 This is one embodiment of the present invention, which provides an audio and video optimization processing method for a handheld terminal, including the following steps:

[0043] S1: Independent of the audio encoding process, a lightweight neural network model is used in parallel to perform real-time analysis of the raw audio data and output a structured audio semantic descriptor.

[0044] S1.1: Collect raw audio samples containing various acoustic events to form a training dataset. Label each raw audio sample in the training dataset according to the corresponding acoustic event type.

[0045] Specifically, raw audio samples containing various acoustic events are obtained from public audio databases or actual recording scenarios. The types of acoustic events covered by the raw audio samples include speech activity types generated by human voice pronunciation, music-dominated types generated by instruments and singing with sustained melodies, and silent listening types generated by environmental noise and silent gaps.

[0046] Use an audio annotation tool (such as Audacity or similar software) to add labels to each raw audio sample in the training dataset; the labels correspond to acoustic event types including speech activity type, music dominance type, and silent listening type.

[0047] S1.2: Construct a neural network structure that includes convolutional layers and fully connected layers.

[0048] Define a neural network structure, including an input layer, at least one convolutional layer, at least one fully connected layer, and an output layer. The input layer receives raw audio data, and the output layer generates acoustic event type classification probabilities.

[0049] It should be noted that the collected and labeled original audio samples are preprocessed to form original audio data; the original audio data serves as the input source for the input layer of the lightweight neural network model. Specifically, the original audio data is a time-domain signal that can be directly input into the neural network model after standardizing the format of the original audio samples (such as unifying the sampling rate and bit depth) and segmenting them.

[0050] Input layer: Accepts raw audio frame data (dimension is frame length × 1); Convolutional layer: Processes the input using a one-dimensional convolutional kernel to extract temporal features. The kernel size and stride follow standard settings (example: kernel size is 3, stride is 1); Fully connected layer: Receives the output feature map from the convolutional layer, performs linear transformation and non-linear activation (using the ReLU function); Output layer: Outputs a three-class probability vector (corresponding to speech activity type, music dominance type, and silent listening type) through the Softmax function.

[0051] It should be noted that the total number of layers in a neural network structure cannot exceed 5 layers to ensure lightweight characteristics.

[0052] S1.3: Use the labeled training dataset to iteratively train the neural network structure until convergence, and deploy the trained neural network structure on a handheld terminal as a lightweight neural network model.

[0053] Specifically, a supervised learning algorithm is used to train the neural network structure. The training process is based on the labeled training dataset, and convergence is achieved by optimizing the loss function.

[0054] The loss function used is the cross-entropy loss function, which is expressed as:

[0055] ;

[0056] In the formula, The loss value is used to quantify the degree of prediction error of a neural network structure on a single sample or a batch of samples. The total number of categories, For category indexing, Indicates real label About categories One-hot encoded value, Represents the logarithmic function. The input is a logarithmic function, indicating that the neural network structure predicts a category. The probability of;

[0057] Stochastic gradient descent is applied to optimize the loss function, with the learning rate set to the initial value.

[0058] The labeled training dataset is divided into a training set and a validation set with a fixed batch size. Training continues until the loss function on the validation set tends to stabilize. Specifically, the fluctuation of the validation set loss value in multiple consecutive training cycles is less than the convergence threshold and there is no significant decrease.

[0059] It should be noted that the convergence threshold is determined by calculating the standard deviation of the validation set loss value over the most recent consecutive training cycles, with an example value ranging from 0.001 to 0.005.

[0060] Export the converged neural network structure parameters into a lightweight format (such as TensorFlow Lite or CoreML) and integrate them into the runtime environment of the handheld terminal.

[0061] S1.4: Segment the raw audio data into fixed time windows to generate continuous audio frames; input the continuous audio frames into a lightweight neural network model for forward computation and output the acoustic event type classification sequence of each audio frame.

[0062] Specifically, raw audio data is captured from the audio input device, and the raw audio data is segmented using a fixed-length time window to generate continuous audio frames of equal length.

[0063] Each consecutive audio frame is taken as input and passed to a lightweight neural network model deployed on a handheld terminal to perform forward computation and generate a sequence of acoustic event type classifications.

[0064] It should be noted that the forward computation includes convolutional layer feature extraction, fully connected layer processing, and Softmax output; the Softmax output is a three-dimensional probability vector for each consecutive audio frame (index 0: probability of active speech type; index 1: probability of dominant music type; index 2: probability of silent listening type).

[0065] S1.5: Simultaneously calculate the frequency domain features of each audio frame and convert them into an audio complexity score; perform a short-time Fourier transform on the audio frame to obtain spectral amplitude information, and calculate the entropy value of the spectral amplitude information as a frequency domain feature quantity.

[0066] Specifically, the Hanning window function is applied to each consecutive audio frame for preprocessing. After preprocessing, a short-time Fourier transform is performed to convert the time-domain signal into a frequency-domain representation, and the spectral amplitude information is output.

[0067] It should be noted that the Hanning window function is a cosine window function used to window continuous audio frames in short-time Fourier transform to reduce spectral leakage and improve frequency resolution, thereby obtaining spectral amplitude information more accurately.

[0068] The spectral amplitude energy in the spectral amplitude information is normalized into a probability distribution, and the entropy value is calculated using the information entropy formula as a frequency domain feature. The information entropy formula is used to quantify the degree of disorder of the spectral amplitude information as a frequency domain feature to drive the subsequent weighted fusion with the zero-crossing rate to generate an audio complexity score.

[0069] S1.6: Extract the zero-crossing rate of audio frames in the temporal domain as a temporal feature; specifically, it counts the proportion of the number of times the symbols of adjacent samples change in consecutive audio frames to the total number of samples, and outputs the temporal feature.

[0070] The audio complexity score is generated by weighted fusion of frequency domain features and time domain features.

[0071] Specifically, frequency domain features and time domain features (zero-crossing rate) are fused according to preset weighting coefficients, which include weighting coefficients for both frequency domain features and time domain features. When the weighting coefficient for frequency domain features is greater than the weighting coefficient for time domain features, the fusion result is within the range of... The scalar value is used as the audio complexity score for the current consecutive audio frames.

[0072] It should be noted that the frequency domain feature weight coefficients are optimized and determined on the validation set using a grid search method to maximize the correlation between the audio complexity score and the perceived audio quality; the time domain feature weight coefficients are set proportionally based on the frequency domain feature weight coefficients, and the proportional setting must ensure that the frequency domain feature weight coefficients are greater than the time domain feature weight coefficients and that their sum is 1.

[0073] If the weight coefficient of the frequency domain feature is not greater than the weight coefficient of the time domain feature, the grid search optimization process is re-executed until the condition that the weight coefficient of the frequency domain feature is greater than the weight coefficient of the time domain feature is met. The constraint is based on the principle of acoustic feature priority to ensure the dominant position of the frequency domain feature in the complexity evaluation.

[0074] S1.7: Combine the acoustic event type classification sequence with the audio complexity score into a structured audio semantic descriptor.

[0075] Specifically, a structured audio semantic descriptor is generated by combining the acoustic event type classification sequence (three-dimensional probability vector) aligned with the same consecutive audio frames and the audio complexity score (scalar value).

[0076] The structured audio semantic descriptor is stored using a key-value pair data structure, containing two fields: "Acoustic Event Type Classification Sequence" field: storing the three-dimensional probability vector of all consecutive audio frames in chronological order; and "Audio Complexity Score" field: storing the scalar score value of all consecutive audio frames in chronological order.

[0077] S2: Based on the acoustic event type and audio complexity score within the structured audio semantic descriptor, dynamically select the current video coding strategy mode from multiple predefined video coding strategy modes; the multiple video coding strategy modes include speech active mode, music dominant mode, and silent listening mode;

[0078] It should be noted that the voice-active mode refers to an encoding configuration that prioritizes video smoothness; the music-dominant mode refers to an encoding configuration that prioritizes video clarity; and the silent listening mode refers to an encoding configuration that prioritizes resource consumption.

[0079] S2.1: Receive acoustic event types from the structured audio semantic descriptor, identify the basic mode category corresponding to the acoustic event type; based on the audio complexity score in the structured audio semantic descriptor, adjust the parameter configuration of the basic mode category, and determine the adjusted basic mode category as the current video coding strategy mode.

[0080] Specifically, the acoustic event type classification sequence generated after performing forward computation is obtained; the acoustic event type classification sequence is a probability vector sequence of multiple consecutive audio frames; each probability vector corresponds to an audio frame and contains three probability values: probability of active speech type, probability of dominant music type, and probability of silent listening type;

[0081] S2.2: Calculate the average probability distribution of the acoustic event type classification sequence.

[0082] The acoustic event type classification sequence is averaged to generate a single probability distribution, representing the dominant acoustic event type trend of the entire acoustic event type classification sequence;

[0083] Specifically, the three-dimensional probability vector contained in the acoustic event type classification sequence is represented as follows:

[0084] ;

[0085] in, Indicates the first The probability of speech activity type in a frame; Indicates the first Probability of the dominant music genre in a frame; Indicates the first The probability of the silent listening type of a frame.

[0086] For each type of acoustic event Calculate the average probability value The expression is:

[0087] ;

[0088] In the formula, The length of the acoustic event type classification sequence field represents the number of consecutive audio frames generated from segments of the original audio data.

[0089] S2.3: Based on the average probability distribution, determine the dominant acoustic event type and map it to a predefined video coding strategy pattern as the basic pattern category (such as speech activity pattern).

[0090] Specifically, the average probability values ​​are compared, and the type corresponding to the maximum value is selected. For example, if the probability of the voice active type is the highest, the basic mode category is voice active mode; if the probability of the voice active type is the highest, the basic mode category is music-dominant mode; if the probability of the voice active type is the highest, the basic mode category is silent listening mode.

[0091] S2.4: Extract the audio complexity score field from the received structured audio semantic descriptor. The audio complexity score is a scalar value, ranging from... The spaces between indicate the complexity of the audio signal.

[0092] Furthermore, predefined default parameter configurations are loaded for the identified basic mode categories; the default parameter values ​​for each video coding strategy mode are set based on the priority target settings of the corresponding video coding strategy mode.

[0093] The default parameter configurations are stored in the handheld terminal's memory in a structured form. Each video encoding strategy mode is stored independently, containing default target frame rate and default resolution parameter values. Specifically, these include: Voice active mode: default target frame rate (high value, prioritizing smoothness), default resolution (medium value); Music dominant mode: default target frame rate (medium value), default resolution (high value, prioritizing clarity); Silent listening mode: default target frame rate (low value), default resolution (low value, prioritizing resource consumption).

[0094] The default parameter values ​​of the base mode category are scaled using the audio complexity score to generate the adjusted parameter configuration; the scaling is based on a linear formula to ensure that the adjustment direction conforms to the priority target settings of the video coding strategy mode.

[0095] S2.5: Determine the adjusted base mode category as the current video coding strategy mode.

[0096] Specifically, the adjusted parameter configuration is bound to the basic mode category, and the output is a key-value pair to form the current video encoding strategy mode. The current video encoding strategy mode is a data structure that includes the mode type (such as voice active mode) and the adjusted parameter configuration (such as target frame rate and resolution).

[0097] S3: In response to the selected current video encoding strategy mode, dynamically reconstruct the operating parameter set of the video encoder to encode the synchronously acquired video frames.

[0098] S3.1: Read the configuration instructions for the current video encoding strategy mode.

[0099] Specifically, the data structure receives the current video encoding strategy mode; the data structure contains two fields: "Mode Type": string value ("Voice-Active Mode", "Music-Dominant Mode" or "Silent Listening Mode"); "Adjusted Parameter Configuration": key-value pairs (containing target frame rate and resolution);

[0100] Furthermore, the configuration instructions in the data structure are parsed and output as configuration instruction tuples.

[0101] Specifically, extract the value of the "Mode Type" field; extract the target frame rate and resolution values ​​from the "Adjusted Parameter Configuration"; and determine whether to enable face region encoding based on the "Mode Type" (only enable it in voice active mode).

[0102] After parsing the configuration instructions, a configuration instruction tuple is output; the configuration instruction tuple contains: mode type, target frame rate, resolution, and face encoding flag; where the face encoding flag is a Boolean value.

[0103] S3.2: Set the frame rate control parameters, resolution control parameters, and bitrate allocation parameters of the video encoder according to the configuration instructions.

[0104] Specifically, the target frame rate in the configuration command is directly mapped to the frame rate control parameter value; the resolution in the configuration command is directly mapped to the resolution control parameter value; the mapping method is specifically to assign the target frame rate and resolution parameter values ​​directly to the corresponding control registers of the video encoder through a one-to-one numerical mapping.

[0105] Based on the mode type and face encoding flag in the configuration instructions, determine the non-uniform bitrate allocation operation;

[0106] Specifically, when setting the bitrate allocation parameters, if the current video encoding strategy mode is the voice active mode and face region encoding needs to be started, a non-uniform bitrate allocation strategy is adopted, which allocates a higher weight to the face region (that is, allocates more than half of the total bitrate to the face region) and a lower weight to other regions (that is, allocates the remaining proportion to non-face regions and a proportion lower than that of the face region).

[0107] If the current video encoding strategy mode is music-dominated mode, silent listening mode, or voice-active mode but face region encoding is not required, then a uniform bitrate allocation strategy is adopted, that is, allocating the same bitrate ratio to all video regions; in the case of non-uniform allocation, the weight values ​​of face regions and other regions are recorded, and in the case of uniform allocation, they are marked as uniform allocation markers, to ensure the correct selection and execution of bitrate allocation strategies under different video encoding strategy modes.

[0108] S3.3: Use the set frame rate control parameters, resolution control parameters, and bitrate allocation parameters as a parameter group to encode the video frames.

[0109] Specifically, the system reads video frames that are strictly aligned with the current audio data timestamp from the camera buffer to ensure audio-video synchronization; and writes the frame rate control parameters into the frame rate controller of the video encoder to dynamically adjust the frame rate output of the video encoding to ensure consistency with the target frame rate of the current video encoding strategy mode.

[0110] The resolution control parameters are written into the resolution scaling component of the video encoder to adjust the encoding resolution of the video frames to meet the requirements of the current video encoding strategy mode. The resolution scaling component is the part of the video encoder that performs the resolution adjustment function, and it achieves resolution conversion by processing the pixel data of the video frames.

[0111] The appropriate operation is performed based on the type of bitrate allocation parameter: If the bitrate allocation parameter is non-uniform (such as voice active mode), a face detection algorithm is used to locate the face region in the video frame; the face region coordinates and corresponding weights are input into the bitrate allocator to increase the bitrate allocation ratio of the face region and decrease the bitrate allocation ratio of the background region; if the bitrate allocation parameter is uniform (such as music-dominated mode or silent listening mode), the default uniform bitrate allocation strategy is enabled, and no bitrate optimization is performed on specific regions.

[0112] The video encoder initializes the encoding pipeline based on the set frame rate control parameters, resolution control parameters, and bit rate allocation parameters; it then inputs the initialized encoding pipeline with video frames to the encoder, executes standard video encoding processes (such as intra-frame prediction, inter-frame prediction, transform quantization, and entropy coding), and outputs a compressed video bitstream, completing the encoding processing of the current video frame.

[0113] S4: If the current video encoding strategy mode is voice-active mode, increase the encoding frame rate and start encoding the region of interest for the face area; if the current video encoding strategy mode is music-dominant mode, increase the encoding resolution; if the current video encoding strategy mode is silent listening mode, decrease the encoding frame rate and resolution.

[0114] S4.1: Increasing the encoding frame rate and starting region of interest encoding for the face region refers to increasing the frame rate control parameter of the video encoder and starting non-uniform bit rate distribution operation for the face region; increasing means setting the value of the corresponding control parameter in the video encoder to a higher level than the current control parameter value.

[0115] Specifically, read the current frame rate control parameter values ​​of the video encoder;

[0116] Set the frame rate control parameter value to a level higher than the current value (e.g., if the current value is 25fps, set it to 30fps), and write the updated frame rate control parameter value into the frame rate controller of the video encoder.

[0117] The face detection algorithm is activated to perform real-time face region localization on the input video frame. The face region coordinates are then input into the bitrate allocator of the video encoder for non-uniform bitrate allocation.

[0118] It should be noted that frame rate enhancement ensures the temporal smoothness of the video stream to avoid motion stuttering in voice scenarios, while non-uniform bitrate distribution enhances the clarity of the subject in the spatial domain. Frame rate enhancement and non-uniform bitrate distribution work together to serve the encoding goals of prioritizing smoothness and subject detail in voice scenarios. The efficiency of non-uniform bitrate distribution ensures that handheld terminals maintain stable output with low power consumption and high performance in complex acoustic scenarios, thereby fully realizing the end-to-end optimization closed loop from audio semantic perception to video encoding strategy execution.

[0119] Furthermore, in voice-active mode, by activating the face detection algorithm to perform real-time face region localization on the input video frame and inputting the face region coordinates into the bitrate allocator of the video encoder, it is possible to effectively identify key facial regions in the video frame, thereby achieving precise bitrate allocation optimization for key regions, ensuring that facial details are preserved with higher quality during the encoding process, while reducing bitrate waste in non-critical regions, ultimately maintaining a balance between improving the clarity of the main subject of the video call and maintaining overall encoding efficiency.

[0120] S4.2: Increasing the encoding resolution refers to adjusting the resolution control parameters of the video encoder.

[0121] Specifically, read the current resolution control parameter values ​​of the video encoder;

[0122] Set the resolution control parameter value to a level higher than the current value (e.g., if the current value is 720p, set it to 1080p), and write the updated resolution control parameter value to the resolution scaling component of the video encoder.

[0123] In particular, increasing the resolution control parameters of the video encoder can significantly improve the clarity and detail of video content in the audio and video optimization processing method of handheld terminals, especially in music-dominated mode. Specifically, when the audio analysis identifies the music-dominated type, increasing the resolution control parameters allows the video encoder to process video frames at a higher resolution, which can better preserve high-frequency details such as instrument textures and stage backgrounds in the music performance, thus meeting the user's demand for high-quality music videos.

[0124] Furthermore, by dynamically adjusting the resolution based on audio semantic analysis, the resource waste caused by fixed high resolution is avoided. While ensuring the optimization of visual experience, the efficient utilization of handheld terminal resources is achieved. This audio-video collaborative optimization processing method effectively solves the problem that traditional fixed parameter encoding is difficult to balance quality and efficiency in complex scenarios.

[0125] S4.3: Lowering the encoding frame rate and resolution means reducing the frame rate control parameters and resolution control parameters of the video encoder; reducing means setting the values ​​of the corresponding control parameters in the video encoder to a level lower than the current control parameter values.

[0126] Specifically, read the current frame rate control parameter values ​​and resolution control parameter values ​​of the video encoder;

[0127] Set the frame rate control parameter value to a level lower than the current value (for example, if the current value is 30fps, set it to 15fps), and write the updated frame rate control parameter value into the frame rate controller of the video encoder.

[0128] Set the resolution control parameter value to a level lower than the current value (e.g., if the current value is 1080p, set it to 480p), and write the updated resolution control parameter value to the resolution scaling component of the video encoder.

[0129] Ideally, reducing the frame rate control parameters and resolution control parameters of the video encoder can effectively reduce the consumption of computing resources and bandwidth usage.

[0130] Specifically, when audio analysis identifies a silent listening type, it intelligently lowers the frame rate and resolution parameters, maintaining basic video continuity while significantly reducing processor load and power consumption, extending the battery life of the handheld terminal in low-activity scenarios. At the same time, this dynamic parameter reduction optimization method avoids the stuttering or blurring problems caused by traditional fixed low parameters, achieving the best balance between resource consumption and user experience, and fully demonstrating the intelligent advantages of audio and video collaborative processing.

[0131] This embodiment also provides an audio and video optimization processing system for a handheld terminal, including: an audio analysis module, a mode decision module, an encoding control module, and a parameter execution module; the audio analysis module is used to independently and in parallel use a lightweight neural network model to perform real-time analysis on the raw audio data and output a structured audio semantic descriptor; the mode decision module is used to dynamically select the current video encoding strategy mode from multiple predefined video encoding strategy modes based on the acoustic event type and audio complexity score in the structured audio semantic descriptor; the multiple video encoding strategy modes include a voice-active mode, a music-dominant mode, and a silent listening mode; the encoding control module is used to dynamically reconstruct the operating parameter group of the video encoder to encode the synchronously acquired video frames in response to the selected current video encoding strategy mode; the parameter execution module is used to increase the encoding frame rate and start encoding the region of interest of the face region if the current video encoding strategy mode is a voice-active mode; increase the encoding resolution if the current video encoding strategy mode is a music-dominant mode; and decrease the encoding frame rate and resolution if the current video encoding strategy mode is a silent listening mode.

[0132] This embodiment also provides a computer device applicable to the audio and video optimization processing method for handheld terminals, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the audio and video optimization processing method for handheld terminals as proposed in the above embodiment.

[0133] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0134] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the audio and video optimization processing method for a handheld terminal as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0135] In summary, this invention, by analyzing acoustic event types and audio complexity scores, intelligently matches predefined video encoding strategy patterns and drives adaptive reconstruction of parameter groups. This ensures that the video encoder prioritizes increasing the frame rate and initiating face region encoding in voice-active mode, significantly enhancing the smoothness of video calls and preserving subject details; it increases resolution in music-dominated mode, effectively enhancing the visual clarity of instrument textures and stage backgrounds; and it reduces frame rate and resolution in silent listening mode, minimizing processor load and bandwidth consumption. Thus, without relying on external input, it accurately balances video quality, smoothness, and resource consumption, improving the overall performance stability and user experience consistency of handheld terminals.

[0136] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for audio and video optimization processing for handheld terminals, characterized in that: include, Independent of the audio encoding process, a lightweight neural network model is used in parallel to perform real-time analysis of the raw audio data and output structured audio semantic descriptors; The pre-training steps for the lightweight neural network model are as follows. A training dataset is constructed by collecting raw audio samples containing various acoustic events. Each raw audio sample in the training dataset is labeled according to the corresponding acoustic event type. Construct a neural network structure that includes convolutional layers and fully connected layers; The neural network structure is iteratively trained using the labeled training dataset until convergence, and the trained neural network structure is deployed on a handheld terminal as a lightweight neural network model. Based on the acoustic event type and audio complexity score within the structured audio semantic descriptor, the current video coding strategy mode is dynamically selected from multiple predefined video coding strategy modes; the multiple video coding strategy modes include speech active mode, music dominant mode, and silent listening mode; In response to the selected current video encoding strategy mode, the operating parameter set of the video encoder is dynamically reconstructed to encode the synchronously acquired video frames; If the current video encoding strategy mode is voice-active mode, then increase the encoding frame rate and start region of interest encoding for the face region; If the current video encoding strategy mode is music-driven mode, the encoding resolution is increased; if the current video encoding strategy mode is silent listening mode, the encoding frame rate and resolution are reduced. Increasing the encoding frame rate and initiating region of interest encoding for the face region refers to increasing the frame rate control parameters of the video encoder and initiating a non-uniform bitrate distribution operation for the face region; increasing the encoding resolution refers to increasing the resolution control parameters of the video encoder; decreasing the encoding frame rate and resolution refers to decreasing the frame rate control parameters and resolution control parameters of the video encoder. "Raising up" means setting the value of the corresponding control parameter in the video encoder to a level higher than the current control parameter value; "lowering down" means setting the value of the corresponding control parameter in the video encoder to a level lower than the current control parameter value.

2. The audio and video optimization processing method for handheld terminals as described in claim 1, characterized in that: The specific steps for outputting the structured audio semantic descriptor are as follows: The original audio data is segmented into segments according to fixed time windows to generate continuous audio frames; The continuous audio frames are sequentially input into a lightweight neural network model for forward computation, and the output is a sequence of acoustic event type classifications for each audio frame. Simultaneously, the frequency domain features of each audio frame are calculated and converted into an audio complexity score; The acoustic event type classification sequence is combined with the audio complexity score to form a structured audio semantic descriptor.

3. The audio and video optimization processing method for handheld terminals as described in claim 2, characterized in that: The acoustic event types include speech activity types generated by human voice pronunciation, music-dominant types generated by instruments and singing producing continuous melodies, and silent listening types generated by environmental noise and silent gaps.

4. The audio and video optimization processing method for handheld terminals as described in claim 3, characterized in that: The steps for calculating the audio complexity score are as follows: Short-time Fourier transform is performed on the audio frame to obtain spectral amplitude information, and the entropy value of the spectral amplitude information is calculated as a frequency domain feature. Extract the zero-crossing rate of the audio frame in the temporal domain as a temporal feature; The audio complexity score is generated by weighted fusion of frequency domain features and time domain features.

5. The audio and video optimization processing method for handheld terminals as described in claim 4, characterized in that: The voice-active mode refers to an encoding configuration that prioritizes video smoothness; the music-dominant mode refers to an encoding configuration that prioritizes video clarity; and the silent listening mode refers to an encoding configuration that prioritizes resource consumption.

6. The audio and video optimization processing method for a handheld terminal as described in claim 5, characterized in that: The specific steps for dynamically selecting the current video encoding strategy mode from multiple predefined video encoding strategy modes are as follows: Receive acoustic event types from structured audio semantic descriptors and identify the basic pattern categories corresponding to the acoustic event types; Based on the audio complexity score in the structured audio semantic descriptor, the parameter configuration of the base mode category is adjusted, and the adjusted base mode category is determined as the current video coding strategy mode.

7. The audio and video optimization processing method for a handheld terminal as described in claim 6, characterized in that: The operating parameter group of the dynamically reconstructed video encoder encodes the synchronously acquired video frames. The specific steps are as follows. Read the configuration instructions for the current video encoding strategy mode; Configure the frame rate control parameters, resolution control parameters, and bitrate allocation parameters of the video encoder according to the configuration instructions; The video frames are encoded using the set frame rate control parameters, resolution control parameters, and bitrate allocation parameters as a parameter group.

8. An audio and video optimization processing system for a handheld terminal, based on the audio and video optimization processing method for a handheld terminal according to any one of claims 1 to 7, characterized in that: It includes an audio analysis module, a mode decision module, an encoding control module, and a parameter execution module; The audio analysis module is used to perform real-time analysis of the raw audio data in parallel using a lightweight neural network model, independent of the audio encoding process, and output a structured audio semantic descriptor. The mode decision module is used to dynamically select the current video coding strategy mode from a plurality of predefined video coding strategy modes based on the acoustic event type and audio complexity score in the structured audio semantic descriptor; the plurality of video coding strategy modes include speech active mode, music dominant mode and silent listening mode; The encoding control module is used to dynamically reconstruct the operating parameter group of the video encoder to encode the synchronously acquired video frames in response to the selected current video encoding strategy mode. The parameter execution module is used to increase the encoding frame rate and start region of interest encoding for the face region if the current video encoding strategy mode is voice active mode. If the current video encoding strategy mode is music-driven, the encoding resolution will be increased; if the current video encoding strategy mode is silent listening, the encoding frame rate and resolution will be reduced.

Images