Audio and video real-time acquisition uploading and data processing method

Abstract

The application discloses a kind of audio and video real-time acquisition upload and data processing method, belong to audio and video data processing and network transmission technical field.The method includes: collecting audio and video original data, and dividing key frame and difference frame by interframe difference analysis;Two-dimensional discrete wavelet transform is carried out to key frame, and after motion compensation residual calculation is carried out to difference frame, two-dimensional discrete wavelet transform is executed;Adaptive quantization of quantization step according to the previous frame quantization error feedback adjustment is executed to wavelet coefficient;After quantization, coefficient is generated compressed code stream by run-length encoding and huffman encoding;According to frame type and importance slice and after inserting time stamp and frame serial number, it is uploaded in real time by UDP protocol;Receiving end carries out retransmission request to data packet by buffering and ordering according to frame serial number to missing data packet, and during retransmission waiting period, linear interpolation recovery is carried out to missing area based on adjacent frame;Finally, audio and video data is output after decoding, inverse quantization, inverse wavelet transform, deblocking effect filtering and color correction.

Description

Technical Field

[0001] This invention relates to the field of audio and video data processing and network transmission technology, specifically to a method for real-time audio and video acquisition, uploading, and data processing. Background Technology

[0002] In the field of real-time audio and video acquisition and transmission, network bandwidth fluctuations and packet loss are the main factors affecting service quality. Most existing real-time transmission solutions employ video coding standards such as H.264 / HEVC combined with UDP transmission, and use buffering and retransmission mechanisms at the receiving end to combat packet loss. In these solutions, the encoder typically adjusts quantization parameters based on a preset bitrate control strategy, such as adjusting according to buffer fullness or estimated channel bandwidth. This adjustment method suffers from feedback lag; when scene content changes abruptly or network conditions deteriorate instantaneously, the quantization parameters cannot adapt in time, leading to severe jitter or blockiness in the reconstructed image quality. Furthermore, fixed-rate bitrate control struggles to simultaneously ensure image smoothness and bitrate utilization in low-latency scenarios.

[0003] Common methods for handling UDP packet loss include Automatic Repeat Request (ATR) and Forward Error Correction (FEC). ATR requests a retransmitted packet from the sender upon detecting packet loss, introducing at least one round-trip delay. For interactive real-time communication, this additional delay can disrupt audio-visual synchronization and interactive continuity. FEC adds redundant packets, allowing the receiver to recover a certain percentage of lost packets. However, redundant data consumes extra bandwidth, resulting in wasted bandwidth under good network conditions, and insufficient recovery capability during sudden, continuous packet loss. Error concealment techniques based on content features can fill in lost areas at the decoding end using information from received frames. However, when loss occurs in key frames or scene transitions, the correlation between adjacent frames is weak, and simply using motion compensation or frame copying will cause extensive motion blur and significantly reduce the subjective visual experience.

[0004] At the encoding level, while conventional wavelet transform coding offers advantages in multi-resolution analysis, its quantization stage often employs a fixed quantization matrix or relies solely on static weight allocation based on sub-band energy, failing to create a closed loop with reconstruction quality. When network bandwidth limitations increase, the encoder is forced to increase the global quantization step size, resulting in uniform detail loss across all sub-bands and making it difficult to differentiate distortion based on the importance of image content. In real-time acquisition and upload scenarios, the computational power and energy efficiency constraints at the acquisition end also require compression algorithms to achieve high compression ratios with low complexity. Existing deep learning methods or block-matching full search algorithms, while highly efficient in encoding, suffer from high computational overhead, making stable real-time operation difficult on low-power devices.

[0005] The problem this invention aims to solve is how to achieve real-time recovery of packet loss areas in low-latency audio and video streams without significantly increasing waiting time in an unreliable UDP transmission environment, and how to enable the quantization step size to self-adjust in a timely manner according to the coding quality, so as to avoid unexpected fluctuations in the quality of the reconstructed image. Summary of the Invention

[0006] The purpose of this invention is to provide a method for real-time audio and video acquisition, uploading, and data processing, resolving the conflict between increased latency due to retransmission waiting and bandwidth waste caused by forward error correction in existing UDP packet loss recovery technologies. It also addresses the problem of lag in quantization adjustment leading to fluctuations in reconstruction quality. By dividing keyframes and differential frames through inter-frame differential analysis and performing two-dimensional discrete wavelet transform, combined with adaptive quantization driven by feedback from the previous frame's quantization error, and a mechanism at the receiving end to fill missing pixel blocks based on bilinear interpolation of adjacent frames after retransmission timeout, this method improves network packet loss resistance and the stability of reconstructed images while ensuring real-time performance, and reduces the sensitivity of encoding quality to instantaneous bandwidth changes.

[0007] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a method for real-time audio and video acquisition, uploading, and data processing, comprising: acquiring raw audio and video data; performing inter-frame difference analysis on the raw audio and video data; dividing the frames into keyframes and difference frames according to the difference results; performing two-dimensional discrete wavelet transform on the keyframes; performing motion compensation residual calculation on the difference frames and then performing two-dimensional discrete wavelet transform to obtain wavelet coefficients; performing adaptive quantization on the wavelet coefficients, wherein the quantization step size is adjusted according to the quantization error feedback of the previous frame; and performing run-length encoding and Huffman coding on the quantized coefficients. The code is used to generate a compressed bitstream; the compressed bitstream is segmented according to frame type and importance, and a timestamp and frame sequence number are inserted into each data segment, which is then uploaded in real time via UDP protocol; the receiving end buffers and sorts the received data packets according to the frame sequence number, initiates a retransmission request for missing data packets, and performs linear interpolation recovery on the missing regions based on adjacent frames during the retransmission waiting period; Huffman decoding and inverse run-length encoding are performed on the recovered bitstream, followed by inverse quantization, and then inverse two-dimensional discrete wavelet transform to obtain the reconstructed frame; deblocking filtering and color correction are performed on the reconstructed frame to output the final audio and video data.

[0008] By dynamically dividing keyframes and difference frames through inter-frame differential analysis and introducing motion-compensated residual calculations into the difference frames, temporal redundancy can be effectively eliminated, improving compression efficiency. The multi-resolution analysis characteristics of two-dimensional discrete wavelet transform help preserve image edge and texture information, avoiding the block effect produced by traditional block transform at high compression ratios. Adaptive quantization adjusts the step size based on the quantization error feedback from the previous frame, achieving a balance between bitrate control and reconstruction quality, especially enabling rapid convergence and avoiding quality fluctuations when scene changes drastically. Run-length encoding of the quantization coefficients, combined with a dynamically generated Huffman code table, further removes statistical redundancy, resulting in a more compact compressed bitstream.

[0009] As a technical solution of the present invention, in the data transmission stage, the compressed bitstream of the key frame is divided into a first data slice with a shorter length, and the compressed bitstream of the differential frame is divided into a second data slice with a variable length. This allows the more important key frames to receive finer-grained fragment protection, reducing the risk of a large amount of data being undecoded due to the loss of a single packet. Each data slice header carries a timestamp, frame sequence number, and check field, and is sent in real time via the UDP protocol. The sending rate can be dynamically adjusted according to the sending buffer occupancy rate, effectively coping with network jitter and ensuring low-latency transmission.

[0010] The receiving end maintains a preset-sized receiving window, sorts and reassembles data packets according to frame sequence numbers, and immediately generates a retransmission request for any detected gaps, sending it back to the sending end. During the timeout period while waiting for retransmission, if the retransmitted data packet does not arrive in time, the corresponding pixel block is extracted from an adjacent, fully received frame and bilinear interpolation is used to fill the missing area. This mechanism ensures image quality continuity while avoiding playback stuttering caused by waiting for retransmissions, enhancing the real-time performance and smoothness of the video.

[0011] Preferably, in the decoding and reconstruction stage, the compressed bitstream is decoded and inverse run-length expanded using the same Huffman code table as the encoder. Inverse quantization is performed according to the subband quantization step size, and inverse two-dimensional discrete wavelet transform is performed on the recovered wavelet coefficients. For keyframes, the reconstructed frame is obtained directly, and for differential frames, the temporal pixel matrix obtained by the inverse transform is added to the motion-compensated prediction frame to obtain the complete reconstructed frame. This process closely corresponds to the encoder stage, ensuring reconstruction accuracy.

[0012] By detecting the pixel gradients on both sides of the macroblock boundary and adaptively applying a three-tap smoothing filter to the reconstructed frame, the block effect that may be caused by quantization and motion compensation is significantly reduced. The filtered frame is converted from the YUV color space to the RGB space, and white balance is adjusted for each channel to make the pixel mean uniform and significantly improve the subjective visual quality.

[0013] As a further improvement of the present invention, the audio signal is pre-emphasized and framed at the acquisition end, the spectrum is obtained by short-time Fourier transform and quantization encoding based on psychoacoustic model is performed to generate an audio bitstream; the audio bitstream and the compressed video bitstream are multiplexed into the same UDP data stream according to timestamp alignment; after demultiplexing at the receiving end, the audio bitstream is inversely quantized and inverse short-time Fourier transform is performed to obtain the reconstructed audio signal, which is output synchronously with the video, realizing integrated real-time acquisition, efficient compression, robust transmission and synchronous presentation of audio and video.

[0014] This invention also provides a real-time audio and video acquisition, uploading, and data processing system, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of the aforementioned real-time audio and video acquisition, uploading, and data processing method. This system features high compression ratio, low latency, strong packet loss resistance, and good reconstructed image quality, making it suitable for scenarios such as remote monitoring and real-time interactive live streaming.

[0015] The technical effects and advantages provided by the present invention in the above technical solution are as follows: Two-dimensional discrete wavelet transform is performed on keyframes. Motion compensation residuals are calculated on difference frames, followed by another two-dimensional discrete wavelet transform. Within the wavelet domain, coefficients are grouped into sub-bands. The quantization step size of each sub-band is adjusted based on the mean square error (MSE) between the reconstructed coefficients and the original coefficients of that sub-band in the previous frame. When the MSE of a sub-band exceeds a set threshold, the quantization step size is increased in the next frame; conversely, it is decreased, with the adjustment magnitude being an integer multiple of the preset step size increment. This closed-loop adjustment method allows the quantization step size to directly reflect the magnitude of distortion introduced by encoding. Sub-bands with increased distortion are quickly suppressed, while sub-bands with less distortion receive finer quantization. Distortion measurement and step size adjustment occur within frame-level intervals, eliminating the need to wait for multi-frame statistics or network state estimation. This significantly improves the response speed to sudden changes in scene content and complex local texture regions compared to indirect adjustment methods based on bandwidth estimation. Under the same bitrate constraint, the energy distribution of the reconstructed image is closer to the original signal. Edge flicker caused by insufficient quantization and loss of detail caused by overquantization are effectively suppressed, thereby reducing the intensity requirement of subsequent deblocking filtering and reducing the overall computational cost.

[0016] Upon receiving a data packet, the receiving end sorts the packets according to their frame sequence number within a preset window. For any detected sequence number gaps, a retransmission request is immediately initiated, and a waiting timer is started. If a retransmission packet is received before the timer expires, the gap is filled normally. If the timer expires, the corresponding pixel block is extracted directly from the adjacent frames that have been fully received, and the missing pixel value is calculated and filled using bilinear interpolation. This process integrates retransmission attempts with error concealment. The retransmission waiting time is set to a controllable window; after the timeout, the interpolated content replaces the missing portion. This avoids indefinite waiting for retransmission packets, which can cause stuttering, and also avoids hastily activating concealment while retransmission opportunities still exist, leading to unnecessary image quality degradation. Because the interpolation operation only utilizes adjacent frames that have been fully received and does not involve complex motion estimation or deep learning inference, the computation is extremely lightweight and can be completed instantly in the packet receiving thread without introducing additional decoding latency. When there is continuous packet loss or missing key frame data, this mechanism can still reduce abrupt block loss traces through the smooth spatial transition of adjacent reconstructed frames, thus limiting the visual interference caused by packet loss to a lower level. Combined with the stable bitstream characteristics of wavelet domain quantization feedback, the overall end-to-end delay jitter is improved. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0018] Figure 1 This is a flowchart of the real-time audio and video acquisition, uploading, and data processing method; Figure 2 It is a flowchart of video coding based on inter-frame difference analysis and adaptive quantization; Figure 3 This is a flowchart of run-length encoding for quantization coefficients, Huffman coding, and UDP transmission. Figure 4 It is a curve showing the change in the average pixel value of the RGB channels before and after color correction; Figure 5 It shows the curves of processor utilization at the sending and receiving ends. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] See Figure 1 This invention provides a method for real-time audio and video acquisition, uploading, and data processing, comprising: acquiring raw audio and video data; performing inter-frame differential analysis on the raw audio and video data; dividing the frames into keyframes and differential frames based on the differential results; performing two-dimensional discrete wavelet transform on the keyframes; performing motion compensation residual calculation on the differential frames and then performing two-dimensional discrete wavelet transform to obtain wavelet coefficients; performing adaptive quantization on the wavelet coefficients, with the quantization step size adjusted according to the quantization error feedback of the previous frame; performing run-length encoding and Huffman encoding on the quantized coefficients to generate a compressed bitstream; segmenting the compressed bitstream according to frame type and importance, inserting a timestamp and frame sequence number into each data segment, and uploading it in real-time via the UDP protocol; the receiving end buffering and sorting the received data packets according to the frame sequence number, initiating retransmission requests for missing data packets, and performing linear interpolation recovery on the missing regions based on adjacent frames during the retransmission waiting period; performing Huffman decoding and inverse run-length encoding on the recovered bitstream, then performing inverse quantization, and finally performing inverse two-dimensional discrete wavelet transform to obtain the reconstructed frame. Deblocking filtering and color correction are performed on the reconstructed frames to output the final audio and video data.

[0021] Example 1: In specific implementation, please refer to Figure 2 The process of performing inter-frame differential analysis on the raw audio and video data and dividing the frames into keyframes and differential frames based on the differential results is as follows: When calculating the sum of absolute pixel differences between the current frame and the previous frame, the absolute value of the difference between the luminance component value of each pixel in the current frame and the luminance component value of a pixel at the same spatial position in the previous frame is accumulated pixel by pixel to obtain the sum of absolute pixel differences. This sum of absolute pixel differences is then compared with a pre-set first threshold. If the sum of absolute pixel differences is greater than the first threshold, the current frame is marked as a keyframe; if the sum of absolute pixel differences is not greater than the first threshold, the current frame is marked as a differential frame. The first threshold is determined based on the noise level of the acquisition scene and the acceptable frame type switching frequency. For example, in a fixed indoor scene, its value can be set to 5% of the sum of the maximum possible pixel values ​​of the entire frame.

[0022] For the current frame marked as a differential frame, motion vectors are calculated using translational motion estimation based on optical flow. Optical flow-based translational motion estimation relies on the assumption of constant brightness and translational constraints on motion, establishing optical flow constraint equations for each pixel in the differential frame. For the set of pixels within a neighborhood window centered on each pixel, the optical flow constraint equations are solved simultaneously to form an overdetermined system of equations. The closed-form solution of this system is obtained using the linear least squares method, yielding the motion vector representing the global translational motion of the entire frame. A motion-compensated prediction frame is generated using the solved motion vector and the previous frame. The generation process is as follows: for each pixel coordinate in the previous frame, the corresponding component of the motion vector is subtracted from its coordinate value to obtain the corresponding pixel coordinate in the motion-compensated prediction frame. The value of that pixel from the previous frame is then filled into the corresponding coordinate position in the motion-compensated prediction frame. Unfilled coordinates are filled using nearest-neighbor interpolation, resulting in a complete motion-compensated prediction frame. The differential frame and the motion-compensated prediction frame are subtracted pixel-by-pixel, i.e., the pixel value of the differential frame is subtracted from the pixel value of the motion-compensated prediction frame at the same spatial position, yielding the motion-compensated residual.

[0023] During adaptive quantization, the wavelet coefficients obtained through two-dimensional discrete wavelet transform are subjected to adaptive quantization. The quantization step size is adjusted based on the quantization error feedback from the previous frame. Wavelet coefficients are grouped by sub-bands, which are divided according to the transform level and direction components. The wavelet coefficients within each sub-band have similar frequency characteristics. For each sub-band, each wavelet coefficient within that sub-band is divided by the quantization step size of that sub-band, and the quotient is rounded to the nearest integer to obtain the quantized coefficient. The initial value of the quantization step size is determined by the average energy of the sub-band. The average energy of the sub-band is the arithmetic mean of the squares of all wavelet coefficients within that sub-band. The initial value of the quantization step size is set to the product of the average energy of the sub-band and a preset scaling factor. The preset scaling factor is selected based on the target compression bitrate and can take values ​​within the range [0.2, 1.0]. After encoding and reconstructing a frame, the mean square error between the reconstructed coefficients of the current frame and the original coefficients is calculated for each sub-band. The reconstructed coefficients are obtained by multiplying the quantized coefficients of the sub-band by the same quantization step size to obtain the reconstructed wavelet coefficients, which are the reconstructed coefficients. The mean square error (MSE) is calculated as the arithmetic mean of the squares of the differences between all reconstructed coefficients and the original wavelet coefficients within the sub-band. The MSE of each sub-band is compared with a pre-set second threshold. The second threshold is set separately for different sub-bands based on their visual perception importance, with higher thresholds for higher-frequency sub-bands. If the MSE of a sub-band is greater than its second threshold, the quantization step size of that sub-band is increased in the next frame; if the MSE is not greater than the second threshold, the quantization step size of that sub-band is decreased in the next frame. The adjustment increment is an integer multiple of the preset step size increment. The preset step size increment is set as a fixed percentage of the initial quantization step size of the sub-band, selected as 8% based on the statistical distribution characteristics of the wavelet coefficient amplitude. The integer multiple used for adjustment is set to 1, ensuring that each adjustment is exactly one step size increment, thus allowing the quantization step size to change smoothly with equal step sizes under frame-by-frame feedback control. The quantization step size is updated according to the following formula:

[0024] in, Indicates the sub-band number. Indicates the timing number of the current frame. Indicates the number is The sub-band in the first Quantization step size of a frame, Indicates the number is The sub-band in the first Quantization step size of a frame, Indicates the number is The preset step size increment of the sub-band, Indicates the number is The preset integer multiple of the sub-band, Indicates the direction coefficient. When the... Frame number is When the mean square error of a sub-band is greater than the second threshold of that sub-band, Values When the mean square error is not greater than the second threshold, The value can be . Set to 1, The value is the number. The initial quantization step size of the sub-band is multiplied by 8%. Before quantizing each frame, the quantization step size of each sub-band in the current frame is calculated using a formula, thereby dynamically adjusting the quantization precision of the current frame based on the reconstruction quality of the previous frame.

[0025] Example 2: In specific implementation, please refer to Figure 3 The process of performing run-length encoding and Huffman encoding on the quantized coefficients is as follows: The quantized coefficients within each sub-band obtained through adaptive quantization are arranged in scanning order, proceeding from the low-frequency sub-band to the high-frequency sub-band, scanning diagonally within each sub-band, so that low-frequency coefficients are arranged at the beginning of the sequence and high-frequency coefficients at the end. After arrangement, run-length encoding is performed on the quantized coefficients in the sequence, recording the number of consecutive zero coefficients. For non-zero coefficients, the value of the non-zero coefficient and a sign bit are recorded, with the sign bit indicating the positive or negative sign of the non-zero coefficient, forming an intermediate symbol sequence composed of zero run length, non-zero coefficient values, and a sign bit. The number of consecutive zero coefficients is recorded as an integer, and the value of non-zero coefficients is recorded as a signed integer. Each element in the intermediate symbol sequence can be represented as a binary or triplet, where one component of the binary represents the zero run length, and the other component represents the non-zero coefficient value and its sign. If only consecutive zero coefficients remain at the end of the sequence, an end block symbol is used.

[0026] After run-length encoding, Huffman encoding is performed on the intermediate symbol sequence. The frequency of each symbol in the intermediate symbol sequence is counted, and each symbol represents a set of symbols consisting of zero run-length, the absolute value of non-zero coefficients, and the symbol bits. A Huffman code table is dynamically generated based on the symbol frequency. When constructing the Huffman tree, the two symbols with the lowest frequencies are merged into a new node. The frequency of the new node is the sum of the frequencies of the two symbols before the merge. This merging process is repeated until all symbols are merged to the root node. Codewords are assigned downwards from the root node: bits "0" are assigned to the left branch, and bits "1" are assigned to the right branch, resulting in variable-length codewords for each symbol. The Huffman code table is transmitted along with the compressed bitstream. When generating the compressed bitstream, each symbol in the intermediate symbol sequence is replaced with the corresponding variable-length codeword from the Huffman code table, and these codewords are concatenated sequentially to obtain the compressed bitstream.

[0027] When uploading compressed bitstreams via UDP by fragmenting them according to frame type and importance, different fragmentation strategies are employed for different frame types. At the acquisition end, the type of the frame corresponding to the current compressed bitstream is determined. If the corresponding frame is a keyframe, the compressed bitstream of the keyframe is divided into a first data fragment of fixed length. The fixed length of the first data fragment is set to 256 bytes, based on the fact that the typical maximum transmission unit (MTU) of a network transmission path is 1500 bytes. Setting the length of the first data fragment to be much smaller than the fixed size of the MTU ensures that keyframe data packets only lose a small amount of data in the event of network packet loss, reducing the amount of retransmission data. If the corresponding frame is a differential frame, the compressed bitstream of the differential frame is divided into a second data fragment of variable length. The length of the second data fragment is dynamically determined based on the total length of the differential frame's compressed bitstream. The length of each second data fragment does not exceed a preset maximum length, which is set to 1024 bytes, and the length of the second data fragment is greater than the length of the first data fragment. When fragmenting data, for the last data fragment, if the number of remaining bytes is less than the preset maximum length, the remaining bytes are treated as a single fragment without padding, making the length of the second data fragment variable.

[0028] A timestamp and frame sequence number are written to the header of each data segment. The timestamp identifies the acquisition time of the frame to which the data segment belongs, and the frame sequence number identifies the position of the data segment within its frame and across frames. Simultaneously, a checksum field is written to the header of each data segment. The checksum field is calculated using a 16-bit cyclic redundancy check, and the calculation covers all fields in the data segment header except for the checksum field itself, as well as the data segment payload. The encapsulated data segment is then sent to the receiving end via a UDP socket.

[0029] The transmission rate is dynamically adjusted based on the sender's transmit buffer occupancy. In practice, the sender maintains a transmit buffer to record encapsulated data segments ready for transmission. The transmit buffer occupancy is periodically checked, with a check period of 200 milliseconds. If the transmit buffer occupancy is higher than a first occupancy threshold (80%), the current transmission rate is multiplied by an attenuation factor of 0.9 to obtain a reduced transmission rate. If the transmit buffer occupancy is lower than a second occupancy threshold (20%), the current transmission rate is multiplied by a growth factor of 1.1 to obtain an increased transmission rate. If the transmit buffer occupancy is between the first and second occupancy thresholds, the current transmission rate remains unchanged. The initial transmission rate is set to an estimate of the available bandwidth of the transmission path. After the transmission rate is adjusted, rate control is achieved by adjusting the interval between adjacent data segment transmissions; the interval is equal to the total data segment length divided by the transmission rate.

[0030] Example 3: In practice, the receiving end buffers and sorts received data packets according to their frame sequence numbers as follows: The receiving end maintains a receiving window, the size of which is set to thirty data packets. The receiving window is a logical container, internally organizing the received data packets using an ordered linked list. The ordered linked list is sorted according to the frame sequence number carried in the packet header; data packets with smaller frame sequence numbers are placed at the beginning of the list, and data packets with larger frame sequence numbers are placed at the end. Whenever a data packet is received from the UDP socket, the receiving end parses the frame sequence number in the packet header, searches for the insertion position in the ordered linked list according to the frame sequence number, and inserts the node corresponding to the data packet into the ordered linked list, ensuring that the ordered linked list is always arranged in ascending order of frame sequence number.

[0031] Holes are detected in the ordered linked list. A hole refers to the position of a data packet corresponding to a missing frame number in a sequence of continuously increasing frame numbers. The receiving end traverses the ordered linked list, comparing the frame numbers of adjacent nodes. If the difference between the frame numbers of two adjacent nodes is greater than 1, then all frame numbers between the two adjacent node numbers are identified as the frame numbers corresponding to the hole. For each frame number corresponding to a hole, a retransmission request is generated, containing the frame number of the missing data packet. The retransmission request is sent back to the sending end via UDP, using the same UDP socket as the data packet transmission.

[0032] Simultaneously with sending the retransmission request, a wait timer is started. The timeout duration of the wait timer is set to twice the estimated round-trip time (RTT), which is obtained by periodically measuring the UDP loopback time from the sender to the receiver. If, before the wait timer expires, the receiver receives a retransmission packet with a sequence number matching the frame number in the retransmission request, the retransmission packet is inserted into the ordered linked list at the position corresponding to the frame number, replacing the original gap, thus filling the gap, and the corresponding wait timer is canceled.

[0033] If a matching retransmission data packet is not received after the waiting timer expires, the receiving end extracts the corresponding pixel block from the adjacent frames that have been fully received, and performs bilinear interpolation to fill the missing pixel block. The adjacent frames that have been fully received refer to the frame immediately preceding and following the missing frame in the time domain that is closest to it and has been fully reconstructed. When extracting the corresponding pixel block from the adjacent frames, the spatial location of the pixel block is determined based on the frame type and motion information of the missing frame. If the missing frame is a keyframe, the corresponding pixel block is the pixel block at the same spatial coordinates in the adjacent frames; if the missing frame is a difference frame, the motion compensation position of the corresponding pixel block in the adjacent frames is used as the extraction position, utilizing the motion vector on which the missing frame depends. For the pixel blocks of the two extracted adjacent frames, bilinear interpolation is performed to generate the reconstructed value of the missing pixel block. The bilinear interpolation formula is:

[0034] in, Represents the relative coordinates in the missing pixel block to be filled. Pixel reconstruction values ​​at that location, The range of values ​​is , which represents the normalized spatial location within the missing pixel block. This represents the top-left reference pixel value extracted from the adjacent frame pixel block. This indicates the reference pixel value in the upper right corner. This indicates the reference pixel value at the bottom left corner. This represents the reference pixel value at the bottom right corner. The four reference pixel values ​​are the four integer coordinate pixel values ​​of the target position in the corresponding pixel block of the adjacent frame. These are the horizontal interpolation coefficients, and their values ​​are the normalized offsets of the target position from the top-left reference pixel in the horizontal direction. The vertical interpolation coefficient is the normalized offset of the target position from the top-left reference pixel in the vertical direction. By traversing the coordinates of all pixels within the missing pixel block, the filling value of each pixel is calculated using the bilinear interpolation formula described above, resulting in a complete filled pixel block. This filled pixel block replaces the missing pixel block, ensuring the frame data remains intact.

[0035] The recovered bitstream is subjected to Huffman decoding and inverse run-length encoding, followed by inverse quantization and then inverse two-dimensional discrete wavelet transform to obtain the reconstructed frame. Huffman decoding uses the same Huffman code table as the encoding end, extracted from the header information of the compressed bitstream. During decoding, the bit sequence of the compressed bitstream is read bit by bit, and based on the codeword-to-symbol mapping relationship of the Huffman code table, each variable-length codeword is identified and converted into its corresponding symbol, forming an intermediate symbol sequence consisting of the number of consecutive zero coefficients, the values ​​of non-zero coefficients, and the sign bit.

[0036] The reconstructed wavelet coefficients are obtained by multiplying each quantization coefficient in the quantization coefficient matrix by the quantization step size of the corresponding sub-band. The quantization step size of the corresponding sub-band is the same as the quantization step size used during quantization of that sub-band at the encoder, and the quantization step size value is extracted from the received compressed bitstream parameter set. An inverse two-dimensional discrete wavelet transform is performed on the reconstructed wavelet coefficients of each sub-band. The inverse two-dimensional discrete wavelet transform is composed of a concatenation of a one-dimensional inverse discrete wavelet transform in the row direction and a one-dimensional inverse discrete wavelet transform in the column direction. Using the wavelet basis functions and decomposition levels corresponding to the forward transform at the encoder, the temporal pixel matrix is ​​obtained.

[0037] The temporal pixel matrix is ​​processed according to the frame type. The frame type is obtained from the data packet header or bitstream type identifier. If the frame type is a keyframe, the temporal pixel matrix is ​​directly used as the reconstructed frame. If the frame type is a differential frame, motion vectors are extracted from the received differential frame header information. Motion-compensated prediction frames are generated using these motion vectors and the reconstructed previous frame. The generation method is as follows: subtract the corresponding component of the motion vector from each pixel coordinate of the reconstructed previous frame to obtain the corresponding coordinate in the motion-compensated prediction frame. The pixel values ​​of the previous frame are mapped to this coordinate, and then the gaps are filled using nearest neighbor interpolation to obtain the motion-compensated prediction frame. The temporal pixel matrix and the motion-compensated prediction frame are added pixel by pixel to obtain the reconstructed frame.

[0038] Example 4: In practice, the process of performing deblocking filtering and color correction on the reconstructed frame is as follows: Deblocking filtering is used to reduce block artifacts that may exist in the reconstructed frame. Boundary detection is performed on the boundaries of each macroblock in the reconstructed frame. A macroblock is a fixed-size pixel block into which the reconstructed frame is divided, and the macroblock size is determined according to the block division method during encoding. For the left and top boundaries of each macroblock, the pixel gradients on both sides of the boundary are calculated. Taking the vertical boundary as an example, one pixel column is taken on each side of the boundary, and the absolute difference is calculated for each pair of adjacent left and right pixels along the boundary line. The absolute differences in all rows are summed to obtain the pixel gradient in the vertical direction. The pixel gradient of the horizontal boundary is calculated in the same way by summing the absolute differences of the pixels on the top and bottom sides of the boundary.

[0039] The calculated pixel gradient is compared with a pre-set third threshold. The third threshold is determined based on the human eye's sensitivity to blockiness; in applications with large quantization parameters, it is set to 6% of the maximum pixel value range. If the pixel gradient is greater than the third threshold, it indicates a significant blockiness at the boundary location, and a three-tap smoothing filter is applied to the boundary pixel. The filter coefficients of the three-tap smoothing filter use an empirical combination of coefficients: the left boundary pixel weight is 0.25, the boundary center pixel weight is 0.5, and the right boundary pixel weight is 0.25. During filtering, the filter coefficients are weighted and summed with one pixel on each side of the boundary and the boundary pixel itself, and the sum replaces the original boundary pixel value. If the pixel gradient is not greater than the third threshold, the original pixel value is retained without modification.

[0040] After deblocking filtering, the reconstructed frame is converted from the YUV color space to the RGB color space. The color space conversion uses a standard conversion matrix, where the YUV components represent the luminance signal, blue chromaticity shift, and red chromaticity shift, respectively. During conversion, each pixel value in the RGB channels is calculated through a linear combination of the three YUV component values, with the coefficients conforming to the values ​​defined in the ITU-R BT.601 standard. During the conversion process, the YUV component values ​​are restricted to their valid representation range: the Y component ranges from 16 to 235, and the U and V components range from 16 to 240. After conversion, the RGB component values ​​are constrained to integers from 0 to 255.

[0041] Within the RGB color space, white balance adjustment is performed on each of the three RGB channels to ensure that the average pixel value of each channel is equal. White balance adjustment calculates the total average of the three channels by taking the average pixel value of each RGB color channel in the reconstructed frame after statistical filtering. Then, a gain coefficient is calculated for each channel, which is the total average divided by the corresponding channel's average. Multiplying all pixel values ​​of each channel by the corresponding channel's gain coefficient yields the white balance-adjusted pixel values, ensuring that the average pixel values ​​of the three RGB channels are equal. If the average value of a channel is zero, the gain coefficient is set to 1 to avoid division by zero. The RGB frame after white balance adjustment constitutes the video frame portion of the final audio and video data.

[0042] In some embodiments, the acquisition end performs pre-emphasis and framing on the audio signal in the raw audio / video data. Pre-emphasis is achieved through a first-order high-pass filter, with the zero position of the filter transfer function selected so that the cutoff frequency is approximately 3.5kHz, to enhance the high-frequency components of the audio signal. The pre-emphasized audio signal is framed with a frame length of 20 milliseconds and a frame shift of 10 milliseconds. Hamming windowing is used during framing to reduce spectral leakage. A short-time Fourier transform is performed on each frame of audio signal, with the number of transform points set to 512, to obtain the spectrum. The spectrum is then quantized and encoded using a psychoacoustic model. The psychoacoustic model calculates the masking threshold curve for each frame of audio signal. The spectral coefficients of frequency components below the masking threshold are set to zero; the spectral coefficients above the masking threshold are scalar quantized according to the quantization accuracy requirements. The quantization step size is dynamically calculated based on the ratio of the target coding bit rate to the masking threshold curve. The quantized and encoded spectral coefficients and auxiliary information are packaged to generate an audio bitstream. The auxiliary information includes the quantization step size, frame length, and coding mode.

[0043] After aligning the audio and compressed bitstreams by timestamp, they are multiplexed into the same UDP data stream for transmission. Timestamp alignment uses the acquisition clock as a unified reference. In the multiplexer, audio and compressed bitstream packets with the same timestamp or timestamp differences within a preset tolerance range are grouped into a single multiplexed data unit. A synchronization identifier and unit length are written to the header of each multiplexed data unit, which is then encapsulated into a UDP datagram and sent through the same UDP socket.

[0044] At the receiving end, the audio bitstream is demultiplexed from the UDP data stream. The receiving end parses the synchronization flag and unit length in the header of the multiplexed data unit, extracts the audio bitstream data and video compressed bitstream data according to the unit length, and sends the audio bitstream to the audio decoding module. During inverse quantization, the quantization step size is extracted from the audio bitstream, and the quantized values ​​of each spectral coefficient are multiplied by the quantization step size to obtain the reconstructed spectral coefficients. An inverse short-time Fourier transform is performed on the reconstructed spectral coefficients. The inverse short-time Fourier transform uses the same number of transform points and window function as at the encoding end, and the time-domain signal is synthesized using the overlap-add method to obtain the reconstructed audio signal. The reconstructed audio signal is synchronized with the video frames in the final audio and video data based on timestamps. The reconstructed audio signal is aligned with the reconstructed video frames by the audio and video synchronization output module before being sent to the playback device.

[0045] See Figure 4 In the graph, the horizontal axis represents the frame number, ranging from 0 to 500, and the vertical axis represents the pixel mean, ranging from approximately 80 to 190. The graph uses red, green, and blue dashed lines to represent the pixel mean trends of the R, G, and B color channels in the original video frame, respectively. It can be seen that the original R channel pixel mean exhibits periodic fluctuations, with a peak value close to 185 and a trough value of approximately 115. The original G channel pixel mean is relatively stable, fluctuating between 120 and 160. The original B channel pixel mean fluctuates significantly, with a minimum value dropping to approximately 85 and a maximum value approaching 170, and its waveform shows a phase difference compared to the R channel.

[0046] The solid black line represents the unified mean curve after color correction processing as described in Example 4. This curve is relatively smooth across the entire frame number range, with the mean value maintained between approximately 140 and 170, significantly reducing the mean value differences between the original color channels. This demonstrates that through color space conversion and white balance adjustment, the mean values ​​of the three RGB channels were successfully corrected, making them more consistent and reducing color deviation.

[0047] The curve changes in the figure demonstrate the effect of color correction: between frame numbers 0 and 100, the original R channel mean value is significantly higher than the G and B channels, and the corrected mean value curve falls in the middle of the three; between frames 100 and 300, the B channel mean value shows a significant trough, and the corrected mean value curve rises significantly, filling the low value gap of the blue channel; between frames 300 and 500, the differences in the mean values ​​of each channel still exist, but the amplitude decreases, and the corrected curve is smooth and stable. This trend fully verifies the technical effect of adjusting the white balance for different channels using the color correction module in Example 4.

[0048] Example 5: In a specific implementation, a real-time audio and video acquisition, uploading, and data processing system includes a memory, a processor, and a computer program stored in the memory and running on the processor. The memory stores the computer program, as well as intermediate and result data generated during its operation. The processor reads computer program instructions from the memory and executes the computer program instructions to implement the steps of the real-time audio and video acquisition, uploading, and data processing method.

[0049] When the processor executes the computer program, it controls the acquisition device to obtain raw audio and video data. After the raw audio and video data enters the encoding module, the processor performs inter-frame differential analysis on the raw audio and video data, dividing the frames into keyframes and difference frames based on the differential results. A two-dimensional discrete wavelet transform is performed on the keyframes, and a two-dimensional discrete wavelet transform is performed on the difference frames after motion compensation residual calculation to obtain wavelet coefficients. Adaptive quantization is performed on the wavelet coefficients, with the quantization step size adjusted based on the quantization error feedback from the previous frame. The quantized coefficients are then processed by run-length encoding and Huffman coding to generate a compressed bitstream.

[0050] In practice, the processor controls the transmission module to fragment the compressed bitstream according to frame type and importance, inserting a timestamp and frame sequence number into each data fragment, and uploading it to the receiving end in real time via the UDP protocol. During fragmentation, the compressed bitstream of key frames is divided into a fixed-length first data fragment, while the compressed bitstream of differential frames is divided into a variable-length second data fragment, with the length of the first data fragment being shorter than the length of the second data fragment. The transmission rate is dynamically adjusted based on the occupancy rate of the sending buffer at the sending end.

[0051] On the receiving end, the system also includes a processor and memory. The processor executes a computer program to perform the receiving process. The processor buffers and sorts the received data packets according to their frame sequence numbers, maintaining a receiving window in memory with a size of thirty data packets. Received data packets are inserted into an ordered linked list within the window according to their frame sequence numbers. A retransmission request is generated for the frame sequence number corresponding to a hole in the ordered linked list and sent back to the sending end via UDP. Simultaneously, a wait timer is started. If a retransmitted data packet is received before the wait timer expires, the retransmitted data packet replaces the hole position; if the wait timer expires, the corresponding pixel block is extracted from a completely received adjacent frame, and bilinear interpolation is performed to fill the missing pixel block.

[0052] The processor performs Huffman decoding and inverse run-length encoding on the recovered bitstream obtained from the receiver, then performs inverse quantization, followed by inverse two-dimensional discrete wavelet transform to obtain the reconstructed frame. Deblocking filtering and color correction are then applied to the reconstructed frame to output the final audio and video data.

[0053] Optionally, the processor also performs pre-emphasis and framing on the audio signal in the raw audio and video data at the acquisition end, performs short-time Fourier transform on each frame of audio signal to obtain the spectrum, performs quantization encoding controlled by a psychoacoustic model on the spectrum to generate an audio bitstream, aligns the audio bitstream and compressed bitstream by timestamp, and then multiplexes them into the same UDP data stream for transmission. At the receiving end, the processor demultiplexes the audio bitstream from the UDP data stream, performs inverse quantization and inverse short-time Fourier transform to obtain the reconstructed audio signal, and outputs it synchronously with the final audio and video data.

[0054] See Figure 5 In the graph, the horizontal axis represents time in seconds, and the vertical axis represents processor utilization in percentage. The graph includes two curves: the transmitter's processor utilization and the receiver's processor utilization, represented by solid and dashed lines respectively. The legend clearly indicates the meaning of each curve.

[0055] As shown in the graph, the processor utilization rate at the transmitting end fluctuates between 12% and 85%, exhibiting a clear periodic trend. The period is approximately 60 seconds. During this fluctuation, the utilization rate gradually rises from a trough to a peak and then falls back down. The trough range is approximately 15% to 25%, and the peak range reaches over 80%. The trend of processor utilization rate at the receiving end is largely similar to that at the transmitting end, but the overall level is slightly lower, ranging from approximately 12% to 72%. The processor utilization rate curves at both ends are well synchronized, with the peak and trough values ​​occurring at roughly the same times.

[0056] The phased analysis shows that within the first 0 to 50 seconds, the processor utilization at the transmitting end initially rises rapidly to approximately 80%, then gradually decreases to around 20%; the processor utilization at the receiving end rises from around 40% to 70% before similarly falling back to around 15%. Between 50 and 100 seconds, both utilization rates rise again to relatively high peaks, with the transmitting end at approximately 85% and the receiving end at approximately 72%. Similar cyclical changes repeat between 100 and 150 seconds and between 150 and 200 seconds, respectively.

[0057] The periodic changes in processor utilization reflect the dynamic changes in the processing load of the transmitting and receiving ends in the real-time audio and video acquisition, uploading, and data processing system of Example 5 over time. The transmitting end, due to performing multiple computationally intensive tasks such as audio and video acquisition, inter-frame differential analysis, two-dimensional discrete wavelet transform, adaptive quantization and run-length encoding, Huffman coding, and fragmented uploading, experiences a high and fluctuating processor load. The receiving end processor, on the other hand, needs to perform tasks such as data packet reception, buffer sorting, retransmission control, bitstream decoding, inverse quantization, inverse wavelet transform, deblocking filtering, and color correction. Its load level is slightly lower than that of the transmitting end, but it also exhibits periodic fluctuations depending on the stage of data processing.

[0058] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for real-time audio and video acquisition, uploading, and data processing, characterized in that, include: Collect raw audio and video data, perform inter-frame difference analysis on the raw audio and video data, and divide the frames into key frames and difference frames according to the difference results; Two-dimensional discrete wavelet transform is performed on the keyframes, and two-dimensional discrete wavelet transform is performed on the differential frames after motion compensation residual calculation to obtain wavelet coefficients. Adaptive quantization is performed on the wavelet coefficients, and the quantization step size is adjusted according to the quantization error feedback of the previous frame; Run-length encoding and Huffman encoding are performed on the quantized coefficients to generate a compressed bitstream; The compressed bitstream is segmented according to frame type and importance, and a timestamp and frame sequence number are inserted into each data segment. The data is then uploaded in real time via the UDP protocol. The receiving end buffers and sorts the received data packets according to the frame sequence number, initiates a retransmission request for the missing data packets, and performs linear interpolation recovery on the missing region based on adjacent frames during the retransmission waiting period. The recovered bitstream is subjected to Huffman decoding and inverse run-length encoding, then inverse quantization, followed by inverse two-dimensional discrete wavelet transform to obtain the reconstructed frame. Deblocking filtering and color correction are performed on the reconstructed frames to output the final audio and video data.

2. The method for real-time audio and video acquisition, uploading, and data processing according to claim 1, characterized in that, The step of performing inter-frame differential analysis on the raw audio and video data, and dividing the frames into keyframes and differential frames based on the differential results, specifically includes: calculating the sum of absolute pixel differences between the current frame and the previous frame, comparing the sum of absolute pixel differences with a first threshold, and marking it as a keyframe if it is greater than the first threshold, otherwise marking it as a differential frame; for the differential frame, calculating motion vectors using translational motion estimation based on optical flow field, generating motion compensation prediction frames using the motion vectors, and subtracting the differential frame from the motion compensation prediction frame pixel by pixel to obtain the motion compensation residual.

3. The method for real-time audio and video acquisition, uploading, and data processing according to claim 1, characterized in that, The adaptive quantization of the wavelet coefficients, wherein the quantization step size is adjusted based on the quantization error feedback of the previous frame, specifically includes: The wavelet coefficients are grouped by sub-bands, and the coefficients in each sub-band are divided by the quantization step size of that sub-band and rounded down. The initial value of the quantization step size is determined by the average energy of the sub-band. For each sub-band, the mean square error between the reconstructed coefficients of the current frame and the original coefficients is calculated. If the mean square error is greater than the second threshold, the quantization step size of the sub-band is increased in the next frame; otherwise, the quantization step size is decreased. The adjustment range is an integer multiple of the preset step size increment.

4. The method for real-time audio and video acquisition, uploading, and data processing according to claim 1, characterized in that, The step of performing run-length encoding and Huffman encoding on the quantized coefficients to generate a compressed bitstream specifically includes: The quantization coefficients within each subband are arranged in scanning order. For consecutively occurring zero coefficients, the number of consecutively occurring zero coefficients is recorded. For non-zero coefficients, their values ​​and a sign bit are recorded to form an intermediate sign sequence. The frequency of occurrence of each symbol in the intermediate symbol sequence is statistically analyzed, a Huffman code table is dynamically generated, each symbol is replaced with the corresponding variable-length codeword, and the compressed code stream is output.

5. The method for real-time audio and video acquisition, uploading, and data processing according to claim 1, characterized in that, The step of segmenting the compressed bitstream according to frame type and importance, inserting a timestamp and frame sequence number into each data segment, and uploading it in real time via UDP protocol specifically includes: The compressed bitstream of the keyframe is divided into a first data slice of fixed length, and the compressed bitstream of the differential frame is divided into a second data slice of variable length, wherein the length of the first data slice is less than the length of the second data slice. Write the timestamp and frame number to the header of each data slice, and write the checksum field; The encapsulated data fragments are sent to the receiving end via UDP sockets, and the sending rate is dynamically adjusted according to the sending buffer occupancy rate of the sending end.

6. The method for real-time audio and video acquisition, uploading, and data processing according to claim 1, characterized in that, The receiving end buffers and sorts the received data packets according to the frame sequence number, initiates a retransmission request for missing data packets, and performs linear interpolation recovery on the missing region based on adjacent frames during the retransmission waiting period. Specifically, this includes: maintaining a receiving window with a size of thirty data packets, inserting the received data packets into an ordered linked list within the window according to the frame sequence number, generating a retransmission request for the frame sequence number corresponding to the hole in the linked list, and sending it back to the sending end via UDP; starting a waiting timer, and if a retransmitted data packet is received before the waiting timer expires, replacing the hole position; otherwise, extracting the corresponding pixel block from the adjacent frames that have been fully received, and performing bilinear interpolation to fill the missing pixel block.

7. The method for real-time audio and video acquisition, uploading, and data processing according to claim 1, characterized in that, The process of performing Huffman decoding and inverse run-length encoding on the recovered bitstream, followed by inverse quantization and then inverse two-dimensional discrete wavelet transform to obtain the reconstructed frame specifically includes: The compressed bitstream is decoded using the same Huffman code table as the encoder to obtain an intermediate symbol sequence consisting of the number of consecutive zero coefficients, the value of non-zero coefficients, and the sign bit. The intermediate symbol sequence is then expanded into a quantization coefficient matrix according to the run-length rule. Multiply the quantization coefficient matrix by the quantization step size of the corresponding sub-band to obtain the reconstructed wavelet coefficients; The reconstructed wavelet coefficients are subjected to an inverse two-dimensional discrete wavelet transform to obtain a temporal pixel matrix. Then, the matrix is ​​processed according to the frame type: if it is a keyframe, it is directly used as the reconstructed frame; if it is a differential frame, the temporal pixel matrix is ​​added to the motion compensation prediction frame to obtain the reconstructed frame.

8. The method for real-time audio and video acquisition, uploading, and data processing according to claim 1, characterized in that, The process of performing deblocking filtering and color correction on the reconstructed frame to output the final audio and video data specifically includes: For each macroblock boundary of the reconstructed frame, detecting the pixel gradient on both sides of the boundary; if the gradient is greater than a third threshold, applying a three-tap smoothing filter to the boundary pixel to reduce blockiness; converting the filtered reconstructed frame from the YUV color space to the RGB color space, and performing white balance adjustment on the three RGB channels respectively to make the pixel mean of each channel equal, thus obtaining the final audio and video data.

9. The method for real-time audio and video acquisition, uploading, and data processing according to claim 1, characterized in that, The method further includes: at the acquisition end, performing pre-emphasis and framing on the audio signal in the original audio and video data, performing short-time Fourier transform on each frame of audio signal to obtain the spectrum, performing psychoacoustic model-controlled quantization encoding on the spectrum to generate an audio bitstream; aligning the audio bitstream and the compressed bitstream by timestamp, and then multiplexing them into the same UDP data stream for transmission; at the receiving end, demultiplexing the audio bitstream from the UDP data stream, performing inverse quantization and inverse short-time Fourier transform to obtain the reconstructed audio signal, and outputting it synchronously with the final audio and video data.

10. A real-time audio and video acquisition, uploading, and data processing system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the real-time audio and video acquisition, uploading, and data processing method as described in any one of claims 1 to 9.

Images