Lightweight audio-video synchronization transmission method based on HTTP multipart response

Abstract

The application discloses a lightweight audio-video synchronization transmission method based on HTTP Multipart response. A server cuts PCM audio stream into continuous audio fragments according to a fixed sampling point number, and generates corresponding video frames; a single HTTP long connection returns a Multipart response, and corresponding video fragments and audio fragments are written in adjacent interlaced mode. A client streamingly analyzes the response, obtains the video fragments and the audio fragments according to a type field and a content length in a fragment header, arranges the audio fragments into a playing queue, takes a predetermined or actual starting playing time of the first audio buffer as an audio anchor point, determines a frame interval according to the sampling point number and a sampling rate, and determines a video frame to be rendered by an audio playing clock. The application can reduce protocol complexity and maintain audio-video synchronization.

Description

Technical Field

[0001] This invention relates to the fields of digital humans, virtual anchors, real-time interactive displays, and browser-based audio and video playback technology. Specifically, it relates to a lightweight audio-visual synchronization transmission method based on HTTP Multipart responses. This method interleaves video and audio segments through HTTP Multipart responses in a single HTTP long connection and achieves lightweight audio-visual synchronization on the client side using an audio playback clock. Background Technology

[0002] Digital humans, virtual anchors, and real-time avatars driven by local AI require synchronized presentation of audio-driven video frames and their corresponding audio on the client side during audio-visual generation and streaming. Existing real-time audio and video streaming and synchronization solutions mainly fall into the following categories, all of which are insufficient for lightweight, low-latency display scenarios.

[0003] The first category is real-time communication solutions based on WebRTC. These solutions are designed for strong real-time bidirectional audio and video communication. They typically require the deployment of STUN / TURN servers and involve SDP negotiation, ICE candidate switching, media track management, and network address translation (NAT) traversal during the session establishment phase. Their protocol stack is large, and deployment and debugging are complex. For unidirectional digital human streaming scenarios such as LAN deployment, local deployment, embedded WebView display, or local inference-driven approaches, the signaling negotiation and NAT traversal mechanisms are redundant overhead, increasing implementation and maintenance complexity.

[0004] The second category is solutions based on WebSocket and overlaid with custom binary protocols. These solutions transmit audio and video packets via WebSocket, and define frame formats, sequence numbers, timestamps, audio / video type fields, and corresponding decoding logic at the application layer. Both the front-end and back-end need to maintain dedicated codecs and protocol parsing logic. Furthermore, in some enterprise proxy, firewall, or embedded WebView environments, additional handling of connection upgrades and proxy compatibility is required, resulting in higher engineering implementation and maintenance costs.

[0005] The third category includes segmented streaming media solutions such as HLS and DASH. These solutions are suitable for large-scale on-demand or live distribution over the Internet, but due to the segment length and playback buffering mechanism, their end-to-end latency is usually in the second range, such as 2 to 10 seconds, which is difficult to meet the needs of interactive scenarios that are sensitive to end-to-end latency, such as digital human Q&A, stand-alone exhibition halls, and LAN interactive demonstrations.

[0006] The fourth category is the traditional audio-video synchronization scheme. This type of scheme typically explicitly carries frame sequence numbers, display timestamps (PTS), decoding timestamps (DTS), playback timestamps, or system clock information in the audio and video streams. The client controls audio-video synchronization based on these timestamps and buffer status. This type of scheme is common in standard media containers and complex players, but if applied to lightweight HTTP fragmented transmission, it requires appending the aforementioned time information to each fragment header and maintaining the corresponding parsing logic and status on the client side. This increases protocol header overhead and implementation complexity, hindering lightweight implementation.

[0007] The fifth category is chunked transmission technology based on HTTP push or multipart / x-mixed-replace. This type of technology can continuously push multiple content chunks through a single HTTP response, and is often used for image streams, incremental page updates, or server pushes. However, existing HTTP Multipart push-based solutions are mostly used for transmitting images or incremental content, and do not disclose how to interleave video chunks and audio chunks in the same HTTP Multipart response for audio-driven real-time audio-visual generation scenarios. Nor do they disclose how to deduce the video rendering timing using the audio playback clock without relying on frame sequence numbers, PTS, DTS, or playback timestamps in the chunk header on the client side. Therefore, they cannot be directly used to maintain synchronization between video rendering and audio playback.

[0008] In summary, existing technologies, when applied to real-time audio-visual display scenarios driven by digital humans, virtual anchors, or local artificial intelligence, suffer from drawbacks such as heavy protocol stacks, complex deployment, the need for custom protocols and codecs, high end-to-end latency, and increased protocol header overhead and parsing status due to reliance on explicit timestamp fields. There is a lack of a complete solution that can achieve lightweight transmission of audio and video segments over ordinary HTTP long connections while maintaining stable audio-visual synchronization on the client side. Summary of the Invention

[0009] To address the aforementioned shortcomings of existing technologies, the present invention aims to provide a lightweight audio-visual synchronization transmission method based on HTTP Multipart responses. This method achieves lightweight transmission of video and audio segments using ordinary HTTP long connections without introducing WebRTC signaling negotiation, WebSocket custom protocols, HLS / DASH long buffers, or segment frame sequence numbers, PTS, DTS, or playback timestamps, while maintaining synchronization between video rendering and audio playback on the client side.

[0010] To achieve the above objectives, the present invention provides a lightweight audio-visual synchronization transmission method based on HTTP Multipart response, comprising the following steps.

[0011] The server obtains the PCM audio stream used for real-time audio-visual generation, divides it into continuous audio segments according to a fixed number of sampling points, and generates corresponding video frames based on each audio segment.

[0012] The server establishes a single long HTTP connection with the client, returns an HTTP Multipart response to the client, and writes the video and audio segments as adjacent segments interleaved in the HTTP Multipart response according to the order of the audio segments.

[0013] The client reads the HTTP Multipart response in a streaming manner and separates the video and audio segments according to the boundary markers.

[0014] The client queues audio segments into the audio playback queue and uses the scheduled start time or the actual start time of the first audio segment as the audio anchor point.

[0015] The client determines the frame interval based on the number of sampling points and sampling rate of the audio segments. Without needing to read the frame sequence number, PTS / DTS or playback timestamp from the segment header of the HTTPMultipart response, the client determines the video frame to be rendered based on the elapsed time of the audio playback clock relative to the audio anchor point and draws the video frame to be rendered to the image display buffer.

[0016] The core of this invention lies in the fact that the server divides the audio stream according to a fixed number of sampling points and generates corresponding video frames for each segment. This allows the arrival order of video and audio segments in the HTTP Multipart response to naturally carry the implicit time relationship between them, thus eliminating the need to transmit frame sequence numbers, PTS, DTS, or playback timestamps in the segment header. The client uses the playback time corresponding to the first audio segment as the audio anchor point and the audio playback clock as a unified clock source. The frame interval is determined by the ratio of the number of sampling points to the sampling rate of the audio segment. Then, the video frame to be rendered is derived based on the elapsed time of the audio playback clock relative to the audio anchor point. This makes video rendering driven by the audio playback progress rather than by the network arrival time, avoiding audio-visual drift caused by network jitter, parsing delay, or image decoding delay.

[0017] In one optional implementation, the server acquires the PCM audio stream and generates corresponding video frames based on each audio segment, including: the server receives PCM S16LE format audio input for digital human audio-visual generation, forming 50ms audio segments with each audio segment containing 800 sampling points and a sampling rate of 16000Hz; each 50ms audio segment is sent to the digital human inference engine to obtain a BGR image frame corresponding to that audio segment. Thus, the audio segmentation granularity corresponds one-to-one with the video frame generation granularity, providing a basis for the audio segment sequence to carry an implicit timeline.

[0018] In one optional implementation, the server returns an HTTP Multipart response and interleaves the written segments, including: setting the content type to multipart / x-mixed-replace in the HTTP response header and setting boundary identifiers to distinguish adjacent segments; writing a video segment header and video segment body to each video segment, and writing an audio segment header and audio segment body to each audio segment; and outputting video segments generated from the same audio segment adjacent to that audio segment in the HTTP Multipart response. Thus, the client can stably segment adjacent segments based on the boundary identifiers and associate video frames with their corresponding audio based on adjacency.

[0019] In one optional implementation, generating corresponding video frames based on each audio segment further includes: encoding the BGR image frames output by the digital human inference engine into video segments in PNG, WebP, or Raw BGRA format; when the video segment uses PNG or WebP format, writing the content type corresponding to that format into the video segment header; when the video segment uses Raw BGRA format, writing a binary content type identifier into the video segment header. Thus, the video segment format can be flexibly selected based on bandwidth, decoding capability, and rendering path, and the client can select the corresponding decoding or conversion path based on the content type in the video segment header.

[0020] In one optional implementation, writing an audio segment header and audio segment body into each audio segment includes: writing a binary content type identifier into the audio segment header, the binary content type identifier indicating that the audio segment body is an unencapsulated audio byte stream, and writing an audio segment identifier field indicating that the segment is an audio segment; writing the corresponding PCM audio data into the audio segment body, so that the client sends the segment into the audio processing flow according to the audio segment identifier field. Therefore, even if both video segments and audio segments use binary content type identifiers, the client can still accurately distinguish between the two based on the audio segment identifier field.

[0021] In one optional implementation, the client reads and segments fragments according to boundary markers in a streaming manner, including: the client continuously reads response bytes through an HTTP response reading interface that supports on-the-fly parsing; during the reading process, the client identifies the boundary marker, and when the boundary marker crosses an adjacent read block, retains the last byte of the previous read block whose length is not less than the boundary marker length minus one, and merges the last byte of the previous read block with the starting byte of the next read block to determine the starting boundary of the adjacent fragment; after identifying the fragment starting boundary, the client parses the fragment header and reads the fragment body of the corresponding length according to the content length field in the fragment header. Therefore, even if the HTTP response bytes are transmitted in chunks by the network or a proxy, and the boundary marker is truncated at the intersection of two read blocks, the client can still correctly identify the complete boundary marker and stably segment the fragments.

[0022] In one optional implementation, the client queues audio segments and determines audio anchor points, including: decoding arriving audio segments into audio buffers; submitting the audio buffers to the audio output queue according to their arrival order in the HTTP Multipart response; and determining the predetermined start time or actual start time of the first audio buffer when it is submitted to the audio output queue, and using the predetermined start time or actual start time as the audio anchor point. Thus, the audio plays continuously and seamlessly in their arrival order, providing a unified time reference for video rendering.

[0023] In one optional implementation, the client determines the video frame to be rendered by: dividing the number of sampling points of the audio segment by the sampling rate to obtain the frame interval; using the difference between the current time of the audio playback clock and the audio anchor point as the audio playback elapsed duration; dividing the audio playback elapsed duration by the frame interval and rounding down to obtain the target frame number; using the first video segment as the starting frame and determining incremental frame numbers for subsequent video segments according to the arrival order of the video segments in the HTTP Multipart response; and selecting the video frame corresponding to the target frame number from the cached video segments as the video frame to be rendered. Thus, the video frame to be rendered is entirely determined by the audio playback progress, without relying on any explicit timestamp field.

[0024] In one optional implementation, the step of selecting a video frame to be rendered from the cached video segments includes: the client establishing a candidate video frame queue according to the arrival order of the video segments in the HTTP Multipart response; when the target frame number is less than the position of the rendered frame in the candidate video frame queue, maintaining the current image display buffer content; when the target frame number is greater than the position of the rendered frame in the candidate video frame queue, skipping the video frames that have not been rendered before the target frame number, and using the video frame corresponding to the target frame number as the video frame to be rendered; when the target frame number is equal to the position of the rendered frame in the candidate video frame queue, or when the video frame corresponding to the target frame number has not yet been cached, maintaining the current image display buffer content.

[0025] In one optional implementation, drawing the video frame to be rendered to the image display buffer includes: the client decoding the video frame to be rendered into a drawable image object, or converting the video frame to be rendered into pixel data according to Raw BGRA format; drawing the drawable image object or pixel data to the image display buffer; wherein the server exposes only one HTTP port, the single HTTP long connection is an HTTP / 1.1 long connection, and the HTTP Multipart response, after being forwarded by an HTTP reverse proxy, does not reassemble the fragment body boundaries and maintains the fragment output order. Therefore, this invention can be implemented in browsers, embedded WebViews, local area networks, local deployments, and deployment environments via HTTP reverse proxies, with low server-side deployment requirements and compatibility with reverse proxies.

[0026] Compared with existing technologies, this invention has the following advantages: First, the protocol stack is lightweight, based solely on ordinary HTTP long connections and HTTP Multipart responses, eliminating the need for WebRTC signaling negotiation, ICE candidate switching, or TURN relay. Second, the engineering complexity is low; the server and client do not need to design complete audio and video container formats, nor do they need to carry frame sequence numbers, PTS, DTS, or playback timestamps in each segment header. Third, the synchronization mechanism is clear; the frame interval is determined by the number of audio segment sampling points and the sampling rate, and video rendering is driven by the audio playback clock, using audio as the synchronization benchmark to avoid audio-visual drift caused by direct rendering based on network arrival time. Fourth, it has good adaptability; the client can use common client capabilities such as HTTP response reading interfaces supporting streaming, audio playback interfaces, image decoding interfaces, and image display buffers, without requiring the installation of a dedicated player, making it suitable for browsers and embedded WebViews. Fifth, deployment requirements are low; the server can expose only one HTTP port, making it suitable for local machines, LANs, exhibition hall terminals, embedded WebViews, or deployment environments after reverse proxying, and reducing end-to-end latency. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A schematic diagram of the overall system architecture of the lightweight audio-visual synchronization transmission method based on HTTP Multipart response provided in an embodiment of the present invention; Figure 2 A schematic diagram of the overall process of a lightweight audio-visual synchronization transmission method based on HTTP Multipart response provided in an embodiment of the present invention; Figure 3 This is a timing diagram illustrating the interleaved arrangement of video and audio segments in an HTTP Multipart response provided in an embodiment of the present invention. Figure 4 This is a schematic diagram illustrating the principle of a client in an embodiment of the present invention that uses the audio playback clock as an anchor point to determine the video frame to be rendered. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Unless otherwise specified, the values ​​of the number of sampling points, sampling rate, frame interval, fragmentation format, and field names are all illustrative examples and do not constitute the sole limitation on the scope of protection.

[0030] This invention is applicable to scenarios where video frames generated by audio drivers and their corresponding audio need to be presented synchronously, such as digital humans, virtual anchors, interactive AI avatars, locally AI-driven explanation terminals, exhibition hall screens, and embedded WebView displays. See also Figure 1 In a typical system architecture, the server side includes an audio segmentation module, a digital human inference engine, a video segmentation generation module, an audio segmentation generation module, and an HTTP Multipart response organization module; the client side includes a response stream reading module, a boundary resolution module, an audio playback queue, a candidate video frame queue, an audio playback clock, a target frame determination module, and an image display buffer. The server and client are connected via a single HTTP persistent connection.

[0031] See Figure 2 The method of the present invention includes steps S1 to S5. Steps S1 to S5 and their optional specific implementations will be described below in sequence.

[0032] Step S1: The server obtains the PCM audio stream used for real-time audio-visual generation, divides it into continuous audio segments according to a fixed number of sampling points, and generates corresponding video frames based on each audio segment.

[0033] This step can be further broken down into operations such as audio acquisition, audio segmentation, video frame generation, and video segmentation encoding, as detailed below.

[0034] First, the server acquires the PCM audio stream used for real-time audio-visual generation. This PCM audio stream can originate from speech synthesized by the text-to-speech module, speech acquired by the real-time speech acquisition module, driving audio provided by the digital human driving module, or speech generated by the local artificial intelligence generation module. The PCM audio stream is a raw audio sampling sequence of pulse code modulation, without being encapsulated into an audio container format.

[0035] Secondly, the server divides the PCM audio stream into continuous audio segments according to a fixed number of sampling points. The fixed number of sampling points can be determined based on the target frame rate, sampling rate, or the processing granularity of the inference engine. As a specific implementation, the server receives PCM S16LE format audio input, i.e., 16-bit little-endian signed integer pulse code modulation, with a sampling rate of 16000Hz. The server accumulates 800 sampling points to form one audio segment. Since the duration of each audio segment is equal to the number of sampling points divided by the sampling rate (800 divided by 16000 equals 0.05 seconds), each audio segment corresponds to 50 milliseconds of audio content, corresponding to a video rhythm of 20 frames per second. It should be noted that the aforementioned 800 sampling points, 16000Hz, and 50 milliseconds are only one example value. In other embodiments, the fixed number of sampling points, sampling rate, and the segment duration determined therefrom can take other values, as long as the server's segmentation granularity corresponds one-to-one with the video frame generation granularity.

[0036] Third, the server sends each audio segment to the digital human inference engine to obtain a corresponding video frame. The digital human inference engine is an audio-driven image generation neural network model that generates corresponding video frames based on input audio. It takes the sampled data of the audio segment (or audio features extracted from the sampled data, such as Mel-spectral features) as input and outputs a digital human video frame corresponding to that audio segment. The video frame reflects the lip movements, facial expressions, or postures matching the audio segment. In one specific implementation, the digital human inference engine outputs a BGR image frame for each 50-millisecond audio segment, i.e., a bitmap image frame arranged in the order of blue, green, and red color channels. Thus, the audio segments and video frames correspond one-to-one in both quantity and order, with the nth audio segment corresponding to the nth video frame.

[0037] Fourth, the server encodes the aforementioned video frames into video segments. As a specific implementation corresponding to video frame encoding, the server can encode the BGR image frames output by the digital human inference engine into video segments in PNG, WebP, or Raw BGRA format, as detailed below.

[0038] When using PNG or WebP format, the server converts and compresses BGR image frames into PNG or WebP image byte streams as video segments. The server then writes the content type corresponding to the format into the video segment header; for example, it writes "image / png" for PNG and "image / webp" for WebP. PNG format is suitable for lossless image transmission and can be directly decoded by the client's image decoding path. WebP format typically has a smaller file size at the same image quality, making it suitable for bandwidth-constrained scenarios.

[0039] When using the Raw BGRA format, the server converts BGR image frames into a raw pixel byte stream arranged pixel-by-pixel according to the four channels of blue, green, red, and transparency. This stream serves as the video segment body. A binary content type identifier, such as application / octet-stream, is written into the video segment header. Simultaneously, a format field indicating that the video segment format is Raw BGRA, as well as the image width and height fields, are also written, allowing the client to reconstruct the pixel arrangement. The Raw BGRA format requires no image encoding or decoding, making it suitable for scenarios where the server and client are on the same local area network or local machine, have sufficient bandwidth, and where reducing image encoding / decoding latency is desired.

[0040] Step S2: The server establishes a single HTTP long connection with the client, returns an HTTP Multipart response to the client, and writes the video and audio segments as adjacent segments interleaved in the HTTP Multipart response according to the order of the audio segments.

[0041] See Figure 3In step S2, the server establishes a single persistent HTTP connection with the client and returns an HTTP Multipart response to the client. As a specific implementation, this single persistent HTTP connection is an HTTP / 1.1 persistent connection. The server sets the content type to multipart / x-mixed-replace in the HTTP response header and sets a boundary identifier to distinguish adjacent fragments. For example, the response header can be set to: content type multipart / x-mixed-replace with a boundary of frame, cache control no-cache, and connection mode keep-alive. Here, "frame" is one possible value for the boundary identifier; in other embodiments, it can be replaced with any boundary string that does not conflict with the fragment body bytes.

[0042] In returning the HTTP Multipart response, the server interleaves video and audio segments as adjacent segments, following the order of the audio segments. Specifically, the server writes a video segment header and body into each video segment, and an audio segment header and body into each audio segment, ensuring that video segments generated from the same audio segment are output adjacent to that audio segment in the HTTP Multipart response. See also Figure 3 The video track and audio track correspond one-to-one in time. The 0th video frame is adjacent to the 0th audio frame, and the 1st video frame is adjacent to the 1st audio frame. The frame interval is... Every 50ms, a pair of adjacent segments arrives. Similarly, every 50ms, a pair of video and audio segments derived from the same audio segment arrives. Therefore, the arrival order of the video and audio segments in the response naturally reflects their temporal correspondence.

[0043] In one specific implementation, when video segments are written in PNG format, the format is as follows: a boundary identifier line "--frame", followed by the video segment header, which includes the content type image / png and a content length field, the value of which is the number of bytes in the video segment body; after the video segment header, a blank line separates the PNG video segment body. Writing video segments in WebP format is similar, except that the content type is image / webp.

[0044] When a video segment is written in Raw BGRA format, it follows the format: a boundary identifier line "--frame", followed by the video segment header. The video segment header includes a content type application / octet-stream indicating that the segment body is a binary byte stream, a segment identifier field indicating that the segment is a video segment (e.g., a segment identifier field with the value "video"), a format field indicating that the video segment format is Raw BGRA (e.g., a format field with the value "raw-bgra"), image width and height fields, and a content length field. After the video segment header, the Raw BGRA video segment body is written, separated by a blank line.

[0045] Corresponding to the writing of audio segment headers and audio segment bodies, the server writes a binary content type identifier in each audio segment header. This binary content type identifier (e.g., content type application / octet-stream) indicates that the audio segment body is an unencapsulated audio byte stream. The server also writes an audio segment identifier field in the audio segment header to indicate that the segment is an audio segment. In one specific implementation, the audio segment header includes: a content type of application / octet-stream, an audio segment identifier field with the value "audio", a field indicating the audio encoding format (e.g., a format field with the value "pcm-s16le"), a field indicating the sampling rate (e.g., a sampling rate field with the value "16000"), a field indicating the number of sample points (e.g., a sample point number field with the value "800"), and a content length field. The server writes the corresponding PCM audio data, i.e., the aforementioned segmented PCM S16LE sample bytes, into the audio segment body. Therefore, the client can send the audio segment into the audio processing flow based on the audio segment identifier field. Even if both the video segment and the audio segment use binary content type identifiers, the client can still accurately distinguish between the video segment and the audio segment by relying on the segment identifier field.

[0046] The specific names of the audio segment identifier field, format field, sampling rate field, and number of sampling points field mentioned above are for illustrative purposes. In a formal implementation, other custom field names or conventions can also be used, as long as the client can distinguish between video segments and audio segments and can know or pre-agree on the number of sampling points and sampling rate of the audio segment. When the client and server pre-agree on the number of sampling points and sampling rate, the sampling rate field and the number of sampling points field do not need to be repeated in the audio segment header.

[0047] It should be noted that although this embodiment describes the order of "writing video segments first, then writing adjacent audio segments", the server can also adopt the order of "writing audio segments first, then writing adjacent video segments", as long as the client processes the adjacent relationship between the two according to the same convention. Both orders belong to the interleaved writing method described in this invention.

[0048] Step S3: The client reads the HTTP Multipart response in a streaming manner and separates the video and audio segments according to the boundary markers.

[0049] In step S3, the client reads the HTTP Multipart response in a streaming manner and separates the video and audio segments according to boundary markers. As a specific implementation, the client continuously reads response bytes through an HTTP response reading interface that supports on-demand parsing. For example, in a browser or embedded WebView, the client can obtain the HTTP response by initiating an HTTP request and then read byte blocks in a loop using a streaming reader that retrieves the response body (i.e., a readable streaming interface that supports reading bytes block by block), obtaining one byte array each time. Because the HTTP Multipart response is sent in blocks during network or proxy transmission, each read byte block may contain only a portion of a segment, or it may contain multiple segments, and it may be truncated in the middle of the boundary markers. Therefore, the client maintains a byte buffer to accumulate bytes that have not yet been fully parsed.

[0050] The client performs the following processing during parsing: First, it appends the newly read byte block to the end of the byte buffer. Second, it searches for boundary markers in the byte buffer, such as the byte sequence "--frame". Third, for cases where the boundary marker may span adjacent read blocks, the client, during boundary marker identification, retains the last byte of the previous read block whose length is not less than the boundary marker length minus one. It then merges the last byte of the previous read block with the starting byte of the next read block to determine the starting boundary of adjacent fragments. After identifying the fragment starting boundary, it parses the fragment header and reads the fragment body of the corresponding length based on the content length field in the fragment header. For example, if the boundary marker "--frame" is 7 bytes long, when the end of the byte buffer may contain part of this boundary marker, the client retains at least the last 6 bytes, temporarily not determining them as fragment body content. After the next read block arrives, it concatenates these last bytes with the starting byte of the next read block before matching. Therefore, even if the boundary marker is truncated at the intersection of two read blocks, the client can still correctly identify the complete boundary marker, avoiding missed or misjudged fragment boundaries.

[0051] Fourth, once the client recognizes the complete boundary marker, it continues parsing the segment header, using the blank line between the segment header and the segment body as the separator to locate the end position of the segment header and the start position of the segment body. Then, it reads the corresponding number of bytes based on the content length field in the segment header to obtain the complete segment body. Fifth, the client determines whether the current segment is a video segment or an audio segment based on the content type, audio segment identifier field, or other agreed fields in the segment header: if it is an audio segment, its segment body is sent to the audio processing flow; if it is a video segment, its segment body is sent to the video processing flow.

[0052] Step S4: The client queues the audio segments into the audio playback queue and uses the scheduled start time or the actual start time of the first audio segment as the audio anchor point.

[0053] See Figure 4 In step S4, the client queues the audio segments into the audio playback queue, using the scheduled start time or actual start time of the first audio segment as the audio anchor point. As a specific implementation, the client decodes the arriving audio segments into audio buffers, that is, converts the PCM audio data in the audio segment body into an audio buffer object acceptable to the audio playback interface. When the audio segment body is in PCM S16LE format, the client converts the 16-bit signed integer sample values ​​into floating-point sample values. For example, it proportionally normalizes integer samples ranging from -32768 to 32767 to floating-point samples ranging from -1.0 to 1.0, and writes them into the audio buffer object.

[0054] The client submits audio buffers to the audio output queue according to their arrival order in the HTTP Multipart response. As a specific implementation, the client maintains a variable indicating the scheduled start time of the next audio buffer. The first audio buffer is scheduled to start playback at a small safety offset after the current time of the audio playback clock, for example, 20 milliseconds after the current time, to allow for submission and scheduling leeway. The scheduled start times of each subsequent audio buffer are arranged sequentially by adding a frame interval to the scheduled start time of the previous audio buffer, thus ensuring continuous and seamless playback of the audio buffers in arrival order.

[0055] When the first audio buffer is submitted to the audio output queue, the client determines the scheduled start time or the actual start time of that first audio buffer and uses this scheduled start time or the actual start time as the audio anchor point. The audio anchor point is the time corresponding to the start of playback of the first frame of audio on the aforementioned unified clock source. In one specific implementation, the client can use the absolute playback time specified when arranging the playback of the first audio buffer as the audio anchor point (i.e., the scheduled start time), or it can obtain the actual start time through a callback when the first audio buffer actually starts playing or by querying the current time of the audio playback clock and use it as the audio anchor point. Thereafter, the audio playback clock continues to advance with the actual audio playback, serving as a unified time reference for determining subsequent video frames to be rendered.

[0056] Step S5: The client determines the frame interval based on the number of sampling points and sampling rate of the audio segment. Without needing to read the frame sequence number, PTS / DTS or playback timestamp from the segment header of the HTTP Multipart response, the client determines the video frame to be rendered based on the elapsed time of the audio playback clock relative to the audio anchor point, and draws the video frame to be rendered to the image display buffer.

[0057] See Figure 4 In step S5, the client determines the frame interval based on the number of sampling points and the sampling rate of the audio segments. Without needing to read the frame sequence number, PTS / DTS, or playback timestamp from the segment header of the HTTP Multipart response, the client determines the video frame to be rendered based on the elapsed time of the audio playback clock relative to the audio anchor point, and then draws the video frame to be rendered to the image display buffer. This step can be further broken down into operations such as video segment caching, determining the frame interval and target frame number, selecting the video frame to be rendered, and drawing, as detailed below.

[0058] As a specific implementation corresponding to video fragment caching, the client caches the video fragments obtained in step S3 as candidate video frames. For PNG or WebP format video fragments, the client can asynchronously decode the video fragment body into a drawable image object through the image decoding interface; for Raw BGRA format video fragments, the client can convert the video fragment body into pixel data acceptable to the image display buffer based on the width and height fields in the video fragment header. The client establishes a candidate video frame queue according to the arrival order of the video fragments in the HTTP Multipart response, using the first video fragment as the starting frame, and determines incremental frame numbers for subsequent video fragments according to their arrival order.

[0059] For a specific implementation corresponding to the determination of frame interval and target frame number, see [link to implementation]. Figure 4The client determines the video frame to be rendered as follows: First, the client divides the number of sampling points in the audio segment by the sampling rate to obtain the frame interval, i.e., the frame interval is equal to the ratio of the number of sampling points to the sampling rate. For example, when the number of sampling points is 800 and the sampling rate is 16000Hz, the frame interval is equal to 800 divided by 16000, which is 0.05 seconds. Second, the client uses the difference between the current time of the audio playback clock and the audio anchor point as the audio playback elapsed duration, i.e., the current audio playback time minus the time corresponding to the audio anchor point. Third, the client divides the audio playback elapsed duration by the frame interval and rounds down to obtain the target frame number, i.e., the target frame number is equal to the quotient of the audio playback elapsed duration and the frame interval, rounded down.

[0060] The above relationship can be expressed as follows: Let the number of sampling points in each audio segment be Ns, the audio sampling rate be Fs, the frame interval be Δt, the audio anchor point be T0, the current time of the audio playback clock be Tc, the audio playback duration be E, and the target frame number be k, then the frame interval... Audio playback Target frame number The result is then rounded down. The client uses the first video segment as the starting frame and determines incremental frame numbers for subsequent video segments according to their arrival order in the HTTP Multipart response. In one implementation where the starting frame is designated as frame 0, the target frame number k directly corresponds to the kth video frame in the candidate video frame queue; in another implementation where the starting frame is designated as frame 1, the target frame number can be incremented by 1 and then matched with the frame order in the candidate video frame queue; a unified starting method is used within the same embodiment. Finally, the client selects the video frame corresponding to the target frame number from the cached video segments as the video frame to be rendered. Thus, the video frame to be rendered is entirely determined by the audio playback progress, and the segment header does not need to carry frame sequence numbers, PTS, DTS, or playback timestamps.

[0061] As a specific implementation corresponding to the lead-or-lag handling of the target frame number, when the client selects a video frame to be rendered from the cached video segments, it processes the selection based on the relationship between the target frame number and the position of already rendered frames in the candidate video frame queue. When the target frame number is less than the position of already rendered frames in the candidate video frame queue, it indicates that the rendering loop or queue state has exceeded that frame. The client keeps the current image display buffer content unchanged, thus avoiding screen rollback. When the target frame number is greater than the position of already rendered frames in the candidate video frame queue, the client skips video frames that have not yet been rendered before the target frame number and uses the video frame corresponding to the target frame number as the video frame to be rendered, thus catching up with the frame position corresponding to the audio when rendering lags behind the audio. When the target frame number is equal to the position of already rendered frames, the client can maintain the current screen. When the video frame corresponding to the target frame number has not yet arrived, the client can maintain the current screen and wait for the subsequent video segments to arrive. This processing ensures that video rendering follows the audio playback clock, rather than the network arrival time, thereby reducing the impact of network jitter, parsing latency, or image decoding latency on audio-visual synchronization.

[0062] As a specific implementation corresponding to rendering, the client decodes the video frame to be rendered into a drawable image object, or converts the video frame to be rendered into pixel data according to the Raw BGRA format, and then draws the drawable image object or pixel data to the image display buffer. In one implementation that uses the canvas as the image display buffer, for drawable image objects decoded by the image decoding interface, the client can directly draw them to the canvas; for video frames in Raw BGRA format, the client can convert the pixels arranged in blue, green, red, and transparency into pixel data arranged in red, green, blue, and transparency, write it into an image data object, and then draw the image data object to the canvas. The client can repeatedly execute the above process of reading the current time of the audio playback clock, calculating the target frame number, selecting and drawing the video frame to be rendered in a frame-by-frame rendering callback or a timed rendering loop, so that the picture continuously updates with the audio playback progress.

[0063] In terms of deployment, the server exposes only one HTTP port, and the single HTTP persistent connection is an HTTP / 1.1 persistent connection. When the HTTP Multipart response is forwarded via an HTTP reverse proxy, the reverse proxy is configured to either disable response buffering or use streaming forwarding to maintain the fragment output order and boundary identifier byte sequence. For example, the reverse proxy's buffering of the response can be disabled, or it can be configured for streaming forwarding to ensure that the client continuously and sequentially receives each fragment in the HTTP Multipart response. Even if the HTTP response bytes are sent in chunks by the proxy, the client's boundary resolution still identifies the boundary identifiers across read blocks by preserving the last byte, ensuring stable fragment resolution. Since the HTTP protocol can penetrate most enterprise firewalls and proxies, this invention has good firewall and proxy compatibility, and the server can be deployed with zero or low configuration.

[0064] Several specific examples will be given below: Example 1: Digital Human WebView Playback Based on PNG Video Segments and PCM Audio Segments.

[0065] The server receives a PCM S16LE format audio stream with a sampling rate of 16000Hz. Every 800 samples read, the server forms an audio segment, corresponding to 50 milliseconds of audio content. This 50-millisecond audio segment is fed into the digital human inference engine to generate a BGR image frame, which is then encoded into a PNG format video segment. The server establishes an HTTP / 1.1 persistent connection and returns an HTTP Multipart response with the content type set to multipart / x-mixed-replace and the boundary set to frame. For each 50-millisecond audio segment, the server first writes the PNG video segment generated from it, then writes the audio segment itself, outputting them adjacent to each other. The client initiates an HTTP request to obtain this response and reads byte blocks cyclically through the response body's streaming reader. The client segments the audio segments according to the frame boundary, identifying segments with the content type image / png as video segments and binary segments with an audio segment identifier field containing the value "audio" as audio segments. The client converts the PCM S16LE data of the audio segments into floating-point samples and writes them to the audio buffer. The first audio buffer is scheduled to play at a safe offset after the current time of the audio playback clock, and its scheduled start time is recorded as the audio anchor point T0. Subsequent audio buffers are played continuously at 50-millisecond frame intervals. In the rendering loop, the client reads the current time Tc of the audio playback clock, calculates the audio playback duration E, which is equal to Tc minus T0, and the target frame number k is equal to E divided by 0.05 seconds and rounded down. The client then selects the k-th frame from the cached PNG video frames, decodes it through the image decoding interface, and draws it onto the canvas. In this embodiment, the segment header does not need to carry frame sequence numbers, PTS, DTS, or playback timestamps. The synchronization relationship between video frames and audio playback is determined by the number of sampling points, the sampling rate, and the audio playback clock.

[0066] Example 2: Low decoding overhead playback based on Raw BGRA video segments.

[0067] In this embodiment, after the server-side digital human inference engine outputs BGR image frames, it no longer encodes them as PNG or WebP, but converts them into Raw BGRA format video segments. The server writes a content type `application / octet-stream` with a value of `raw-bgra` into the video segment header, and also writes image width and height fields into the segment header. After the client recognizes the Raw BGRA video segment, it converts the pixels of the segment body arranged in blue, green, red, and transparency according to the width and height fields into pixel data arranged in red, green, blue, and transparency, writes it into an image data object, and then draws it onto the canvas. Since the image encoding and decoding steps are eliminated, this embodiment is suitable for scenarios where the server and client are on the same local area network or local machine, have sufficient bandwidth, and want to reduce image encoding and decoding latency. Its audio anchor point determination and target frame number calculation methods are the same as in Embodiment 1.

[0068] Example 3: WebView digital human display deployed via HTTP reverse proxy.

[0069] In enterprise intranet or showroom deployment scenarios, the server exposes only one HTTP port and is located behind the HTTP reverse proxy. The client is a browser or embedded WebView. When the reverse proxy forwards the HTTP Multipart response, it uses a streaming forwarding configuration (e.g., disabling buffering for the response) to maintain the sequence of fragment boundary identifier bytes and the fragment output order. In this deployment method, the client still continuously receives video and audio fragments through the same HTTP long connection. Even if the HTTP response bytes are sent in chunks by the proxy and the boundary identifier is truncated at the intersection of two read blocks, the client's boundary resolution still identifies cross-read block boundaries by retaining the last byte of the previous read block whose length is not less than the boundary identifier length minus one and merging it with the starting byte of the next read block, ensuring stable fragment resolution.

[0070] Example 4: Handling of target frame number being ahead or behind.

[0071] The client establishes a candidate video frame queue and uses the first video segment as the starting frame according to the arrival order of the video segments, with subsequent video segments numbered sequentially. At a certain rendering moment, the client calculates the target frame number and compares it with the position of the already rendered frame: if the target frame number is less than the position of the already rendered frame, the client maintains the current image display buffer content and does not backtrack; if the target frame number is greater than the position of the already rendered frame, the client skips the video frames that have not yet been rendered before the target frame number and directly renders the video frame corresponding to the target frame number; if the video frame corresponding to the target frame number has not yet arrived, the client maintains the current image and waits for the subsequent segments to arrive before rendering. For example, if a network jitter causes frames 5 to 8 to arrive simultaneously, and the target frame number corresponding to the audio playback duration is already frame 8, the client skips frames 5 to 7 and directly renders frame 8, bringing the image to the same position as the audio. This processing method makes video rendering follow the audio playback clock, rather than the network arrival time, thereby reducing the impact of network jitter, parsing latency, or image decoding latency on audio-visual synchronization.

[0072] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of protection of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. For example, adjusting the specific values ​​of the number of sampling points, sampling rate, segmentation format, field names, boundary markers, or security offsets, or swapping the adjacent writing order of video segments and audio segments, are all within the scope of protection of the present invention.

Claims

1. A lightweight audio-visual synchronization transmission method based on HTTP Multipart response, characterized in that, include: The server obtains the PCM audio stream used for real-time audio-visual generation, divides it into continuous audio segments according to a fixed number of sampling points, and generates corresponding video frames based on each audio segment. The server establishes a single long HTTP connection with the client, returns an HTTP Multipart response to the client, and writes the video and audio segments as adjacent segments interleaved in the HTTP Multipart response according to the order of the audio segments. The client reads the HTTP Multipart response in a streaming manner and separates the video and audio segments according to boundary markers; The client queues audio segments into the audio playback queue and uses the scheduled start time or the actual start time of the first audio segment as the audio anchor point. The client determines the frame interval based on the number of sampling points and sampling rate of the audio segments. Without needing to read the frame sequence number, PTS / DTS or playback timestamp from the segment header of the HTTPMultipart response, the client determines the video frame to be rendered based on the elapsed time of the audio playback clock relative to the audio anchor point and draws the video frame to be rendered to the image display buffer.

2. The method according to claim 1, characterized in that, The server acquires the PCM audio stream used for real-time audio-visual generation, divides it into continuous audio segments according to a fixed number of sampling points, and generates corresponding video frames based on each audio segment, including: The server receives PCM S16LE format audio input for digital human audio-visual generation and forms 50ms audio segments with each audio segment containing 800 sampling points and a sampling rate of 16000Hz. Each 50ms audio segment is fed into the digital human inference engine to obtain a BGR image frame corresponding to that audio segment.

3. The method according to claim 1, characterized in that, The server establishes a single long HTTP connection with the client, returns an HTTP Multipart response to the client, and interleaves video and audio segments as adjacent segments in the HTTP Multipart response according to the order of the audio segments, including: The server sets the content type to multipart / x-mixed-replace in the HTTP response header and sets a boundary identifier to distinguish adjacent fragments; Write a video segment header and video segment body to each video segment; write an audio segment header and audio segment body to each audio segment; The video segment generated from the same audio segment is then output adjacent to the audio segment in the HTTP Multipart response.

4. The method according to claim 2, characterized in that, The step of sending each 50ms audio segment into the digital human inference engine to obtain a BGR image frame corresponding to that audio segment includes: Encode the BGR image frames output by the digital human inference engine into video segments in PNG, WebP, or Raw BGRA format; When video segments are in PNG or WebP format, the content type corresponding to that format should be written in the video segment header; When video segments are in Raw BGRA format, a binary content type identifier is written into the video segment header.

5. The method according to claim 3, characterized in that, The process of writing an audio segment header and an audio segment body into each audio segment includes: Write a binary content type identifier into the audio segment header. The binary content type identifier is used to indicate that the audio segment body is an unencapsulated audio byte stream. Write an audio segment identifier field to indicate that the segment is an audio segment. Write the corresponding PCM audio data into the audio segment body, so that the client can send the segment into the audio processing flow according to the audio segment identifier field.

6. The method according to claim 1, characterized in that, The client reads the HTTPMultipart response in a streaming manner and separates the video and audio segments according to boundary markers, including: The client continuously reads response bytes through an HTTP response reading interface that supports simultaneous reception and parsing; During the reading process, the boundary marker is identified, and when the boundary marker crosses an adjacent reading block, the last byte of the previous reading block with a length not less than the length of the boundary marker minus one is retained. The last byte of the previous reading block is merged with the starting byte of the next reading block to determine the starting boundary of the adjacent fragment. After identifying the fragment starting boundary, the fragment header is parsed, and the fragment body of the corresponding length is read according to the content length field in the fragment header.

7. The method according to claim 1, characterized in that, The client arranges audio segments into an audio playback queue, and uses the scheduled start time or actual start time of the first audio segment as the audio anchor point, including: The client decodes the arriving audio segments into an audio buffer; The audio buffers are submitted to the audio output queue in the order they arrive in the HTTP Multipart response; When the first audio buffer is submitted to the audio output queue, the predetermined start time or the actual start time of the first audio buffer is determined, and the predetermined start time or the actual start time is used as the audio anchor point.

8. The method according to claim 1, characterized in that, The client determines the frame interval based on the number of sampling points and sampling rate of the audio segments. Without needing to read the frame sequence number, PTS / DTS, or playback timestamp from the segment header of the HTTP Multipart response, it determines the video frame to be rendered based on the elapsed duration of the audio playback clock relative to the audio anchor point, including: The client divides the number of sampling points in the audio segment by the sampling rate to obtain the frame interval; The difference between the current time of the audio playback clock and the audio anchor point is used as the elapsed audio playback time. Divide the audio playback duration by the frame interval and round down to obtain the target frame number; The first video segment is used as the starting frame, and the subsequent video segments are assigned incremental frame numbers according to the arrival order of the video segments in the HTTP Multipart response; Select the video frame corresponding to the target frame number from the cached video segments as the video frame to be rendered.

9. The method according to claim 8, characterized in that, The step of selecting the video frame corresponding to the target frame number from the cached video fragments as the video frame to be rendered includes: The client establishes a queue of candidate video frames according to the arrival order of the video segments in the HTTP Multipart response; When the target frame number is less than the position of the rendered frame in the candidate video frame queue, maintain the contents of the current image display buffer. When the target frame number is greater than the position of the rendered frame in the candidate video frame queue, skip the unrendered video frames before the target frame number and use the video frame corresponding to the target frame number as the video frame to be rendered; when the target frame number is equal to the position of the rendered frame in the candidate video frame queue, or when the video frame corresponding to the target frame number has not yet been cached, keep the contents of the current image display buffer.

10. The method according to claim 1, characterized in that, The step of drawing the video frame to be rendered to the image display buffer includes: The client decodes the video frames to be rendered into drawable image objects, or converts the video frames to be rendered into pixel data in Raw BGRA format; Draw the drawable image object or pixel data to the image display buffer.

Images