A Frame Synthesis and Audio / Video Processing Method Based on PIS System

By accelerating the processing of raw video frames and overlay layers with hardware and processing audio independently, the problem of repeated full-screen capture and synchronization difficulties in the PIS system is solved, achieving efficient resource utilization and audio-video synchronization, which is suitable for multimedia information push scenarios.

CN123123630APending Publication Date: 2026-11-17HUNAN SUPERSTRING TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611628203.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-21
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing PIS systems suffer from problems such as repeated full-screen capture, invalid encoding, and difficulties in audio and video synchronization, resulting in high resource consumption, weak synchronization capabilities, and difficulty in meeting the needs of multimedia information push.

Method used

A frame synthesis and audio/video processing method based on the PIS system is adopted. The original video frames and overlay layers are processed by hardware acceleration, and the audio is processed independently. Hardware synthesis and hardware encoding are used to achieve synchronous output of video and audio, reducing invalid screenshots and duplicate encoding.

Benefits of technology

It significantly reduces repeated full-screen capture and invalid encoding, lowers bandwidth and computing power overhead, improves audio and video synchronization stability and scalability, and is suitable for various playback modes and output scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123630A_ABST
    Figure CN123123630A_ABST
Patent Text Reader

Abstract

This invention discloses a frame synthesis and audio / video processing method based on a PIS system. The control unit issues a playback mode command, and the video source unit selects a video source from network video, local files, or central file video data accordingly. The video source is then decoded to obtain the original video frame, which is stored in the synthesis buffer unit. When the interface detection and acquisition unit detects a change in the interface state, it acquires the corresponding interface area image as an overlay layer and stores it in the layer buffer unit. The hardware synthesis unit reads the original video frame and the overlay layer, writes both into a shared buffer, and synthesizes them to obtain a synthesized image frame. The hardware encoding unit performs a first encoding on the synthesized image frame to obtain a video stream. The audio processing unit performs a second decoding, sampling rate processing, and second encoding on the audio data to obtain an audio stream. The transmission encapsulation unit encapsulates the audio data into transmission stream packets according to the video timestamp and audio timestamp respectively. The timestamps are compared using the same time base, and the transmission time is adjusted to achieve synchronized audio and video output.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of passenger information system and multimedia processing technology, specifically relating to a frame synthesis and audio-visual processing method based on a PIS system, so as to achieve unified synthesis, encoding, encapsulation and synchronous output of the original video frame and the interface overlay layer after separate processing. Background Technology

[0002] Currently, PIS (Passenger Information System) relies on multimedia network technology to push multimedia information. Its application scenarios are relatively wide. The front end collects audio and video streams of digital or analog signals, and processes train time, direction of travel, station information, alarm information, etc., and finally provides passengers with multimedia audio and video playback and station service information on the carriage display screen.

[0003] This field involves multimedia processing, video compositing, hardware encoding, and streaming media delivery.

[0004] Existing full-screen video capture solutions typically recapture the entire screen or window as a single bitmap at every moment before encoding and transmission. While simple to implement, this approach presents three main problems: First, the need to capture the entire desktop results in additional memory copying and UI resource consumption. Second, even if only partial alerts, subtitles, or status bar changes occur, the entire background must still be captured repeatedly. Third, audio often needs to be spliced ​​separately, which may struggle to maintain stable synchronization with the captured frames due to card performance bottlenecks or peak performance.

[0005] Specifically, the three typical problems mentioned above arise from the following reasons: First, the full screenshot solution requires capturing the entire desktop, and each capture involves reading, copying, and format conversion of the entire screen's pixels, resulting in significant additional memory copying and UI resource consumption. Second, even if only partial alarms, subtitles, or status bar changes occur in the interface, the entire background must still be captured repeatedly, leading to high pixel redundancy and a heavier encoding burden. Third, audio often needs to be spliced ​​separately, and due to board performance bottlenecks or peak performance, it may be difficult to maintain stable synchronization with the screenshot frames, resulting in weak audio-video synchronization capabilities.

[0006] In scenarios requiring video playback, interface overlay, alarm switching, and audio synchronization, it is necessary to avoid the problems of high copying, high coupling, and synchronization difficulties caused by using a complete screenshot method, while maintaining a clear clock relationship and a directly distributable bitstream format in the output screen.

[0007] Therefore, how to reduce invalid screenshots, lower the encoding burden, improve the stability of audio and video synchronization, and enhance scalability are technical problems that urgently need to be solved by those skilled in the art. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention provides a frame synthesis and audio / video processing method based on a PIS system, overcoming the technical shortcomings of existing technologies such as repeated full-screen capture, invalid encoding, and difficulties in audio / video synchronization. This solution can be summarized as a main link: "raw video frame → overlay layer → RGA synthesis → MPP hard encoding → TS encapsulation → network / storage output." The audio link separately completes "audio decoding → resampling → AAC encoding → ADTS / TS encapsulation → output," ultimately achieving audio / video synchronization at the transmission layer through timestamps and queue control.

[0009] This invention separates the original video frame, overlay layer, audio transcoding, and transmission encapsulation into different modules, with hardware synthesis and hardware encoding jointly completing the final output. Compared to a complete video screenshot solution, this solution retains the original video frame as the main image, only making local updates to the changed overlay layer, and the main video frame can reuse hardware decoding or file decoding results; while the complete video screenshot solution directly captures the entire screen or window of pixels, and even if only a corner is changed, the entire screenshot must be generated repeatedly.

[0010] The core methods of this invention include: using raw video frames for hardware acceleration, which is orders of magnitude faster and uses fewer resources than traditional software-based desktop capture; subsequent frame processing is based on RGA acceleration and an on-demand synthesis scheme, achieving on-demand allocation of resources rather than processing all resources at once. Specifically, this invention retains the raw video frames as the main image, only updating the changed overlay layers locally; the main video frames can reuse hardware decoding or file decoding results; image pasting and format conversion are performed by a hardware image processing unit, and the synthesized final image enters the hardware encoder for low-latency output; audio is independently decoded, resampled, and encoded using AAC, with timestamps and video links controlled separately; finally, audio and video synchronization is achieved at the transmission layer through timestamps and queue control.

[0011] Thus, this invention realizes a complete closed loop from video source selection, original video frame decoding, overlay layer acquisition, hardware synthesis, hardware encoding, audio processing to transmission encapsulation and synchronous output, significantly reducing full-screen repeated acquisition and invalid encoding, reducing bandwidth and computing power overhead, and improving audio and video synchronization stability.

[0012] According to one aspect of the present invention, a frame synthesis and audio / video processing method based on a PIS system is provided. The PIS system includes a control unit, a video source unit, an interface detection and acquisition unit, a synthesis buffer unit, a layer buffer unit, a hardware synthesis unit, a hardware encoding unit, an audio processing unit, and a transmission encapsulation unit, comprising: The control unit sends a playback mode instruction to the video source unit, and the playback mode instruction carries the source identifier of the video data to be processed. The video source unit selects one path from network video data, local file video data, or central file video data as the video data to be processed based on the source identifier. The video source unit performs a first decoding on the video data to be processed to obtain the original video frame, and stores the original video frame into the synthesis buffer unit; When the interface detection and acquisition unit detects a change in the interface state of the PIS system, it acquires an image of the interface area corresponding to the change in the interface state and stores the interface area image as an overlay layer in the layer cache unit. The hardware compositing unit reads the original video frame from the compositing buffer unit, reads the overlay layer from the layer buffer unit according to the playback mode instruction, and writes both the original video frame and the overlay layer into a pre-allocated shared buffer for compositing, or writes the original video frame into a pre-allocated shared buffer for compositing, to obtain a composite image frame. The hardware encoding unit performs a first encoding on the synthesized image frame to obtain a video stream; The audio processing unit performs second decoding, sampling rate processing, and second encoding on the audio data to obtain an audio bitstream. The transmission encapsulation unit encapsulates the video stream into a video transmission stream packet according to the video timestamp, and encapsulates the audio stream into an audio transmission stream packet according to the audio timestamp; The transmission encapsulation unit obtains the video timestamp corresponding to the video transmission stream packet and the audio timestamp corresponding to the audio transmission stream packet, compares the video timestamp and the audio timestamp with the same time base, and adjusts the transmission time of the video transmission stream packet and the audio transmission stream packet according to the comparison result, so that the video transmission stream packet and the audio transmission stream packet are output synchronously according to the same time base.

[0013] Compared with the prior art, the present invention has the following beneficial effects: First, it reduces invalid screenshots. Compared to capturing the entire video, this solution only updates the changing overlay, eliminating the need to reconstruct the entire screen image for each frame.

[0014] Second, it reduces the encoding burden. The original video frames and overlay layers are combined on the hardware side and then encoded, which reduces the overhead of repeated pixel handling and software stitching.

[0015] Third, it facilitates audio and video synchronization. Video, audio, and transmission clock are processed in layers, which helps maintain a stable frame rate under different input sources and playback modes.

[0016] Fourth, it offers better scalability. This solution is suitable for live streaming, local file playback, central file playback, and alarm scenarios, and can switch between different overlay layers and output channels according to the mode.

[0017] The table below compares the proposed solution with the complete screenshot video solution across several metrics:

[0018] This invention is not only applicable to train carriage displays, but also to other PIS application scenarios that require multimedia information push and station service information, and has broad application prospects. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the overall process of the frame synthesis and audio / video processing method based on the PIS system provided in the embodiments of the present invention, showing the complete process from the issuance of playback mode instructions, video source selection, first decoding, overlay layer acquisition, hardware synthesis, first encoding, audio processing to transmission encapsulation and synchronous output; Figure 2 This is a flowchart of the main video link provided in an embodiment of the present invention, illustrating the specific steps from issuing playback mode instructions, selecting video sources, first decoding, storing in the synthesis buffer, reading original video frames, hardware synthesis, first encoding to generating video bitstream; Figure 3 This is a flowchart illustrating the process of overlay layer acquisition and hardware synthesis provided in this embodiment of the invention, showing the specific steps from interface state detection, interface area image acquisition, storage in layer cache, reading overlay layer, writing to shared buffer to obtaining synthesized image frame; Figure 4 This is a flowchart illustrating the audio link and transmission encapsulation process provided in this embodiment of the invention, showing the specific steps from second decoding, sampling rate determination, resampling or skipping resampling, second encoding, AAC encoding, adding ADTS header information to transmission encapsulation and synchronous output. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The embodiments of this invention are written in a progressive manner.

[0023] Terminology Explanation: PIS (Passenger Information System) refers to a system that uses multimedia network technology to provide passengers with multimedia audio and video playback and station service information. It is widely used in rail transit, public transportation and other scenarios.

[0024] RGA (Raster Graphic Acceleration) is a hardware image processing unit used to perform two-dimensional graphics acceleration operations such as image pasting, format conversion, scaling, and rotation.

[0025] MPP (Media Processing Platform) refers to a hardware media processing platform that includes modules such as VENC (Video Encoder) for hardware encoding of video frames.

[0026] VENC (Video Encoder) refers to the video encoding module in MPP, which is used to create encoding context, input buffer and output buffer, and asynchronously send the frames to be encoded into the encoding queue, and then retrieve the bitstream from the output queue.

[0027] YUV refers to a color encoding method, where Y represents luminance and U and V represent chrominance. It is commonly used in image formats before video encoding.

[0028] PTS (Presentation Time Stamp) is a timestamp used to identify the display time of audio and video frames, and is used for audio and video synchronization.

[0029] TS (Transport Stream) is a container format used for audio and video transmission, which encapsulates video and audio streams into transport stream packets according to their respective timestamps.

[0030] ADTS (Audio Data Transport Stream) is a transport stream format for AAC audio data. Its header information includes a sync word, sample rate index, channel configuration, and frame length field.

[0031] AAC (Advanced Audio Coding) is an audio coding format used to compress and encode pulse code modulation data.

[0032] PCM (Pulse Code Modulation) is a method of converting analog audio signals into digital audio signals to obtain pulse code modulated data.

[0033] The terms used above are only for explaining specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art will understand that other existing technologies with the same or similar functions can be employed without departing from the concept of the present invention.

[0034] Example 1: Overall Architecture like Figure 1 As shown, this invention provides a frame synthesis and audio / video processing method based on a PIS system. Its core lies in separating the original video frame, overlay layers, audio transcoding, and transmission encapsulation into different modules, with hardware synthesis and hardware encoding jointly completing the final output. The PIS system includes a control unit, a video source unit, an interface detection and acquisition unit, a synthesis buffer unit, a layer buffer unit, a hardware synthesis unit, a hardware encoding unit, an audio processing unit, and a transmission encapsulation unit. These units work collaboratively to achieve a complete link from video source selection to synchronized audio and video output.

[0035] Specifically, the PIS system includes the following functional units: The control unit issues playback mode commands to the video source unit, which carry the source identifier of the video data to be processed. This source identifier instructs the video source unit to select one of the following as the video data to be processed: network video data, local file video data, or central file video data. Supported playback modes include: analog signal input, network signal input, and local file playback.

[0036] The video source unit is used to select the video data to be processed according to the source identifier, perform the first decoding of the video data to be processed to obtain the original video frame, and store the original video frame into the synthesis buffer unit.

[0037] This unit retains the original video frames as the main screen and does not directly take full-screen screenshots. For local playback and center playback modes, the interface layer only captures the areas that need to be overlaid, such as the main interface, alarm interface, or scrolling text area.

[0038] This function supports the following playback modes: analog signal input, network signal input, and local file playback. In analog signal input and network signal input modes, the video source unit selects the video data to be processed from network video data; in local file playback mode, the video source unit selects the video data to be processed from local file video data; and in central file playback mode, the video source unit selects the video data to be processed from central file video data.

[0039] The interface detection and acquisition unit is used to acquire images of the interface area corresponding to the detected change in the interface state of the PIS system, and store the interface area image as an overlay layer in the layer cache unit. The overlay layer is not re-captured from the full screen every frame, but is cached as a QImage or similar pixel cache after the interface is triggered. For example, in normal display mode, the visible area of ​​the main window is captured, and in alarm mode, the alarm panel image is captured. The cached layer is then reused in the compositing thread to reduce the frequency of repeated acquisition.

[0040] Synthesis buffer unit, used to store raw video frames.

[0041] Layer cache unit, used to store overlay layers.

[0042] The hardware compositing unit is used to read the original video frames from the compositing buffer unit, read the overlay layer from the layer buffer unit according to the playback mode instruction, and write both the original video frames and the overlay layer into a pre-allocated shared buffer for compositing, or write the original video frames into a pre-allocated shared buffer for compositing, to obtain the composite image frame.

[0043] The shared buffer is pre-allocated before the hardware compositing unit performs compositing, and its capacity corresponds to the size of the composite image frame. The original video frame and the overlay layer complete image pasting and format conversion in the shared buffer.

[0044] In terms of implementation, the main interface determines whether to synthesize a desktop image or an alarm image based on the mode state. During synthesis, a shared buffer and RGA are used for image pasting, outputting a YUV image of uniform size, which is then sent to the Rockchip MPP hardware encoder to generate an H.264 bitstream.

[0045] Wherein, RGA is the raster graphics acceleration unit of the Rockchip platform, MPP is the media processing platform of the Rockchip platform, and VENC is the video encoding module in the MPP.

[0046] The overlay layer is not re-captured from the full screen every frame. Instead, it is cached as a QImage or similar pixel cache after being triggered by the interface. For example, in normal display mode, the visible area of ​​the main window is captured, and in alarm mode, the image of the alarm panel is captured. The cached layer is then reused in the compositing thread to reduce the frequency of repeated capture.

[0047] In the compositing stage, RGA handles image stitching and format conversion, and video frames and overlays are written to a shared target buffer. In the encoding stage, the VENC module creates the MPP encoding context, input buffer, and output buffer, asynchronously sending frames to be encoded into the encoding queue and retrieving the bitstream from the output queue. This structure ensures that both image stitching and compression encoding utilize hardware paths as much as possible.

[0048] The hardware encoding unit is used to perform the first encoding on the synthesized image frames to obtain the video bitstream.

[0049] In implementation, the hardware encoding unit creates an MPP encoding context, input buffer, and output buffer through the VENC module, asynchronously sends the frames to be encoded into the encoding queue, and then retrieves the bitstream from the output queue to generate an H.264 bitstream. This structure ensures that both image stitching and compression encoding follow the hardware path as much as possible.

[0050] The audio processing unit is used to perform secondary decoding, sampling rate processing, and secondary encoding on the audio data to obtain the audio bitstream.

[0051] The audio link is processed independently of the video link. For file playback paths, the system first uses FFmpeg to read the audio stream, and then performs resampling, PCM processing, and AAC encoding as needed; if the input stream already meets the target sampling rate, some resampling steps can be skipped. The encoded AAC data is then supplemented with ADTS information and finally handed over to the TS encapsulation and transmission module.

[0052] The key to this design is that the audio does not depend on the screenshot results, but maintains an independent time base, and is then aligned with the video at the sending layer through a unified PTS strategy.

[0053] The transmission encapsulation unit is used to encapsulate the video stream into a video transport stream packet according to the video timestamp, and the audio stream into an audio transport stream packet according to the audio timestamp. It also obtains the video timestamp corresponding to the video transport stream packet and the audio timestamp corresponding to the audio transport stream packet, compares the video timestamp and audio timestamp with the same time base, and adjusts the transmission time of the video transport stream packet and audio transport stream packet according to the comparison result, so that the video transport stream packet and audio transport stream packet are output synchronously with the same time base.

[0054] Video and audio are ultimately unified into the TS encapsulation module. On the video side, the H.264 bitstream is encapsulated into TS packets according to frame number and timestamp, while on the audio side, AAC frames are encapsulated into TS packets according to sampling intervals, and then output through multicast or RTSP related interfaces. The system uses queue length control, throttling waiting, and PTS increment strategies to suppress backlogs caused by inconsistent upstream and downstream speeds.

[0055] The aforementioned units work together to form a complete main link: "raw video frame → overlay layer → RGA synthesis → MPP hard encoding → TS encapsulation → network / storage output". The audio link separately completes "audio decoding → resampling → AAC encoding → ADTS / TS encapsulation → output", and finally achieves audio and video synchronization at the transmission layer through timestamps and queue control.

[0056] Example 2: General Process like Figure 2 As shown, the present invention provides a frame synthesis and audio / video processing method based on a PIS system, comprising the following steps S1 to S5: S1: Playback mode command issuance and video source selection This step is performed by the control unit and the video source unit. The control unit sends a playback mode command to the video source unit, which carries the source identifier of the video data to be processed. Based on the source identifier, the video source unit selects one of the following as the video data to be processed: network video data, local file video data, or central file video data. This function supports the following playback modes: analog signal input, network signal input, and local file playback.

[0057] The source identifier is used to indicate the source of the video data to be processed, and the source identifier corresponds to the playback mode indicated by the playback mode instruction; when the playback mode instruction indicates analog signal input or network signal input, the source identifier indicates that the video data to be processed comes from network video data; when the playback mode instruction indicates local file playback, the source identifier indicates that the video data to be processed comes from local file video data or central file video data.

[0058] S2: Video Main Link: First Decoding and Synthesis Buffer The video source unit performs the first decoding of the video data to be processed, obtaining the raw video frames, and stores the raw video frames in the compositing buffer unit. In the main process, the system first selects network video, local file video, or central file video according to the current playback mode. After the video frames are decoded, they are not directly used to take full-screen screenshots, but are instead entered into the compositing buffer.

[0059] S3: Overlay Layer Acquisition and Hardware Compositing When the interface detection and acquisition unit detects a change in the interface state of the PIS system, it acquires an image of the interface area corresponding to the change and stores this image as an overlay layer in the layer cache unit. The overlay layer is not re-captured from the full screen every frame; instead, it is cached as a QImage or similar pixel cache after the interface is triggered.

[0060] The hardware compositing unit reads the original video frames from the compositing buffer unit and the overlay layer from the layer buffer unit according to the playback mode instruction. It then writes both the original video frames and the overlay layer into a pre-allocated shared buffer for compositing, or writes only the original video frames into the pre-allocated shared buffer for compositing, resulting in a composite image frame. During the compositing stage, the RGA handles image pasting and format conversion, outputting a YUV image of uniform size.

[0061] S4: Hardware Coding and Audio Processing The hardware encoding unit performs the first encoding on the synthesized image frames to obtain the video bitstream. During the encoding stage, the VENC module creates the MPP encoding context, input buffer, and output buffer, asynchronously sends the frames to be encoded into the encoding queue, and then retrieves the bitstream from the output queue.

[0062] The audio processing unit performs secondary decoding, sampling rate processing, and secondary encoding on the audio data to obtain the audio bitstream. The audio link is processed independently of the video link. For file playback paths, the system first uses FFmpeg to read the audio stream, then performs resampling, PCM processing, and AAC encoding as needed; if the input stream already meets the target sampling rate, some resampling steps can be skipped. ADTS information is then added to the encoded AAC data.

[0063] S5: Transmission Encapsulation and Synchronous Output The transmission encapsulation unit encapsulates the video stream into video transport stream packets according to the video timestamp and the audio stream into audio transport stream packets according to the audio timestamp; it obtains the video timestamp corresponding to the video transport stream packet and the audio timestamp corresponding to the audio transport stream packet, compares the video timestamp and audio timestamp with the same time base, and adjusts the sending time of the video transport stream packet and audio transport stream packet according to the comparison result, so that the video transport stream packet and audio transport stream packet are output synchronously with the same time base.

[0064] The key to this design is that the audio does not rely on screenshot results but maintains an independent time base, and is then aligned with the video at the transmission layer using a unified PTS strategy. Both video and audio ultimately enter the TS encapsulation module. On the video side, the H.264 bitstream is encapsulated into TS packets according to frame number and timestamp; on the audio side, AAC frames are encapsulated into TS packets according to sampling intervals, and then output through multicast or RTSP-related interfaces. The system uses queue length control, throttling, and PTS incrementing strategies to suppress backlogs caused by inconsistent upstream and downstream speeds.

[0065] Example 3: Application of Interface State Switching and Overlay Layer Acquisition Combination Figure 3 As shown in the figure, this embodiment illustrates the specific acquisition method of the interface detection and acquisition unit under different interface states.

[0066] The PIS system interface status includes normal display status, alarm display status, and scrolling text update status.

[0067] The interface detection and acquisition unit detects whether the interface state has changed by changing the state flag bit. The state flag bit is updated by the interface layer of the PIS system when the interface state is switched.

[0068] When the interface status changes from normal display to alarm display, the interface detection and acquisition unit acquires the image of the alarm panel area corresponding to the alarm display state and stores the alarm panel area image as an overlay layer in the layer cache unit. In implementation, the main interface determines whether to composite a desktop image or an alarm image based on the mode status; during compositing, a shared buffer and RGA are used for image pasting, outputting a YUV image of uniform size, which is then sent to the Rockchip MPP hardware encoder to generate an H.264 bitstream.

[0069] When the interface state changes from normal display state to scrolling text update state, the interface detection and acquisition unit acquires the scrolling text area image corresponding to the scrolling text update state, and stores the scrolling text area image as an overlay layer in the layer cache unit. For example, in normal display state, the visible area of ​​the main window is captured, and in alarm state, the alarm panel image is captured; then, the cached layer is reused in the compositing thread to reduce the frequency of repeated acquisition.

[0070] When the interface state is detected to switch from alarm display state to normal display state, the interface detection and acquisition unit acquires the image of the main interface area corresponding to the normal display state, and stores the main interface area image as an overlay layer in the layer cache unit.

[0071] In this embodiment, the overlay layer is not re-captured from the full screen every frame, but is cached as a QImage or similar pixel cache after being triggered by the interface, thereby reducing invalid screenshots and reducing the coding burden.

[0072] Example 4: Playback Mode and Hardware Synthesis Application Combination Figure 2 and Figure 3 As shown in the figure, this embodiment illustrates the specific synthesis method of the hardware synthesis unit in different playback modes.

[0073] When the playback mode instruction indicates that the video data to be processed comes from local file video data or central file video data, the hardware compositing unit reads the original video frame from the compositing buffer unit and the overlay layer from the layer buffer unit. Both the original video frame and the overlay layer are written to a pre-allocated shared buffer for compositing to obtain a composite image frame. For local playback and central playback modes, the interface layer only captures the area to be overlaid, such as the main interface, alarm interface, or scrolling subtitle area, and then overlays it with the current video frame through the compositing function.

[0074] When the playback mode instruction indicates that the video data to be processed comes from network video data, the hardware synthesis unit reads the original video frame from the synthesis buffer unit, writes the original video frame into a pre-allocated shared buffer, and obtains the synthesized image frame.

[0075] When the hardware compositing unit includes a hardware image processing subunit, the hardware compositing unit reads the original video frames from the compositing buffer unit and reads the overlay layers from the layer buffer unit according to the playback mode instruction. The hardware image processing subunit then writes both the original video frames and the overlay layers into a pre-allocated shared buffer, or writes the original video frames into the pre-allocated shared buffer. In the shared buffer, image pasting and format conversion are performed on the written original video frames and overlay layers, or the format conversion is performed on the written original video frames to obtain the composite image frames. During the compositing stage, the RGA handles image pasting and format conversion, and the video frames and overlay layers are written to the shared target buffer.

[0076] In this embodiment, the original video frames and the overlay layer are synthesized on the hardware side and then encoded, which reduces the overhead of repeated pixel handling and software splicing.

[0077] Example 5: Audio Link Application Combination Figure 4 As shown, this embodiment illustrates the specific processing method of the audio processing unit.

[0078] The audio processing unit performs second decoding, sampling rate processing, and second encoding on the audio data to obtain an audio bitstream, including: The audio data is then decoded a second time to obtain pulse code modulation data; Determine whether the sampling rate of the pulse code modulation data is consistent with the preset target sampling rate; When the sampling rate of the pulse code modulation data is inconsistent with the target sampling rate, the pulse code modulation data is resampled to obtain the resampled pulse code modulation data, and the resampled pulse code modulation data is used as the pulse code modulation data to be encoded. When the sampling rate of the pulse code modulation data is consistent with the target sampling rate, the pulse code modulation data is used as the pulse code modulation data to be encoded. The pulse code modulation data to be encoded is then subjected to a second encoding to obtain the audio bitstream.

[0079] The target sampling rate is preset by the audio processing unit or set according to the output requirements of the transmission encapsulation unit.

[0080] The process involves a second encoding of the pulse code modulation data to be encoded, resulting in an audio bitstream, including: The pulse code modulation data to be encoded is AAC encoded to obtain AAC audio data; Based on the sampling rate, number of channels, and frame length of the AAC audio data, ADTS header information is added before the AAC audio data to obtain the audio bitstream. The ADTS header information includes a synchronization word, sampling rate index, channel configuration, and frame length field.

[0081] In this embodiment, the audio link is processed independently of the video link. For the file playback path, the system first uses FFmpeg to read the audio stream, and then performs resampling, PCM processing, and AAC encoding as needed; if the input stream already meets the target sampling rate, some resampling steps can be skipped. The encoded AAC data is then supplemented with ADTS information and finally handed over to the TS encapsulation and transmission module. The audio does not depend on the screenshot results, but maintains an independent time base, and is then aligned with the video at the transmission layer using a unified PTS strategy.

[0082] Example 6: Transmission Encapsulation and Synchronous Output Application Combination Figure 4 As shown, this embodiment illustrates the specific processing method of the transmission encapsulation unit.

[0083] The transmission encapsulation unit encapsulates the video stream into video transport stream packets according to the video timestamp, and encapsulates the audio stream into audio transport stream packets according to the audio timestamp, including: Determine the video timestamp corresponding to each video frame in the video stream based on the frame number of the video stream; Each video frame in the video stream and its corresponding video timestamp are encapsulated into a video transport stream packet. Based on the sampling interval of the audio bitstream, determine the audio timestamp corresponding to each audio frame in the audio bitstream; Each audio frame in the audio stream and its corresponding audio timestamp are encapsulated into an audio transport stream packet.

[0084] The transmission encapsulation unit obtains the video timestamp corresponding to the video transport stream packet and the audio timestamp corresponding to the audio transport stream packet, compares the video timestamp and audio timestamp with the same time base, and adjusts the transmission time of the video transport stream packet and audio transport stream packet according to the comparison result, including: Obtain the video timestamp corresponding to the video transport stream packet and the audio timestamp corresponding to the audio transport stream packet; Based on the same time base, determine the first transmission time corresponding to the video timestamp and the second transmission time corresponding to the audio timestamp; If the first transmission time is earlier than the second transmission time, the transmission of the video transport stream packet is delayed; When the second transmission time is earlier than the first transmission time, the audio transport stream packet is transmitted with a delay.

[0085] The same time reference is the media unified clock where the transmission encapsulation unit is located, and the video timestamp and the audio timestamp are both generated based on the system unified clock.

[0086] In this embodiment, video and audio are ultimately unified into the TS encapsulation module. On the video side, the H.264 bitstream is encapsulated into TS packets according to frame number and timestamp, and on the audio side, AAC frames are encapsulated into TS packets according to sampling point intervals, and then output through multicast or RTSP related interfaces.

[0087] The multicast interface is used to send the video and audio transport stream packets to the PIS system's carriage display screen in a multicast manner; the RTSP interface is used to respond to the carriage display screen's playback request and send the video and audio transport stream packets to the carriage display screen in a streaming media manner (passive mode), or output them to the LCD display screen in a multicast manner (active mode).

[0088] The system suppresses backlogs caused by inconsistent speeds between upstream and downstream processes through queue length control, throttling waiting, and PTS increment strategies.

[0089] Example 7: Application of Train Carriage Display Screens Taking train carriage display screens as an example, the front end collects digital or analog audio and video streams, and processes train time, direction of travel, station information, alarm information, etc., and finally provides passengers with multimedia audio and video playback and station service information on the carriage display screen.

[0090] This function is implemented at the relay layer. Pure audio and video are generated and pushed from the ground, which can be understood as a television source (analog signal of type AV, digital signal of the network layer). The image is then processed and overlaid with station information, such as current station: xxx, next station: xxx, to create a complete image, which is then pushed to the large screen in the train carriage via the network. This function supports playback modes including analog signal input, network signal input, and local file playback.

[0091] In scenarios requiring video playback, interface overlay, alarm switching, and audio synchronization, this embodiment avoids the problems of high copying, high coupling, and synchronization difficulties caused by using a complete screenshot method, while maintaining a clear clock relationship and a directly distributable bitstream format in the output screen.

[0092] This invention is not only applicable to train carriage displays, but also to other PIS application scenarios that require multimedia information push and station service information, and has broad application prospects.

[0093] One or more embodiments in this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments in this application should be included within the protection scope of this application.

[0094] If flowcharts are used in this application, they are used to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these flows, or one or more steps can be removed from these flows.

[0095] The foregoing has provided a detailed description of a frame synthesis and audio / video processing method based on a PIS system provided in this application. The above description of the disclosed embodiments enables those skilled in the art to implement or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A frame composition and audio-video processing method based on a PIS system, the PIS system comprising a control unit, a video source unit, an interface detection and acquisition unit, a composition cache unit, a layer cache unit, a hardware composition unit, a hardware encoding unit, an audio processing unit and a transmission and packaging unit, characterized in that, include: The control unit sends a playback mode instruction to the video source unit, and the playback mode instruction carries the source identifier of the video data to be processed. The video source unit selects one path from network video data, local file video data, or central file video data as the video data to be processed based on the source identifier. The video source unit performs a first decoding on the video data to be processed to obtain the original video frame, and stores the original video frame into the synthesis buffer unit; When the interface detection and acquisition unit detects a change in the interface state of the PIS system, it acquires an image of the interface area corresponding to the change in the interface state and stores the interface area image as an overlay layer in the layer cache unit. The hardware compositing unit reads the original video frame from the compositing buffer unit, and reads or does not read the overlay layer from the layer buffer unit according to the playback mode instruction. It writes both the original video frame and the overlay layer into a pre-allocated shared buffer for compositing, or writes the original video frame into a pre-allocated shared buffer for compositing, to obtain a composite image frame. The hardware encoding unit performs a first encoding on the synthesized image frame to obtain a video stream; The audio processing unit performs second decoding, sampling rate processing, and second encoding on the audio data to obtain an audio bitstream. The transmission encapsulation unit encapsulates the video stream into a video transmission stream packet according to the video timestamp, and encapsulates the audio stream into an audio transmission stream packet according to the audio timestamp; The transmission encapsulation unit obtains the video timestamp corresponding to the video transmission stream packet and the audio timestamp corresponding to the audio transmission stream packet, compares the video timestamp and the audio timestamp with the same time base, and adjusts the transmission time of the video transmission stream packet and the audio transmission stream packet according to the comparison result, so that the video transmission stream packet and the audio transmission stream packet are output synchronously according to the same time base.

2. The method according to claim 1, characterized in that, The interface states of the PIS system include normal display state, alarm display state, and scrolling text update state. When the interface detection and acquisition unit detects a change in the interface state of the PIS system, it acquires an image of the interface region corresponding to the change in interface state, and stores the interface region image as an overlay layer in the layer cache unit, including: When the interface state is detected to switch from the normal display state to the alarm display state, the alarm panel area image corresponding to the alarm display state is acquired, and the alarm panel area image is stored as the overlay layer in the layer cache unit.

3. The method according to claim 2, characterized in that, When the interface detection and acquisition unit detects a change in the interface state of the PIS system, it acquires an image of the interface region corresponding to the interface state change and stores the interface region image as an overlay layer in the layer cache unit. The system also includes: When the interface state is detected to switch from the normal display state to the scrolling text update state, the scrolling text area image corresponding to the scrolling text update state is acquired, and the scrolling text area image is stored as the overlay layer in the layer cache unit.

4. The method according to claim 2, characterized in that, When the interface detection and acquisition unit detects a change in the interface state of the PIS system, it acquires an image of the interface region corresponding to the interface state change and stores the interface region image as an overlay layer in the layer cache unit. The system also includes: When the interface state is detected to switch from the alarm display state to the normal display state, the main interface area image corresponding to the normal display state is acquired, and the main interface area image is stored as the overlay layer in the layer cache unit.

5. The method according to claim 1, characterized in that, The hardware synthesis unit includes a hardware image processing subunit; The hardware compositing unit reads the original video frame from the compositing buffer unit, reads the overlay layer from the layer buffer unit according to the playback mode instruction, and writes both the original video frame and the overlay layer into a pre-allocated shared buffer for compositing, or writes the original video frame into a pre-allocated shared buffer for compositing, to obtain a composite image frame, including: The original video frame is read from the synthesis buffer unit, and the overlay layer is read from the layer buffer unit according to the playback mode instruction; The hardware image processing subunit writes both the original video frame and the overlay layer into a pre-allocated shared buffer, or writes the original video frame into a pre-allocated shared buffer. In the shared buffer, the written original video frame and the overlay layer are image-fitted and format-converted, or the written original video frame is format-converted to obtain the composite image frame.

6. The method according to claim 1, characterized in that, The hardware compositing unit reads the original video frame from the compositing buffer unit, and reads or does not read the overlay layer from the layer buffer unit according to the playback mode instruction. It then writes both the original video frame and the overlay layer into a pre-allocated shared buffer for compositing, or writes the original video frame into a pre-allocated shared buffer for compositing, to obtain a composite image frame, including: When the playback mode instruction indicates that the video data to be processed comes from local file video data or central file video data, the original video frame is read from the synthesis cache unit, the overlay layer is read from the layer cache unit, and the original video frame and the overlay layer are both written into a pre-allocated shared buffer for synthesis to obtain the synthesized image frame. When the playback mode instruction indicates that the video data to be processed originates from network video data, the original video frame is read from the synthesis buffer unit, the overlay layer is not read, and the original video frame is written into a pre-allocated shared buffer to obtain the synthesized image frame.

7. The method according to claim 1, characterized in that, The audio processing unit performs second decoding, sampling rate processing, and second encoding on the audio data to obtain an audio bitstream, including: The audio data is then subjected to the second decoding to obtain pulse code modulation data; Determine whether the sampling rate of the pulse code modulation data is consistent with the preset target sampling rate; When the sampling rate of the pulse code modulation data is inconsistent with the target sampling rate, the pulse code modulation data is resampled to obtain resampled pulse code modulation data, and the resampled pulse code modulation data is used as the pulse code modulation data to be encoded. When the sampling rate of the pulse code modulation data is consistent with the target sampling rate, the pulse code modulation data is used as the pulse code modulation data to be encoded. The pulse code modulation data to be encoded is subjected to the second encoding to obtain the audio bitstream.

8. The method according to claim 7, characterized in that, The second encoding of the pulse code modulation data to be encoded to obtain the audio bitstream includes: The pulse code modulation data to be encoded is AAC encoded to obtain AAC audio data; Based on the sampling rate, number of channels, and frame length of the AAC audio data, ADTS header information is added before the AAC audio data to obtain the audio bitstream. The ADTS header information includes a synchronization word, sampling rate index, channel configuration, and frame length field.

9. The method according to claim 1, characterized in that, The transmission encapsulation unit encapsulates the video stream into a video transport stream packet according to the video timestamp, and encapsulates the audio stream into an audio transport stream packet according to the audio timestamp, including: Determine the video timestamp corresponding to each video frame in the video stream according to the frame number of the video stream; Each video frame in the video stream and its corresponding video timestamp are encapsulated into a video transport stream packet. Based on the sampling point interval of the audio bitstream, determine the audio timestamp corresponding to each audio frame in the audio bitstream; Each audio frame in the audio stream and its corresponding audio timestamp are encapsulated into an audio transport stream packet.

10. The method according to claim 9, characterized in that, The transmission encapsulation unit obtains the video timestamp corresponding to the video transport stream packet and the audio timestamp corresponding to the audio transport stream packet, compares the video timestamp and the audio timestamp with the same time base, and adjusts the transmission time of the video transport stream packet and the audio transport stream packet according to the comparison result, including: Obtain the video timestamp corresponding to the video transport stream packet and the audio timestamp corresponding to the audio transport stream packet; Based on the same time reference, determine the first transmission time corresponding to the video timestamp and the second transmission time corresponding to the audio timestamp; If the first transmission time is earlier than the second transmission time, the transmission of the video transport stream packet is delayed; When the second transmission time is earlier than the first transmission time, the audio transport stream packet is transmitted with a delay.