A Collaborative Suppression Method for Video Stream Packet Loss and Screen Flickering Based on Heterogeneous SoC NPU

By working together with the main control unit and the NPU to distinguish and process abnormal data packets, perform out-of-order reordering and fragment splicing, and combine NAL unit integrity verification and reference availability status, the problem of abnormal data packet delivery control during concurrent multi-channel video decoding and artificial intelligence inference is solved, thereby improving the stability and consistency of video decoding.

CN123123586APending Publication Date: 2026-11-17WUHAN LINGJIU MICROELECTRONICS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611632953.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-21
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

When multiple video decoding and artificial intelligence inference occur concurrently, there is a problem that the verification results of abnormal data packets caused by out-of-order or packet loss in network transmission are difficult to keep in line with the delivery control of NAL unit, especially when the hardware decoder receives incomplete fragments or references unavailable images.

Method used

The master controller distinguishes between normal and abnormal data packets, uses the NPU to perform out-of-order reordering and fragment splicing, combines NAL unit integrity verification to form the code stream processing result, and performs delivery control based on the reference availability status to ensure that the integrity of the NAL unit and the reference dependency relationship meet the requirements.

Benefits of technology

The system associates the verification results of abnormal data packets with the NAL unit, ensuring the accuracy of delivery control, reducing the probability of the hardware decoder receiving incomplete bitstreams, and improving the stability and consistency of video decoding.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123586A_ABST
    Figure CN123123586A_ABST
Patent Text Reader

Abstract

This invention relates to the field of video stream transmission and hardware decoding fault tolerance technology, and discloses a collaborative suppression method for video stream packet loss and screen tearing based on a heterogeneous SoC NPU. The method receives and buffers RTP data packets from multiple video streams; the master controller forms a message status record based on sequence continuity and FU start / end status; normal data packets are de-encapsulated and concatenated into NAL units; abnormal candidate data packets are submitted to the NPU for rearrangement, concatenation, and integrity verification, associating the verification results with the corresponding NAL units; a delivery control result is formed based on the integrity of the NAL units and the reference availability status, and the release unit is sent to the VPU for decoding; successfully decoded images are written to the valid frame buffer; in case of decoding failure, the most recent valid frame is output, and the reference availability status of subsequent NAL units is updated based on the decoding result. This method ensures that the verification result of abnormal bitstreams corresponds to the hardware decoding input, reducing the number of bitstreams with incomplete fragments or referencing unavailable reference images entering the VPU.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of video stream transmission and hardware decoding fault tolerance technology, and in particular to a collaborative suppression method for video stream packet loss and screen tearing based on heterogeneous system-on-a-chip (SoC) and neural network processor (NPU). Background Technology

[0002] Multi-channel real-time video is typically transmitted via RTP data packets. The receiving end performs out-of-order reordering and fragment splicing based on the packet sequence and fragmentation information, and then hands the resulting Network Abstraction Layer (NAL) unit to the hardware video decoder for decoding. Some end-side devices also run artificial intelligence inference tasks simultaneously. The receiving end typically has the main controller responsible for packet inspection and bitstream organization, and uses historical valid frames to compensate for images with decoding errors.

[0003] When multiple video decoding and AI inference occur concurrently, and network transmission experiences out-of-order delivery or packet loss, message inspection, abnormal bitstream handling, and hardware decoding input may be at different processing stages. If the data packets involved in a sequence number break have not yet been rearranged or their integrity checked, subsequent NAL units may have already entered the delivery queue. For video bitstreams with reference dependencies, simply checking the fragment integrity of the current unit is insufficient to reflect the availability of the reference images it references.

[0004] When the processing progress and reference state are not aligned, the verification results of abnormal data packets can easily become disconnected from the NAL units to be delivered. The hardware decoder may receive bitstreams with incomplete fragments or references to unusable reference images. After a decoding anomaly occurs, the delivery judgment of subsequent units may still use an outdated reference state. Therefore, in scenarios involving concurrent multi-channel video decoding and artificial intelligence inference, it is necessary to address the delivery control problem of maintaining consistency between the verification results of abnormal data packets, the corresponding NAL units, and the available reference state. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a collaborative suppression method for video stream packet loss and screen tearing based on heterogeneous SoC NPU, comprising: S100: Receive and buffer RTP data packets from multiple video streams; The master controller reads the RTP sequence number and the FU start flag and FU end flag in the RTP payload according to the video stream, and marks normal data packets and abnormal candidate data packets according to the sequence continuity and FU start and end status to form a message status record. S200. Based on the message status record, normal data packets are de-encapsulated and concatenated into NAL units; abnormal candidate data packets are submitted to the NPU for out-of-order reordering, FU fragmentation and concatenation, and NAL unit integrity verification; the verification results are associated with the corresponding NAL units to form the code stream processing results. S300: Based on the bitstream processing result and the reference availability status of the corresponding video stream, mark the corresponding NAL unit as allowed or blocked according to the NAL unit integrity and reference dependency relationship to form a delivery control result; send the allowed NAL unit into the VPU according to the delivery control result; S400: Read the decoding result of the VPU for the released NAL unit; write the successfully decoded image into the valid frame buffer; when decoding fails, call the output of the most recent valid frame; and update the reference available state according to the decoding result for subsequent NAL unit calls.

[0006] Furthermore, in S100, the RTP sequence number and RTP timestamp are recorded according to the video stream identifier; for data packets using FU fragmentation, the FU start flag and FU end flag in the RTP payload are read; the message status record is generated based on the continuity of the RTP sequence number of the same video stream and the FU start and end status.

[0007] Furthermore, when the RTP sequence numbers of the same video stream are consecutive and the corresponding FU start and end status is complete, the corresponding data packet is marked as the normal data packet; when there is an RTP sequence number gap, out-of-order, or incomplete FU start and end status, the corresponding data packet is marked as the abnormal candidate data packet, and the corresponding video stream identifier and RTP sequence number are retained in the message status record.

[0008] Further, in S200, the NPU reads the abnormal candidate data packets from the shared memory according to the message status record; rearranges the corresponding data packets according to the RTP sequence number using OpenCL, concatenates NAL units according to the FU start flag and FU end flag, and records the integrity result of the concatenated NAL units.

[0009] Furthermore, the integrity result is correlated with the video stream identifier to which the abnormal candidate data packet belongs and the NAL unit obtained by splicing; for the NAL unit obtained by splicing the normal data packet, its corresponding video stream identifier and fragment integrity result are recorded to form the bitstream processing result.

[0010] Furthermore, in S300, when the image to be delivered is determined to be an IDR frame based on the encoding type information in the bitstream processing result, and the NAL unit required by the IDR frame is incomplete or its integrity is not confirmed, a blocking mark is formed on the IDR frame, the reference availability status of the video stream to which it belongs is set to unavailable, and a refresh request is sent to the sending end of the video stream.

[0011] Furthermore, based on the reference availability status and the GOP reference dependency relationship of the corresponding video stream, P frames referencing unavailable reference images are blocked; broken B frames are blocked based on the bitstream processing result, and release marks are formed for complete and reference-available NAL units.

[0012] Furthermore, in S400, the images successfully decoded by the VPU are written into the circular valid frame buffer in the order of decoding; when the VPU decoding fails, the most recently written valid image in the circular valid frame buffer is read as the current output image.

[0013] Furthermore, when the NPU simultaneously undertakes both the AI ​​inference task and the bitstream preprocessing task for the abnormal candidate data packets, the AI ​​inference task is submitted first; when the NPU task occupancy is high, the bitstream preprocessing task is submitted later, and it remains in a waiting-for-delivery state until the corresponding NAL unit obtains the integrity result.

[0014] Furthermore, upon receiving the IDR frame corresponding to the refresh request, the integrity of the NAL units constituting the IDR frame is checked based on the bitstream processing result; when all the NAL units required for the IDR frame meet the delivery conditions, a release mark is formed for the corresponding NAL units and sent to the VPU; after the VPU successfully decodes the IDR frame, the reference availability state of the corresponding video stream is updated, and subsequent NAL units are processed according to the updated reference availability state.

[0015] The key innovations of this invention include: (1) The master controller distinguishes normal data packets and abnormal candidate data packets based on the continuity of the Real-time Transport Protocol (RTP) sequence and the start and end status of the fragmentation unit (FU). Normal data packets are decapsulated and spliced ​​before entering the regular delivery path. Abnormal candidate data packets are handed over to the Neural Processing Unit (NPU) for rearrangement, splicing and verification. The verification results are associated with the corresponding Network Abstraction Layer (NAL) unit to form the unit-by-unit bitstream processing results.

[0016] (2) Delivery control simultaneously checks the integrity of the NAL unit and the reference availability status of the video stream to which it belongs; it forms a blocking mark for units that reference unavailable reference images and a release mark for units that meet both conditions, so that the bitstream processing result corresponds to the specific input unit of the video processing unit (VPU).

[0017] (3) The decoding results of the VPU are applied to the valid frame buffer and the reference available state respectively; the successfully decoded image enters the buffer, the most recent valid frame is called during the decoding exception, and the reference available state participates in the delivery control of the subsequent NAL unit.

[0018] The following are its main beneficial effects: (1) In response to the situation where the main control continuously processes all bitstreams when receiving multiple video streams and the abnormal verification results are easily disconnected from the units to be delivered, the split processing ensures that normal data packets do not occupy the NPU bitstream preprocessing task; after the verification results of abnormal data packets are associated with the corresponding NAL unit, the subsequent delivery control can read the integrity conclusion of the current unit itself.

[0019] (2) For the case where the unit fragment is complete but the reference image is unavailable, the integrity and the reference availability status jointly constrain the VPU input; the corresponding unit is blocked before entering the hardware decoder, reducing the situation where such bitstreams continue to participate in the reference relationship after entering the decoder.

[0020] (3) In the case that the release conclusion of the bitstream layer may be inconsistent with the actual decoding result, the VPU decoding anomaly will change the reference available state used by the subsequent units; the effective frame buffer also provides the most recently successfully decoded image for the current output during the anomaly period, avoiding writing the abnormal decoded image into the buffer. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a collaborative suppression method for video stream packet loss and screen tearing based on a heterogeneous SoC NPU, provided in an embodiment of this application. Detailed Implementation

[0022] Example 1: Refer to Figure 1 This is a flowchart illustrating a collaborative suppression method for video stream packet loss and screen tearing based on a heterogeneous SoC NPU, provided by an embodiment of the present invention. The process may include at least steps S100-S400: S100: Receive and buffer RTP data packets from multiple video streams; The master controller reads the RTP sequence number and the FU start flag and FU end flag in the RTP payload according to the video stream, and marks normal data packets and abnormal candidate data packets according to the sequence continuity and FU start and end status to form a message status record. S200. Based on the message status record, normal data packets are de-encapsulated and concatenated into NAL units; abnormal candidate data packets are submitted to the NPU for out-of-order reordering, FU fragmentation and concatenation, and NAL unit integrity verification; the verification results are associated with the corresponding NAL units to form the code stream processing results. S300: Based on the bitstream processing result and the reference availability status of the corresponding video stream, mark the corresponding NAL unit as allowed or blocked according to the NAL unit integrity and reference dependency relationship to form a delivery control result; send the allowed NAL unit into the VPU according to the delivery control result; S400: Read the decoding result of the VPU for the released NAL unit; write the successfully decoded image into the valid frame buffer; when decoding fails, call the output of the most recent valid frame; and update the reference available state according to the decoding result for subsequent NAL unit calls.

[0023] S100: Receive and buffer RTP data packets from multiple video streams; The master controller reads the RTP sequence number and the FU start flag and FU end flag in the RTP payload according to the video stream, and marks normal data packets and abnormal candidate data packets according to the sequence continuity and FU start and end status to form a message status record. In one implementation, this method operates on a heterogeneous SoC (System on Chip) integrating a main controller, an NPU (Neural Processing Unit), and a VPU (Video Processing Unit). After multiple video streams enter the receive buffer via the network interface, the main controller caches the RTP (Real-time Transport Protocol) data packets of each video stream in shared memory. During the pre-inspection phase, the main controller reads the packet header and the payload preamble required to identify fragment boundaries, without performing integrity checks byte-by-byte on the entire video payload. Each video stream maintains its own receive status, and if one video stream experiences out-of-order delivery or sequence number breaks, the data packet markings of other video streams are not altered.

[0024] After receiving an RTP data packet, the master controller looks up the previously saved reception status of that video stream based on the video stream identifier and matches the RTP sequence number of the current data packet with the previous reception position. The RTP timestamp is used to help distinguish data packets from different images, but multiple NAL units may exist under the same timestamp; the timestamp alone cannot determine whether a fragment of a NAL unit has been fully received. In one implementation, the reception status also retains the position of the current data packet in shared memory, and subsequent bitstream processing reads the payload according to this position, eliminating the need for the master controller to copy the entire data packet during pre-check.

[0025] For data packets transmitted in FU (Fragmentation Unit) format, the master controller reads the start and end flags of the FU header in the RTP payload and verifies the header, intermediate fragments, and trailer of a fragment sequence in conjunction with the RTP sequence number. The FU start and end flags are located in the RTP payload and are not part of the RTP fixed header fields. For cases where a single data packet carries a complete NAL (Network Abstraction Layer) unit, the master controller records its reception status according to the payload encapsulation format of the data packet, without marking it as abnormal due to the absence of FU start and end flags. For an FU fragment sequence, the fragment sequence is only included in the normal path if the sequence relationships of the start flag, end flag, and intermediate data packets correspond.

[0026] When data packets for the same video stream arrive consecutively according to their RTP sequence numbers, and the corresponding FU fragments have a corresponding start point, end point, and consecutive intermediate fragments, the master controller marks the corresponding data packets as normal data packets. If the received sequence number exceeds the expected sequence number, the master controller records the location of the gap and marks the data packets affected by the gap as abnormal candidate data packets. This marking indicates that a gap was found during the pre-inspection phase, but does not mean that the missing data packets are determined to be unrecoverable. If subsequent data packets falling into the gap are received, the master controller updates the reception status of that video stream, and S200 still performs verification based on the actual set of fragments received. In the case where the starting fragment has not yet been received, but the ending fragment has arrived, the master controller retains the state of incomplete fragment boundaries and does not infer the content of the preceding payload based on the ending flag.

[0027] When the arrival order of data packets is inconsistent with the RTP sequence number order, the master controller retains the payload and buffer position of the original data packets in shared memory, and marks the affected video stream and sequence number in the message status record. Subsequently, the S200 completes the rearrangement. For multiple video streams arriving simultaneously, the buffer position of each data packet corresponds to the reception status of that video stream. Data packets that have not yet completed pre-inspection, rearrangement, or delivery processing still occupy the corresponding buffer position. After processing is completed, the receiving side reuses the buffer according to the buffer usage status to avoid the position being overwritten by newly arrived data while the NPU is reading abnormal candidate data packets.

[0028] The message status record is thus associated with the video stream identifier, the sequence status of the inspected data packet, the start and end status of the FU and its buffer location, and distinguishes between normal data packets and abnormal candidate data packets; the S200 selects the regular decapsulation path or the NPU bitstream preprocessing path accordingly. The message status record only expresses the reception status seen by the master control pre-inspection. Whether the NAL unit is ultimately complete is still checked by the S200 after decapsulating, rearranging or splicing the corresponding data packets. The pre-inspection flag cannot be directly used as the VPU release conclusion.

[0029] S200. Based on the message status record, normal data packets are de-encapsulated and concatenated into NAL units; abnormal candidate data packets are submitted to the NPU for out-of-order reordering, FU fragmentation and concatenation, and NAL unit integrity verification; the verification results are associated with the corresponding NAL units to form the code stream processing results. After obtaining the message status record formed by S100, the receiving side accesses the buffer locations of normal data packets and abnormal candidate data packets according to the video stream. For normal data packets, the transmission message header is first removed according to the RTP payload encapsulation method of the corresponding video encoding format, and then the FU fragments belonging to the same NAL unit are sequentially concatenated. When a single RTP data packet carries a complete NAL unit, the decapsulated payload directly enters the normal delivery path of that unit; when carried by multiple FU fragments, the NAL unit for VPU input control is formed only after concatenation. The normal path does not submit stream preprocessing tasks to the NPU, but RTP decapsulation must still be completed, and the original RTP data packets cannot be directly handed over to the VPU.

[0030] When the message status log indicates out-of-order delivery, sequence number gaps, or incomplete FU start / end status, the master controller submits an abnormal bitstream preprocessing task corresponding to that video stream to the NPU, and makes the task reference the position of the relevant data packets and RTP sequence number in shared memory. The NPU reads the payload from these positions, arranges the data packets according to the sequence relationship of the same video stream, and then splices the corresponding fragments according to the FU start and end flags. In one implementation, the NPU uses OpenCL (Open Computing Language) to perform tasks and check the sequence position and fragmentation boundaries of multiple data packets in parallel. After the task is completed, the splicing result and check status are returned to the receiving side, and the master controller does not repeat the same check on all video payloads during the task waiting period.

[0031] When RTP sequence numbers wrap around as specified in the protocol, the rearrangement process interprets the sequence relationship before and after the wrap around in conjunction with the previously saved reception status of the video stream, avoiding misinterpreting normal wrap around as a large number of missing data packets. For genuine sequence number gaps, the process continues to check whether there are any late data packets within the gap range. If a gap is detected during the pre-inspection phase, and the corresponding data packet arrives before the NPU reads it, the NPU checks the start point, end point, and intermediate sequences according to the updated fragment set. If the gap still exists after rearrangement, the bitstream processing result retains the unconfirmed incomplete state, and the missing payload is not inferred from the content of adjacent fragments.

[0032] FU segmentation begins with the starting segment of a NAL unit, sequentially connecting intermediate segments and closing the unit at the ending segment. Integrity verification checks the start and end flags and the continuity of the segment sequence. If the start or end segment is missing, the existing payload is not marked as a complete NAL unit. In the absence of a starting segment, the receiving side may not be able to accurately recover the type of the NAL unit. In this case, the bitstream processing result retains the unconfirmed complete state of the video stream and the affected segment sequence for S300 blocking delivery, without fabricating a complete NAL unit.

[0033] The verification result of the NPU task must correspond to the checked bitstream object. In one implementation, the receiving side associates a NAL unit with the video stream identifier, the range of RTP sequences involved in the splicing, and the position of the spliced ​​payload; when the NPU returns the integrity result, the receiving side maps the result to the corresponding NAL unit according to this association. The NAL unit obtained by splicing normal data packets also retains the video stream identifier, payload position, and fragment integrity result; when the S300 reads the bitstream processing result, it can obtain the verification conclusion of the current NAL unit to be delivered, and will not use the abnormal result of another unit in the same video stream for the current delivery.

[0034] When the NPU simultaneously undertakes AI (Artificial Intelligence) inference tasks and bitstream preprocessing tasks for anomalous candidate data packets, the task submission order is arranged so that AI inference tasks precede bitstream preprocessing tasks. When the NPU is under high load, bitstream preprocessing tasks are submitted later, and the corresponding NAL units remain in a waiting-for-delivery state until they obtain a complete result. The waiting state is not equivalent to normal release; the VPU input side does not receive the unverified payload corresponding to the anomalous candidate data packet during this period. In one implementation, if the waiting time has exceeded the delivery time of the current image, the receiving side ends the waiting and causes S300 to block the corresponding unit in an unconfirmed complete state.

[0035] The bitstream processing results provide the S300 with two types of processing criteria: the payload and integrity results corresponding to a specific NAL unit, and the fragmented sequences that cannot be recovered into a complete NAL unit and their unconfirmed integrity status. Both types of results retain their respective video streams, allowing the S300 to read the reference usability status of that video stream. Thus, although normal data packets and abnormal candidate data packets have passed through different processing paths, their delivery judgment still falls on the specific NAL unit or affected fragmented sequence in the same video stream.

[0036] S300: Based on the bitstream processing result and the reference availability status of the corresponding video stream, mark the corresponding NAL unit as allowed or blocked according to the NAL unit integrity and reference dependency relationship to form a delivery control result; send the allowed NAL unit into the VPU according to the delivery control result; After receiving the bitstream processing result from S200, the VPU input control process first reads the current reference availability status of the video stream according to the video stream identifier, and then checks the integrity result of the current unit. The reference availability status is a software state maintained by the receiving side based on the previous bitstream processing and VPU decoding results, and does not indicate that the receiving side directly modifies the internal reference image storage area of ​​the VPU. Fragment sequences that are still in the waiting-for-delivery state do not enter the VPU input queue; units with clear integrity results then enter the reference dependency check. Subsequent delivery judgments cannot be omitted simply because the master control pre-detection shows normal data packets.

[0037] For units with missing segments or whose integrity is not confirmed, the VPU input control process generates a blocking flag. For NAL units with complete segments, the encoding type and the reference dependency relationship of their respective Group of Pictures (GOP) are read. If an image consists of multiple NAL units, the receiver maps related units of the same image to each other according to the video encoding format. Only when the required bitstream portion of the image meets the delivery conditions is it considered a usable image candidate in subsequent reference relationships; the entire reference image cannot be considered valid simply because one NAL unit is complete.

[0038] In one implementation, the NAL type and reference relationship of the abnormal bitstream are provided by the NPU bitstream processing task after reading the corresponding encoded information, while the normal path reads the encoded information required for delivery control after forming the NAL unit. The receiving side maintains the GOP reference dependency relationship according to the video stream and uses the available status of existing reference images for the current unit verification. This relationship corresponds to a specific reference object in actual delivery, and the reference method of all images cannot be inferred solely from the names "I-frame, P-frame, B-frame". For abnormal units without a verifiable reference relationship, delivery control does not use the unverified reference relationship as a basis for release.

[0039] An Instantaneous Decoding Refresh (IDR) frame corresponds to a reference relationship refresh point. When the received encoded information can determine that the image to be delivered is an IDR frame, and the required NAL unit for that IDR frame has missing fragments or its integrity is not confirmed, the VPU input control process marks the IDR frame as blocked and sets the reference availability state of its video stream to unavailable; the receiving side sends a refresh request to the sending end of that video stream. If the image type cannot be confirmed due to the missing initial fragment, the corresponding fragment sequence is blocked according to the integrity unconfirmed state, and it is not directly judged as a broken IDR frame. Subsequent NAL units in the current GOP that depend on this refresh point are blocked according to the unavailable reference state, even if their own fragments are complete. After receiving the sender's response, the new IDR frame still needs to undergo the integrity check of the required NAL unit, and is not directly allowed because of its frame type; the reference availability state is updated after the IDR frame is successfully decoded by the VPU.

[0040] For predicted P-frames, delivery control checks the referenced images according to the GOP reference dependency relationship of the video stream; if the referenced image is unavailable, a blocking mark is formed for the P-frame. If the P-frame itself is damaged, delivery control blocks it directly based on its integrity result, no longer using the validity of the previous reference image as a reason for release. For bidirectional predicted B-frames, the damage result corresponds to a blocking mark; complete B-frames are still checked according to the actual reference relationship. B-frames in different coding structures may participate in subsequent image references. The receiver maintains the state according to the reference relationship represented by the bitstream, and does not presuppose that all B-frames have no reference function.

[0041] Delivery control results will assign release or blocking markers to the current NAL unit and its associated video stream, and retain the processing status that affects subsequent reference checks. For released NAL units, the VPU input side reads its spliced ​​payload and submits it to the VPU according to the encoding order of that video stream. RTP timestamps are involved in subsequent image output mapping, but do not replace the encoding order or reference relationship independently. Blocked units do not enter the VPU input queue, and the reference availability status of their associated video stream still participates in the delivery judgment of subsequent units.

[0042] The release flag indicates that the delivery conditions of the bitstream layer have been verified, but does not mean that the VPU has successfully decoded; if the VPU receiving unit has not returned a result, the receiving side maintains the pending confirmation state of that unit. For subsequent units referencing the pending confirmation image, in one implementation, the delivery control waits for the corresponding decoding result before verifying the reference relationship; if the required reference image is clearly unavailable, a blocking flag is directly formed. S400 then reads the actual decoding result of the VPU, updates the reference availability state, and makes the updated state available for use in the next round of S300.

[0043] S400: Read the decoding result of the VPU for the released NAL unit; write the successfully decoded image into the valid frame buffer; when decoding is abnormal, call the most recent valid frame output; and update the reference available state according to the decoding result for subsequent NAL unit calls. After the VPU performs hardware decoding on the NAL unit submitted by the S300, the receiving side reads the decoding result according to the correspondence between the video stream and the submitted units. In the implementation where the VPU asynchronously returns an image or an abnormal status, the receiving side correlates the returned result with the previous delivery control result, and units that have not yet received a result remain in a pending confirmation state. If a unit that has been marked as complete and released by the bitstream layer still experiences a VPU decoding abnormality, the receiving side handles the output and reference status according to the actual VPU result, without replacing the hardware decoding result with the previous integrity mark.

[0044] Images successfully decoded by the VPU are entered into the circular valid frame buffer in the order of decoding. This buffer stores decoded images and serves a different function from the shared memory buffer in S100 that stores RTP data packets. When a new valid image is written, the receiving side updates the write position of the circular buffer and points the most recent valid frame to that image. For units that have not formed a successfully decoded image, the previously stored most recent valid frame is not overwritten. In one embodiment, the display output and the valid frame buffer each reference the same image output by the VPU, and the buffer position remains readable until it is reused by subsequent valid images.

[0045] When a decoding error occurs in the VPU, the output side reads the most recently written valid image from the circular valid frame buffer of the video stream and uses it as the current output image. This compensation action reuses the already decoded image, without rewriting the abnormal NAL unit into a complete compressed bitstream, nor changing the reference image inside the VPU. Multiple video streams maintain their most recently valid frames separately; a decoding error in one video stream does not call the image from another. If a video stream has not yet successfully decoded an image, then that stream has no recently valid frame to call, and the receiver waits for a subsequent valid decoded image, without writing the abnormal output into the valid frame buffer.

[0046] The update of the reference availability status simultaneously verifies the reference relationships of delivered images and the VPU decoding results. When an image participating in subsequent references is successfully decoded, the receiver records that the reference image is available; when the VPU reports a decoding error for the image, the receiver records it as unavailable, and subsequent NAL units referencing it undergo a re-verification in S300. For images not participating in subsequent references, decoding errors correspond to output compensation for the current image; whether to change the subsequent reference availability status still depends on the actual reference relationships represented by the bitstream. The image validity and bitstream reference availability status in the circular valid frame buffer are maintained separately. The availability of historical images for display does not indicate that the VPU already has reference images available for subsequent units to reference.

[0047] After the sending end responds to the refresh request, the receiving end still checks the integrity of the newly received data packets and their IDR frames according to S100 and S200. The NAL units constituting the IDR frame are checked by S300 to meet the delivery conditions and sent to the VPU. When the VPU successfully decodes the IDR frame, S400 updates the reference availability state of the video stream to which it belongs based on the decoding result. Subsequent P frames and B frames are then accepted for delivery and verification according to the new GOP reference dependency relationship. If the new IDR frame still has missing fragments, or the VPU fails to decode the frame successfully, the reference availability state remains unavailable. The most recent valid frame participates in the output according to the buffer state, and subsequent units are not prematurely released because the sending end has responded to the refresh request.

[0048] Upon arrival of the next batch of RTP data packets, S100 continues to maintain the message status records for each video stream, S200 generates the corresponding bitstream processing results, and S300 reads the updated reference availability status from S400 to execute the next delivery control. Successfully decoded images enter the valid frame buffer, while blocked units and decoding error units do not enter this buffer. The actual decoding results of the previous VPU are then used for the release verification of the next NAL unit, and image compensation output and reference relationship restoration operate according to their corresponding states.

Claims

1. A collaborative suppression method for video stream packet loss and screen tearing based on a heterogeneous SoC NPU, wherein the heterogeneous SoC includes a main control unit, a neural network processor (NPU), and a video processing unit (VPU), characterized in that, include: S100: Receive and buffer RTP data packets from multiple video streams; The master controller reads the RTP sequence number and the FU start flag and FU end flag in the RTP payload according to the video stream, and marks normal data packets and abnormal candidate data packets according to the sequence continuity and FU start and end status to form a message status record. S200. Based on the message status record, normal data packets are de-encapsulated and concatenated into NAL units; abnormal candidate data packets are submitted to the NPU for out-of-order reordering, FU fragmentation and concatenation, and NAL unit integrity verification; the verification results are associated with the corresponding NAL units to form the code stream processing results. S300: Based on the bitstream processing result and the reference availability status of the corresponding video stream, mark the corresponding NAL unit as allowed or blocked according to the NAL unit integrity and reference dependency relationship to form a delivery control result; send the allowed NAL unit into the VPU according to the delivery control result; S400: Read the decoding result of the VPU for the released NAL unit; write the successfully decoded image into the valid frame buffer; when decoding fails, call the output of the most recent valid frame; and update the reference available state according to the decoding result for subsequent NAL unit calls.

2. The method according to claim 1, characterized in that, In S100, the RTP sequence number and RTP timestamp are recorded according to the video stream identifier; for data packets using FU fragmentation, the FU start flag and FU end flag in the RTP payload are read; the message status record is generated based on the continuity of the RTP sequence number of the same video stream and the FU start and end status.

3. The method according to claim 2, characterized in that, When the RTP sequence numbers of the same video stream are consecutive and the corresponding FU start and end status is complete, the corresponding data packet is marked as the normal data packet; when there is an RTP sequence number gap, out-of-order, or incomplete FU start and end status, the corresponding data packet is marked as the abnormal candidate data packet, and the corresponding video stream identifier and RTP sequence number are retained in the message status record.

4. The method according to claim 3, characterized in that, In S200, the NPU reads the abnormal candidate data packets from the shared memory according to the message status record; rearranges the corresponding data packets according to the RTP sequence number using OpenCL; splices NAL units according to the FU start flag and FU end flag; and records the integrity result of the spliced ​​NAL units.

5. The method according to claim 4, characterized in that, The integrity result is correlated with the video stream identifier to which the abnormal candidate data packet belongs and the NAL unit obtained by splicing; for the NAL unit obtained by splicing the normal data packet, its corresponding video stream identifier and fragment integrity result are recorded to form the bitstream processing result.

6. The method according to claim 5, characterized in that, In S300, when the image to be delivered is determined to be an IDR frame based on the encoding type information in the bitstream processing result, and the NAL unit required by the IDR frame is incomplete or its integrity is not confirmed, a blocking mark is formed on the IDR frame, the reference availability status of the video stream to which it belongs is set to unavailable, and a refresh request is sent to the sending end of the video stream.

7. The method according to claim 6, characterized in that, Based on the reference availability status and the corresponding GOP reference dependency relationship of the video stream, P frames referencing unavailable reference images are blocked; based on the bitstream processing result, broken B frames are blocked, and release marks are formed for complete and reference-available NAL units.

8. The method according to claim 1, characterized in that, In S400, the images successfully decoded by the VPU are written into the circular valid frame buffer in the order of decoding; when the VPU decoding fails, the most recently written valid image in the circular valid frame buffer is read as the current output image.

9. The method according to claim 4, characterized in that, When the NPU simultaneously undertakes both AI inference tasks and bitstream preprocessing tasks for abnormal candidate data packets, the AI ​​inference task is submitted first. When the NPU task is under high load, the bitstream preprocessing task is submitted later, and it remains in a waiting-for-delivery state until the corresponding NAL unit obtains the integrity result.

10. The method according to claim 6, characterized in that, Upon receiving the IDR frame corresponding to the refresh request, the integrity of the NAL units constituting the IDR frame is checked based on the bitstream processing result. When all the NAL units required for the IDR frame meet the delivery conditions, a release mark is formed for the corresponding NAL units and sent to the VPU. After the VPU successfully decodes the IDR frame, the reference availability status of the corresponding video stream is updated, and subsequent NAL units are processed according to the updated reference availability status.