A video decoding exception detection and recovery method and device, a video receiving device and a storage medium

Abstract

The application discloses a video decoding exception detection and recovery method and device, a video receiving device and a storage medium. The method obtains decoding state information of a video decoder corresponding to a target video stream, determines whether a decoding output abnormal state of a video frame input active and a video decoding output stagnation is present according to a video frame input state and a video decoding output state, saves a decoding configuration context when the abnormal state is determined, and performs recovery processing on the video decoder without reestablishing a media session to which the target video stream belongs, and sends a key frame request to a sending end after recovery is completed, so that the video decoder after recovery restores video decoding output based on the key frame. The application can realize accurate identification and local recovery of a video decoding exception, reduce influence on an audio decoding link and a control channel, and improve video output recovery efficiency.

Description

Technical Field

[0001] This application relates to the fields of video processing and video decoding health monitoring technology, and in particular to a method, apparatus, video receiving device and computer-readable storage medium for video decoding anomaly detection and recovery. Background Technology

[0002] In scenarios such as screen mirroring, real-time audio and video communication, remote display, or other video receiving, video receiving devices typically receive video streams sent by the sending end through a video data receiving interface, and use hardware video decoders or software video decoders to decode the video frames, and then output the decoded video images to the rendering target for display.

[0003] In actual operation, an abnormal state may occur in the video decoding link: video frames from the target video stream continue to arrive and be input to the video decoder, but the video decoder does not produce a valid video decoding output for a considerable period of time. This abnormal state can also be referred to as decoding output starvation, decoding output stagnation, or decoder no-output anomaly in engineering implementations.

[0004] Existing anomaly detection schemes typically determine whether a video link is abnormal based on network connection status, whether video data is continuously received, or whether new video frames are being input. However, when video frames are still being input continuously but the video decoder is experiencing an internal malfunction, it is difficult to accurately identify the aforementioned decoding output anomalies based solely on network input status or video frame input status.

[0005] On the other hand, when a video freeze is detected, some existing solutions restore video output by disconnecting and re-establishing the entire media session. Media session reconstruction typically involves multiple parts, including video links, audio links, and control channels, which may cause normally functioning audio decoding links or control channels to be interrupted simultaneously, and increase the overhead of resource initialization and protocol interaction during the recovery process.

[0006] Furthermore, when a video decoder that has malfunctioned is refreshed, reconfigured, or recreated, the new or recovered video decoder typically needs to establish the correct decoding reference relationship starting from the keyframes. If, after recovery, the decoder simply continues to receive the original video stream without actively requesting keyframes from the sender, the video decoder may need to wait for the sender to generate subsequent keyframes according to the original encoding cycle, thus extending the video decoding output recovery time.

[0007] Therefore, it is necessary to provide a technical solution that can identify abnormal states where video frame input is active but video decoding output is stagnant, and can partially restore the video decoder while keeping other links in the media session working as much as possible. Summary of the Invention

[0008] This application provides a method, apparatus, video receiving device, and storage medium for detecting and recovering video decoding anomalies. It addresses the challenges of identifying anomalies where video frames continue to input while video decoding output has stalled, based solely on network connection status or video frame input status. Furthermore, it addresses the issue that re-establishing the entire media session after a video decoding anomaly expands the fault recovery scope and may affect normally functioning audio decoding links or control channels. Finally, it addresses the technical problem that if a new keyframe is missing after video decoder recovery, the recovered video decoder may not be able to promptly restore video decoding output due to the lack of a valid decoding starting point.

[0009] To achieve the above objectives, this application adopts the following technical solution: Firstly, this application provides a method for detecting and recovering video decoding anomalies.

[0010] The method includes: Obtain the decoding status information of the video decoder corresponding to the target video stream. The decoding status information includes a first status parameter for characterizing the input state of the video frame and a second status parameter for characterizing the output state of the video decoding. Based on the first state parameter and the second state parameter, determine whether the target video stream is in an abnormal decoding output state where video frame input is active and video decoding output is stagnant; If it is determined that the target video stream is in the abnormal decoding output state, the decoding configuration context corresponding to the target video stream is saved; Without re-establishing the media session to which the target video stream belongs, recovery processing is performed on the video decoder corresponding to the target video stream according to the decoding configuration context; After the recovery process is completed, a keyframe request is sent to the sending end of the target video stream so that the recovered video decoder can recover the video decoding output based on the keyframes sent by the sending end in response to the keyframe request.

[0011] In conjunction with the first aspect described above, in one possible implementation, the first state parameter includes the most recent input frame reception time, and the second state parameter includes the most recent decoded output time.

[0012] The time difference between the current time and the time of the most recent input frame reception can be used to determine whether the video frame input remains active, and the time difference between the current time and the time of the most recent decoding output can be used to determine whether the video decoding output has stopped.

[0013] Specifically, if the difference between the current time and the most recent input frame reception time does not exceed the input active window threshold, and the difference between the current time and the most recent decoding output time is not lower than the output stagnation threshold, the basic judgment conditions for decoding output abnormality can be determined.

[0014] In conjunction with the first aspect above, in one possible implementation, the decoding status information further includes a continuous output unavailability count, a continuous input unavailability count, and the number of input queue frames.

[0015] Based on meeting the aforementioned basic judgment conditions, further judgments can be made: Is the count of consecutive unavailable outputs not lower than the threshold for the number of unavailable outputs? Does the consecutive input failure count not exceed the maximum number of input failures? Check if the number of frames in the input queue does not exceed the low watermark of the queue.

[0016] When the above basic judgment conditions, as well as the judgment conditions corresponding to the continuous output unavailability count, the continuous input unavailability count, and the input queue frame number are all satisfied, it can be determined that the target video stream is in a decoding output abnormal state.

[0017] Therefore, the operating status of the video decoder can be judged by comprehensively considering the video frame input time, video decoding output time, input buffer status, output buffer status, and input queue status, thereby reducing the probability of identifying other abnormal errors such as network input interruption and input congestion as abnormal video decoding output.

[0018] In conjunction with the first aspect above, in one possible implementation, at least one of the input active window threshold, output stagnation threshold, output unavailable times threshold, and queue low water level can be configured according to at least one of the frame rate of the target video stream, the video codec format, and the chip platform on which the video decoder resides.

[0019] For example, the input active window threshold can be set to 800 milliseconds, the output stagnation threshold can be set to 2000 milliseconds, the output unavailable number threshold can be set to 60 times, the maximum number of consecutive input unavailable times can be set to 1 time, and the queue low water level can be set to 2 frames, or set to a frame number threshold that does not exceed 25% of the input queue capacity, determined based on the input queue capacity.

[0020] The specific values ​​mentioned above are only used to illustrate one feasible implementation method and do not constitute a limitation on the scope of protection of this application.

[0021] In conjunction with the first aspect above, in one possible implementation, the decoding state information further includes rendering state parameters for characterizing the video rendering state.

[0022] When the second state parameter determines that the video decoder has continuous decoding output, and the rendering state parameter determines that the video rendering output has not been updated for more than the preset stagnation time, it can be determined that the target video stream is in an abnormal rendering state.

[0023] In this scenario, the rendering target corresponding to the target video stream can be rebuilt first. If the video rendering output is still not restored after rebuilding the rendering target, then the video decoder corresponding to the target video stream can be restored.

[0024] By distinguishing between video decoding output anomalies and video rendering anomalies, different recovery measures can be adopted according to the scope of the link where the anomaly occurred, thereby avoiding the unnecessary re-creation of the video decoder when it is still working normally.

[0025] In conjunction with the first aspect above, in one possible implementation, the decoding configuration context includes a media type identifier corresponding to the video codec format, a codec parameter set, a media format field, a rendering target reference, and a video stream channel identifier.

[0026] To ensure that the codec parameter set can still be effectively used after the original video decoder is released, the buffer content corresponding to the codec parameter set in the media format field can be copied, so that the copied buffer content is independent of the lifecycle of the original video decoder.

[0027] For HEVC format video streams, the codec parameter set may include, for example, VPS, SPS, and PPS; for AVC format video streams, the codec parameter set may include, for example, SPS and PPS. The specific parameter set type can be determined according to the video codec format.

[0028] In conjunction with the first aspect described above, in one possible implementation, the recovery process performed on the video decoder may include at least one of the following: Perform a refresh operation on the video decoder; Reconfigure the video decoder based on the decoding configuration context; Release the current video decoder and create and start another video decoder based on the decoding configuration context.

[0029] Recovery can be performed in a progressively more intensive manner. For example, you can first perform a refresh operation on the video decoder; if the video decoding output is still not restored after the refresh, then release the current video decoder and create and start another video decoder.

[0030] During the recovery process described above, it is not necessary to re-establish the media session to which the target video stream belongs, thus enabling at least one of the audio decoding links and control channels in the media session to continue operating.

[0031] In conjunction with the first aspect above, in one possible implementation, a keyframe request can be sent immediately after the video decoder completes the recovery process.

[0032] If no keyframe is received within the preset waiting time after sending the keyframe request, the keyframe request can be resent until a keyframe is received or the preset number of resentments is reached.

[0033] After the video decoder recovery process is completed and before the key frame is received, the received non-key frames can be prevented from being input into the recovered video decoder for normal decoding.

[0034] The blocking may include discarding non-key frames, caching non-key frames but not sending them into the normal decoding process, or preventing non-key frames from entering the recovered video decoder by inputting a gating state.

[0035] Keyframe requests can be sent via feedback control messages from the real-time transmission control protocol, or via private control messages or the keyframe request interface provided by the sending end.

[0036] In conjunction with the first aspect above, in one possible implementation, video stream channel identifiers are maintained for different video streams, and the corresponding soft recovery count and last recovery timestamp are maintained using the video stream channel identifiers as indexes.

[0037] A video stream channel identifier can be an identifier that can uniquely distinguish different video streams, or at least within the scope of a media session. For example, in implementations that support real-time transport protocols, it can be determined based on a synchronization source identifier, while in other implementations it can be generated based on a video track identifier or from at least some of the information including a session identifier, stream identifier, media type, and codec type.

[0038] Within the preset cooldown time window, if the number of soft recovery attempts corresponding to the same video stream channel identifier exceeds the preset limit, the automatic recovery process for that video stream channel identifier can be stopped to avoid continuous and repeated reconstruction of the video decoder under abnormal conditions.

[0039] In the event of recovery failure, a diagnostic event can be reported. The diagnostic event may include at least one of the following: video stream channel identifier, video decoder identifier, chip platform, status snapshot of decoding status information, and recovery time.

[0040] In addition, a video transmission parameter downgrade request can be sent to the sending end.

[0041] Degrading video transmission parameters can include at least one of the following: reducing video resolution, reducing video frame rate, reducing encoding complexity, or switching video codec formats.

[0042] If the video transmission parameter downgrade negotiation fails, or if the target video stream is found to be in an abnormal decoding output state after the video transmission parameter downgrade is completed, the media session to which the target video stream belongs can be re-established.

[0043] Secondly, this application also provides a video decoding anomaly detection and recovery device, including a status acquisition module, an anomaly recognition module, a context management module, a recovery control module, and a keyframe request module.

[0044] The status acquisition module is used to obtain the decoding status information of the video decoder corresponding to the target video stream.

[0045] The anomaly identification module is used to determine whether the target video stream is in an abnormal decoding output state where video frame input is active and video decoding output is stagnant, based on the first state parameter and the second state parameter.

[0046] The context management module is used to save the corresponding decoding configuration context when it is determined that the target video stream is in an abnormal decoding output state.

[0047] The recovery control module is used to perform recovery processing on the video decoder based on the decoding configuration context without re-establishing the media session to which the target video stream belongs.

[0048] The keyframe request module is used to send a keyframe request to the sender after the recovery process is completed, so that the recovered video decoder can recover the video decoding output based on the keyframes sent by the sender in response to the keyframe request.

[0049] Thirdly, this application also provides a video receiving device, including a video data receiving interface, a video decoder, a memory, and a processor.

[0050] The video data receiving interface is used to receive the target video stream, the video decoder is used to decode the video frames of the target video stream, the memory stores the computer program, and the processor is connected to the video data receiving interface and the video decoder.

[0051] When the processor executes the computer program, it causes the video receiving device to execute the video decoding anomaly detection and recovery method provided in the embodiments of this application.

[0052] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor performs the video decoding anomaly detection and recovery method provided in the embodiments of this application.

[0053] Beneficial effects: 1. The application utilizes a combination of first and second state parameters to determine the input and output states of video frames, enabling the identification of abnormal states where video frame input remains active while video decoding output has stalled. Further combining continuous output unavailability counts, continuous input unavailability counts, and the number of frames in the input queue allows for confirmation of the video decoder's operational status from multiple dimensions, reducing misjudgments caused by network interruptions, input buffer anomalies, or data backlog.

[0054] 2. After determining that the decoding output is abnormal, by saving the decoding configuration context and performing recovery processing such as refreshing, reconfiguring or recreating the video decoder corresponding to the target video stream, the recovery scope can be limited to the video decoding link without immediately re-establishing the entire media session, so that the audio decoding link or control channel in the media session can continue to work.

[0055] 3. By actively sending a keyframe request to the sending end after the recovery process is completed, and restricting non-keyframes from entering the normal decoding process before the keyframe arrives, the recovered video decoder can obtain an effective decoding reference starting point more quickly, reducing the recovery delay caused by waiting for natural keyframes.

[0056] 4. By maintaining soft recovery counts and last recovery timestamps for different video streams, and setting cooldown time windows and recovery count limits, the possibility of repeated recovery triggered by the same fault can be reduced. After a soft recovery failure, a step-by-step upgrade mechanism is formed by reporting diagnostic events, downgrading video transmission parameters, and rebuilding media sessions, which helps improve system stability in abnormal scenarios. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the overall architecture of the video decoding anomaly detection and recovery system provided in the embodiments of this application.

[0058] Figure 2 This is a schematic diagram of the decoding output abnormal combination judgment logic provided in the embodiments of this application.

[0059] Figure 3 This is a schematic diagram of the video decoder session-free reconstruction and recovery process provided in the embodiments of this application.

[0060] Figure 4 This is a schematic diagram of the soft recovery limitation and fault escalation processing logic provided in the embodiments of this application.

[0061] Reference numerals: 101—Sender; 102—Media Session; 103—Video Decoder; 104—Decoding Status Information; 105—Anomaly Detection and Recovery Control; 106—Rendering Target; 201—Decoding Status Parameters; 202—Basic Time Judgment Conditions; 203—Counting and Queue Judgment; 204—Combination Judgment; 205—Decoding Output Abnormal Status; 206—Continue Status Monitoring; 301—Save Decoding Configuration Context; 302—Refresh and / or Reconfigure Video Decoder; 303—Decoding Output Recovery Judgment; 304—Release Video Decoder and Create and Start Another Video Decoder; 305—Keyframe Request and Waiting Control; 306—Recover Video Decoding Output Based on Keyframes; 401—Video Stream Recovery Record; 402—Automatic Execution of Soft Recovery; 403—Diagnostic Event Reporting and Video Transmission Parameter Degradation; 404—Degradation Failure or Re-abnormal Judgment After Degradation; 405—Re-establish Media Session; 406—Media Session Link Maintenance. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0063] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0064] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0065] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0066] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0067] The video decoding anomaly detection and recovery method provided in this application embodiment can be executed by a video decoding anomaly detection and recovery device. The video decoding anomaly detection and recovery device can be implemented by software and / or hardware, and can consist of two or more physical entities, or a single physical entity. The video decoding anomaly detection and recovery device can be installed in a screen projection sending end, a screen projection receiving end, or other electronic devices communicatively connected to the screen projection sending end. The electronic devices can be mobile phones, tablets, laptops, desktop computers, interactive whiteboards, conference terminals, screen projectors, or other intelligent devices with data processing and communication functions.

[0068] For ease of understanding, this application uses a screen projector as an application carrier for a video decoding anomaly detection and recovery device in its embodiments. The screen projector can be an electronic device used to establish a communication connection between the screen projector and the display device, and to transmit image data, audio data, video data, control data, or other data to be projected from the screen projector to the display device. The screen projector can be set up independently or integrated into a monitor, television, interactive whiteboard, conference terminal, projection device, or other display device.

[0069] A screen mirroring device can function as a screen mirroring sender, transmitting locally generated or acquired data from other devices to a screen mirroring receiver. It can also function as a screen mirroring receiver, receiving screen mirroring data from mobile phones, tablets, laptops, or other screen mirroring senders, and outputting that data to a display device connected to the screen mirroring device. Furthermore, a screen mirroring device can simultaneously perform both screen mirroring sending and receiving functions.

[0070] Generally, a screen mirroring device may include at least one of a processor, memory, communication module, and display interface. The communication module may include a wireless communication module and / or a wired communication module. The wireless communication module may support wireless LAN, Bluetooth, ultra-wideband, near-field communication, mobile communication, acoustic communication, or other wireless communication methods; the wired communication module may support Ethernet, Universal Serial Bus, or other wired communication methods. The display interface may include a high-definition multimedia interface, a DisplayPort interface, a Universal Serial Bus interface, a video graphics array interface, or other interfaces capable of transmitting image and / or audio data.

[0071] The screen mirroring device can access a local area network (LAN) via Wi-Fi and search for available screen mirroring receivers within the LAN using a network service discovery mechanism. The device can receive device discovery messages broadcast by one or more screen mirroring receivers within the LAN and, based on these messages, obtain at least one of the following: device name, device identifier, network address, service type, port number, connection status, and device capabilities, thereby generating or updating the list of available screen mirroring devices.

[0072] The screen mirroring device can also be equipped with a microphone, a sound wave receiving module, or other sound acquisition modules to collect sound wave signals sent by the screen mirroring receiver. The screen mirroring device can demodulate and analyze the collected sound wave signals to obtain sound wave candidate information. The sound wave candidate information may include at least one of the following: screen mirroring code, network address, device identifier, device name, timestamp, verification information, and sound wave signal strength. The screen mirroring device can associate the sound wave candidate information with the corresponding device in the list of eligible screen mirroring devices based on the matching relationship between the sound wave candidate information and the device information obtained through a network service discovery mechanism.

[0073] The screen mirroring device can establish an association between candidate acoustic wave information and screen-sharing devices based on a device identifier, network address, screen mirroring code, device name, or a combination of at least two of the above information. This association can be stored independently of the list row objects in the screen-sharing device list, so that even when the screen-sharing device list is refreshed, the list row objects are rebuilt, or the device display order changes, the candidate acoustic wave information can still be re-associated to the corresponding screen-sharing device based on the combined identifier.

[0074] Typically, the screen mirroring device can install at least one operating system, including but not limited to Android, Linux, Windows, HarmonyOS, or an embedded real-time operating system. The screen mirroring device can run at least one screen mirroring application based on the operating system. The screen mirroring application can be an application that comes pre-installed with the operating system, or an application downloaded and installed from a server or other electronic device. The video decoding anomaly detection and recovery device can be a standalone application or implemented as a functional module within the screen mirroring application.

[0075] The screen mirroring application can perform at least one of the following operations: device discovery, device list generation, acoustic signal parsing, candidate information caching, device matching, connection request sending, screen mirroring code verification, screen mirroring link establishment, and screen mirroring status update. For example, a user can select a target screen mirroring device from the list of available devices. The screen mirroring device obtains the corresponding network address and / or screen mirroring code based on the selected target device, sends a connection request to the target device, and establishes a screen mirroring communication link after successful connection verification.

[0076] Understandably, users can select a target projection device or initiate a projection operation via touchscreen, keyboard, mouse, remote control, physical buttons, voice commands, or other input methods. After receiving the user's operation, the projection device can determine the target projection device and its corresponding connection parameters based on the user's selected device information, acoustic wave candidate information, and the current network connection status, and then execute the projection connection operation.

[0077] The projection device may include one or more processing cores. The processing core can implement the video decoding anomaly detection and recovery method provided in this application embodiment using pure software, or it can be implemented using a combination of software and hardware. For example, it can be implemented using at least one hardware form selected from digital signal processors, field-programmable gate arrays, application-specific integrated circuits, and programmable logic arrays. The projection device may also integrate one or more of a central processing unit, graphics processor, network processor, audio processor, and modem.

[0078] The memory of the screen projector can be used to store at least one of the following: computer program, list of projectable devices, device discovery information, acoustic wave candidate information, device association relationships, screen projecting connection parameters, and screen projecting status information. The processor can call and execute the computer program stored in the memory to enable the screen projector to perform the video decoding anomaly detection and recovery method provided in the embodiments of this application.

[0079] The computer program can be presented in a form suitable for the operation of the screen projector, such as an application, system service, background process, system plugin, software development kit, firmware program, or web page plugin. In some embodiments, the video decoding anomaly detection and recovery method can also be executed collaboratively by multiple software modules in the screen projector, which may include at least one of a device discovery module, a sound wave analysis module, a candidate information management module, a device matching module, a list update module, and a connection control module.

[0080] Example 1: Overall System and Basic Recovery Method like Figure 1 As shown, in one implementation, the video receiving device establishes a media session with the sending end.

[0081] The sending end 101 sends the target video stream to the video receiving device through media session 102. The video decoder 103 in the video receiving device receives the video frames of the target video stream and performs video decoding, providing the video decoded output to the rendering target 106.

[0082] The video receiving device also maintains decoding status information 104 corresponding to the video decoder 103, and detects video decoding anomalies based on the decoding status information 104 through the anomaly detection and recovery control section 105.

[0083] In addition to carrying the target video stream, media session 102 can also carry audio data or control messages. Therefore, when the video decoder 103 of the target video stream malfunctions, this embodiment prioritizes partial recovery of the video decoder 103 instead of re-establishing media session 102, ensuring that at least one of the audio decoding link and control channel in the media session can remain operational.

[0084] When it is determined that the target video stream is in an abnormal decoding output state, the anomaly detection and recovery control section 105 first saves the decoding configuration context corresponding to the target video stream, and then performs recovery processing on the video decoder 103.

[0085] After the recovery process is completed, the anomaly detection and recovery control unit 105 sends a keyframe request to the sending end 101. The sending end 101 responds to the keyframe request by sending a keyframe, and the recovered video decoder 103 recovers the video decoding output based on the keyframe and continues to provide the video decoding result to the rendering target 106.

[0086] Therefore, partial recovery of the video decoding link that has malfunctioned can be achieved without immediately re-establishing the entire media session.

[0087] Example 2 like Figure 2 As shown in the figure, this embodiment further illustrates the method for determining abnormal states of the decoding output.

[0088] The video receiving device acquires the decoding status parameter 201 corresponding to the target video stream.

[0089] The decoding status parameter 201 includes at least the most recent input frame reception time and the most recent decoding output time, and may also include a continuous output unavailability count, a continuous input unavailability count, and the number of input queue frames.

[0090] First, execute the basic time determination 202.

[0091] Let the current time be T, the most recent input frame reception time be Tin, the most recent decoded output time be Tout, the input active window threshold be Ta, and the output stagnation threshold be Ts.

[0092] when: T-Tin≤Ta, and: T-Tout≥Ts, It can be determined that the target video stream meets the basic criteria of "active video frame input and stagnant video decoding output".

[0093] Further execution of counting and queue determination 203.

[0094] Let Cout be the count of consecutive unavailable outputs, and Cth be the threshold for the number of unavailable outputs; let Cin be the count of consecutive unavailable inputs, and Cmax be the upper limit for the number of unavailable inputs; let Q be the number of frames in the input queue, and Qlow be the low watermark of the queue.

[0095] When both conditions are met: Cout≥Cth; Cin≤Cmax; and Q≤Qlow, Then the counting and queue determination conditions are met.

[0096] In combination determination 204, if the conditions corresponding to basic time determination 202 and counting and queue determination 203 are both met, then the decoding output abnormal state 205 is entered.

[0097] If at least one set of judgment conditions is not met, the system will enter the continued state monitoring 206 phase without triggering the video decoder recovery.

[0098] In one specific implementation, the input active window threshold Ta can be 800 milliseconds, the output stagnation threshold Ts can be 2000 milliseconds, the output unavailable number threshold Cth can be 60 times, the maximum number of consecutive input unavailable times Cmax can be 1 time, and the queue low water level Qlow can be 2 frames.

[0099] In another implementation, the queue low water level (Qlow) can be determined based on the input queue capacity of the video decoder, for example, set to a frame number threshold not exceeding 25% of the input queue capacity.

[0100] The above parameters can be adjusted based on at least one of the following: the frame rate of the target video stream, the video codec format, and the chip platform on which the video decoder is located.

[0101] For example, different chip platforms may have different scheduling methods for input buffers, output buffers, and asynchronous callbacks in their video decoders. Therefore, different output stall thresholds or output unavailability thresholds can be configured for different chip platforms.

[0102] It should be noted that a low input queue frame count can help eliminate other abnormal states caused by a large amount of data backlog on the decoder input side; a continuous input unavailability count not exceeding the upper limit can further indicate that the video input side is not continuously unavailable.

[0103] Therefore, by using multiple parameters to make a joint judgment, the accuracy of identifying the specific abnormal state of "the input is still active but the decoding output is stagnant" can be improved.

[0104] In one implementation, the video receiving device maintains an independent decoding status record for each target video stream.

[0105] When a video frame from the target video stream is received and successfully sent to the input path of the video decoder, the most recent input frame reception time can be updated.

[0106] When the acquisition of the input buffer fails or the input buffer becomes unavailable within a predetermined time, the count of consecutive unavailable inputs can be incremented; when the input buffer is successfully acquired subsequently, the count of consecutive unavailable inputs can be cleared or decremented according to a predetermined rule.

[0107] When the video decoding output fails to be obtained from the video decoder or when the output is continuously unavailable, the continuous output unavailable count can be increased.

[0108] When the video decoder produces a valid video decoding output, it updates the time of the most recent decoding output and can either clear the count of unavailable continuous outputs to zero or reduce it according to a predetermined rule.

[0109] The number of input queue frames can be dynamically maintained based on the number of video frames sent to the video decoder and the number of video frames that have been processed.

[0110] It should be noted that the most recent decoding output time can be determined based on the specific video decoding framework. For example, it can be updated when a valid output buffer is acquired, a valid output buffer is released, a valid decoded frame is acquired, or a callback that can represent the video decoding output is received, as long as it can reflect the video decoding output status of the video decoder.

[0111] In one implementation, the decoding state information also includes rendering state parameters for characterizing the video rendering state.

[0112] Rendering status parameters may include, for example, the time since the last render update, the time since the last displayed frame, or the time since the render target callback.

[0113] When the second state parameter indicates that the video decoder continuously outputs valid video decoding, and the rendering state parameter indicates that the video rendering output has not been updated for more than the preset pause time, it can be determined that the target video stream is in an abnormal rendering state, rather than an abnormal decoding output state.

[0114] In case of rendering anomalies, the rendering target corresponding to the target video stream can be rebuilt first.

[0115] The rendering target may include a Surface, a SurfaceTexture, or other display targets capable of receiving video decoding output.

[0116] Once the rendering target is reconstructed, you can continue to monitor the video rendering output.

[0117] If the video rendering output is restored, there is no need to recreate the video decoder.

[0118] If the video rendering output still cannot be restored after rebuilding the rendering target, further recovery processing can be performed on the video decoder corresponding to the target video stream.

[0119] By using the above-mentioned grading method, the recovery range can be selected based on the actual state of the video decoding link and the video rendering link.

[0120] When a target video stream is determined to be in an abnormal decoding output state, the decoding configuration context corresponding to the target video stream is saved before performing a recovery action that may cause the original video decoder's internal state or resources to be released.

[0121] The decoding configuration context may include: Media type identifier corresponding to video codec format; Codec parameter set; Media format field; Render target reference; and Video stream channel identifier.

[0122] The media format field can include video width, video height, frame rate, color format, and other media format information required for video decoder configuration.

[0123] For codec parameter sets saved via buffer references in the media format field, the corresponding buffer contents can be copied before releasing the original video decoder.

[0124] For example, when the original media format field stores a reference to a parameter buffer, and the lifecycle of that parameter buffer is related to the original video decoder, the parameter data can be copied to a separate buffer, and the copied buffer can be recorded in the decoding configuration context.

[0125] Therefore, even after the original video decoder is released, the recovery control can still reconfigure the video decoder or create another video decoder based on the independently saved parameter set.

[0126] Example 3 like Figure 3 As shown, once it is determined that the target video stream is in an abnormal decoding output state, the following recovery process can be executed.

[0127] First, save the decoding configuration context in 301.

[0128] Subsequently, a refresh and / or reconfiguration is performed on the video decoder in step 302.

[0129] In video decoder implementations that support refresh operations, a refresh operation can be performed first to clear the temporary state inside the video decoder and continue using the current video decoder instance.

[0130] In other implementations, the current video decoder can be reconfigured based on the decoding configuration context.

[0131] In step 303, determine whether the video decoding output has been restored.

[0132] If the video decoding output has been restored, there is no need to release the current video decoder; the keyframe request and synchronization recovery process can proceed.

[0133] If the video decoding output is not restored after refreshing or reconfiguring, the current video decoder is released in 304, and another video decoder is created and started based on the previously saved decoding configuration context.

[0134] When creating another video decoder, at least one of the following can be reused: media type identifier, codec parameter set, media format field, render target reference, and video stream channel identifier.

[0135] During the recovery process described above, the media session to which the target video stream belongs remains active, without the need to re-execute the entire media session establishment process.

[0136] Therefore, at least one of the audio decoding link and control channel corresponding to the media session can continue to work.

[0137] After the video decoder has finished refreshing, reconfiguring, or recreating and starting, a keyframe request is sent to the sender in 305.

[0138] Keyframe requests can be sent via RTCP feedback control messages, or via private control messages in the media session, or via the keyframe request interface provided by the sender.

[0139] The video receiving device can send a keyframe request once the recovery process is complete.

[0140] If no keyframe is received from the sender within the preset waiting time, the keyframe request will be sent again.

[0141] For example, in one implementation, the keyframe request can be retransmitted every 500 milliseconds, with a preset maximum number of retransmissions. The above times and numbers are merely examples and do not constitute a limitation.

[0142] To prevent the restored video decoder from starting decoding directly from the predicted frame that depends on the reference frame before restoration, a keyframe wait state can be set before the video decoder receives the keyframe after restoration is complete.

[0143] When waiting for keyframes, received non-keyframes are not decoded normally by the video decoder after being input to the recovered video.

[0144] Once a keyframe is detected, it is input into the recovered video decoder, and the keyframe waiting state is lifted.

[0145] Subsequently, in step 306, the restored video decoder establishes a new decoding reference relationship based on this keyframe and restores the video decoding output.

[0146] In multi-video stream or multi-media scenarios, video stream channel identifiers are maintained separately for different target video streams.

[0147] For each video stream channel identifier, maintain the corresponding soft recovery count and the last recovery timestamp.

[0148] After performing an automatic recovery process for a specific video stream channel identifier, the soft recovery count for that channel can be incremented, and the recovery time can be recorded.

[0149] If a decoding output anomaly occurs again, the system will determine whether automatic recovery processing is still allowed based on the current time, the last recovery timestamp, and the soft recovery count.

[0150] For example, if the number of soft recovery attempts corresponding to the same video stream channel identifier exceeds the preset limit within a preset cooling time window, automatic recovery processing for that channel can be stopped.

[0151] In one specific implementation, the cooldown time window can be 60 seconds, and the maximum number of soft recoveries can be 2. These specific values ​​are merely examples.

[0152] By managing the recovery status according to the video stream channel identifier, it is possible to prevent abnormal recovery behavior of one video stream from erroneously affecting other video streams.

[0153] Example 4 like Figure 4 As shown, in 401, the video receiving device maintains the soft recovery count and the last recovery timestamp using the video stream channel identifier as an index, and controls whether automatic recovery is allowed according to a preset cooling time window.

[0154] If automatic recovery is enabled, perform a soft recovery for the video decoder that malfunctioned in 402.

[0155] During a soft recovery, the media session link shown in 406 remains active, and at least one of the audio decoding link and the control channel can continue to function.

[0156] If soft recovery fails, or if the number of soft recovery attempts reaches the preset limit within the cooldown window, a diagnostic event can be reported in 403, and a video transmission parameter downgrade request can be sent to the sending end.

[0157] Diagnostic events may include at least one of the following: video stream channel identifier, video decoder identifier, chip platform, status snapshot of decoding status information, and recovery time.

[0158] Video transmission parameter downgrade requests can be used to request the sender to reduce resolution, frame rate, encoding complexity, or switch video codec formats.

[0159] In a 404 error, determine whether the video transmission parameter downgrade failed, or whether a decoding output error occurred again after the video transmission parameter downgrade.

[0160] If the video transmission parameter downgrade negotiation fails, or if a decoding output error occurs again after the downgrade is completed, the media session to which the target video stream belongs will be re-established in 405.

[0161] Therefore, this embodiment forms a fault handling hierarchy from local recovery to transmission parameter degradation, and then to media session reconstruction, making media session reconstruction, which has a large impact, a subsequent upgrade measure, rather than the default recovery method when decoding output abnormality is first detected.

[0162] Example 5 This embodiment provides a video decoding anomaly detection and recovery device.

[0163] The device includes a status acquisition module, an anomaly recognition module, a context management module, a recovery control module, and a keyframe request module.

[0164] The status acquisition module is used to obtain the decoding status information of the video decoder corresponding to the target video stream.

[0165] The anomaly identification module is used to determine whether the target video stream is in an abnormal decoding output state where video frame input is active and video decoding output is stagnant, based on the first state parameter and the second state parameter.

[0166] The anomaly detection module can further confirm decoding output anomalies based on the continuous output unavailability count, continuous input unavailability count, and input queue frame count.

[0167] The context management module is used to save the decoding configuration context corresponding to the target video stream after determining that the decoding output is in an abnormal state.

[0168] The recovery control module is used to perform at least one of the following recovery processes on the video decoder, based on the decoding configuration context: refresh, reconfigure, or release and create another video decoder, without re-establishing the media session to which the target video stream belongs.

[0169] The keyframe request module is used to send a keyframe request to the sender after the recovery process is completed, and can control non-keyframes to not enter the recovered video decoder for normal decoding before the keyframe arrives.

[0170] In some embodiments, the device may further include a recovery count control function, a diagnostic event reporting function, and a video transmission parameter degradation control function.

[0171] It should be noted that the above modules can be divided according to functional logic, or they can be implemented by the same processor executing different program instructions. The module names and module division methods do not constitute a limitation on the actual hardware structure.

[0172] Example 6: Video Receiving Device This embodiment provides a video receiving device.

[0173] The video receiving device includes a video data receiving interface, a video decoder, a memory, and a processor.

[0174] The video data receiving interface is used to receive the target video stream sent by the sending end.

[0175] A video decoder is used to decode video frames in a target video stream.

[0176] The memory is used to store computer programs and state information, decoding configuration context, or recovery control information used during the execution of video decoding anomaly detection and recovery.

[0177] The processor connects to the video data receiving interface and the video decoder, and is used to execute computer programs stored in memory.

[0178] When the processor executes the computer program, it causes the video receiving device to execute the video decoding anomaly detection and recovery method in any of the above embodiments.

[0179] The video receiving device may be a screen mirroring receiving terminal, a smart TV, a set-top box, a display device, a conference terminal, a vehicle-mounted display terminal, a mobile terminal, a computing device, or other electronic devices capable of receiving and decoding video streams.

[0180] Example 7 This application also provides a computer-readable storage medium storing a computer program thereon, the instructions of which are adapted to be loaded by a processor and executed by the method steps of Embodiment 1 shown above. The specific execution process can be found in the detailed description of the embodiments, and will not be repeated here. The device containing the storage medium can be an electronic device such as a personal computer, laptop computer, smartphone, or tablet computer.

[0181] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0182] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0183] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function selected in one or more boxes.

[0184] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function selected in one or more boxes.

[0185] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0186] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0187] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0188] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0189] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for video decoding exception detection and recovery, the method comprising: The method comprises: obtaining decoding state information of a video decoder corresponding to a target video stream, the decoding state information comprising a first state parameter for representing a video frame input state and a second state parameter for representing a video decoding output state; determining whether the target video stream is in a decoding output abnormal state of video frame input active and video decoding output stagnation according to the first state parameter and the second state parameter; in a case where it is determined that the target video stream is in the decoding output abnormal state, saving a decoding configuration context corresponding to the target video stream; without re-establishing a media session to which the target video stream belongs, performing a recovery process on the video decoder corresponding to the target video stream according to the decoding configuration context; after the recovery process is completed, sending a key frame request to a sending end of the target video stream, so that the video decoder after recovery restores video decoding output based on a key frame sent by the sending end in response to the key frame request.

2. The method of claim 1, wherein: the first state parameter comprises a latest input frame receiving time, and the second state parameter comprises a latest decoding output time; the determination of whether the target video stream is in the decoding output abnormal state comprises: in a case where a difference between a current time and the latest input frame receiving time does not exceed an input active window threshold, and a difference between the current time and the latest decoding output time is not less than an output stagnation threshold, determining that a basic judgment condition of decoding output abnormality is met.

3. The method of claim 2, wherein: the decoding state information further comprises a continuous output unavailable count, a continuous input unavailable count, and an input queue frame number; the determination of whether the target video stream is in the decoding output abnormal state further comprises: determining that the continuous output unavailable count is not less than an output unavailable times threshold, the continuous input unavailable count does not exceed an upper limit of input unavailable times, and the input queue frame number does not exceed a queue low water level; wherein, in a case where the basic judgment condition and judgment conditions corresponding to the continuous output unavailable count, the continuous input unavailable count, and the input queue frame number are all met, it is determined that the target video stream is in the decoding output abnormal state.

4. The method of claim 3, wherein: at least one of the input active window threshold, the output stagnation threshold, the output unavailable times threshold, and the queue low water level is configured according to at least one of a frame rate of the target video stream, a video codec format, and a chip platform in which the video decoder is located; wherein, the input active window threshold is 800 milliseconds, the output stagnation threshold is 2000 milliseconds, the output unavailable times threshold is 60 times, the upper limit of continuous input unavailable times is 1 time, and the queue low water level is 2 frames or a frame number threshold not exceeding 25% of an input queue capacity.

5. The method of any one of claims 1 to 4, wherein: the decoding state information further comprises a rendering state parameter for representing a video rendering state. The method further comprises: in the case that the video decoder is determined to have continuous decoding output according to the second state parameter, and the video rendering output is determined not to be updated for more than a preset inactivity duration according to the rendering state parameter, determining that the target video stream is in a rendering abnormal state; in response to the rendering abnormal state, rebuilding a rendering target corresponding to the target video stream; in the case that the video rendering output is still not recovered after rebuilding the rendering target, performing the recovery processing on the video decoder corresponding to the target video stream.

6. The method of any one of claims 1-5, wherein the decoding configuration context comprises a media type identifier corresponding to a video codec format, a codec parameter set, a media format field, a rendering target reference, and a video stream channel identifier; the saving of the decoding configuration context comprises: copying buffer content corresponding to the codec parameter set in the media format field, so that the copied buffer content remains valid after the video decoder is released.

7. The method of any one of claims 1-6, wherein the performing of the recovery processing on the video decoder corresponding to the target video stream comprises at least one of: performing a flushing operation on the video decoder; reconfiguring the video decoder according to the decoding configuration context; releasing the video decoder and creating and starting another video decoder according to the decoding configuration context; wherein in the case that the video decoding output is still not recovered after performing the flushing operation, the video decoder is released and the another video decoder is created and started.

8. The method of any one of claims 1-7, wherein the performing of the recovery processing on the video decoder is performed without reestablishing a media session to which the target video stream belongs, and at least one of an audio decoding link and a control channel corresponding to the media session is kept working.

9. The method of any one of claims 1-8, wherein the key frame request is sent once after the recovery processing is completed; in the case that a key frame is not received within a preset waiting time after the sending of the key frame request, the key frame request is resent until a key frame is received or the number of resending reaches a preset number; and, before the key frame is received after the completion of the recovery processing, received non-key frames are prevented from being input to the recovered video decoder for normal decoding; the key frame request is sent through a feedback control message of a real-time transport control protocol, a private control message, or a key frame request interface provided by a sending end.

10. The method of any one of claims 1-9, wherein a video stream channel identifier is maintained for different video streams respectively, and a soft recovery count and a last recovery timestamp corresponding to the video stream channel identifier are maintained as an index; in the case that a soft recovery count corresponding to a same video stream channel identifier exceeds a preset upper limit within a preset cooling time window, automatic execution of the recovery processing for the same video stream channel identifier is stopped. In the case where the recovery processing fails, a diagnostic event including at least one of a video stream channel identifier, a video decoder identifier, a chip platform, a state snapshot of a decoding state information, and a recovery time consumption is reported, and a video transmission parameter degradation request is sent to the sending end; In the case where the target video stream is determined to be in the decoding output abnormal state again after the video transmission parameter degradation negotiation fails or the video transmission parameter degradation is completed, the media session to which the target video stream belongs is re-established.

11. A video decoding anomaly detection and recovery device, characterized in that, Comprise: a state acquisition module, configured to acquire decoding state information of a video decoder corresponding to a target video stream, the decoding state information comprising a first state parameter for representing a video frame input state and a second state parameter for representing a video decoding output state; an abnormality identification module, configured to determine whether the target video stream is in a decoding output abnormal state in which a video frame input is active and a video decoding output is stagnant according to the first state parameter and the second state parameter; a context management module, configured to save a decoding configuration context corresponding to the target video stream in the case where the target video stream is determined to be in the decoding output abnormal state; a recovery control module, configured to perform recovery processing on the video decoder corresponding to the target video stream according to the decoding configuration context without re-establishing a media session to which the target video stream belongs; a key frame request module, configured to send a key frame request to a sending end of the target video stream after the recovery processing is completed, so that the video decoding output is recovered based on a key frame sent by the sending end in response to the key frame request.

12. A video receiving apparatus characterized by comprising: comprise a video data receiving interface, a video decoder, a memory and a processor; the video data receiving interface is configured to receive a target video stream; the video decoder is configured to decode a video frame of the target video stream; the memory has stored therein a computer program; the processor is connected with the video data receiving interface and the video decoder, and is configured to execute the computer program, so that the video receiving device executes the video decoding abnormality detection and recovery method in any one of claims 1 to 10.

13. A computer readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, causes the processor to execute the video decoding abnormality detection and recovery method in any one of claims 1 to 10.

13. A computer readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, causes the processor to execute the video decoding abnormality detection and recovery method in any one of claims 1 to 10.

Images