A method and related device for processing a remote video loss

Abstract

The application provides a remote video loss processing method and related device, and relates to the technical field of software, and comprises the following steps: determining whether a frame loss state exists according to a network state and a data check result of a remote video frame; in the case of determining that the remote video frame is lost, obtaining a target placeholder picture, the size of the target placeholder picture being matched with a remote video window; writing the target placeholder picture into the remote video window, and performing pixel fusion on the target placeholder picture and a local video picture under the remote video window. According to the application, whether the remote video frame is lost is detected comprehensively through the network state and the data check result, so that the misjudgment rate and the omission rate are reduced; on the other hand, pixel fusion is performed on the target placeholder picture and the local video picture under the remote video window, so that the sense of picture fragmentation and the sense of mutation are eliminated, the user does not need to adapt to the layout again, and the stability of the canvas layout is improved.

Description

Technical Field

[0001] This application relates to the field of software technology, and in particular to a method and related apparatus for processing remote video loss. Background Technology

[0002] With the widespread application of audio and video call technology in social, office, and medical fields, users are increasingly demanding higher standards for video stability and seamless experience during calls.

[0003] During audio and video calls, if the remote video is missing, the local remote video window will show a black screen or the screen will freeze. In addition, some solutions only display simple prompt text, lacking a unified layout and rendering with the local video. Summary of the Invention

[0004] In view of the above problems, this application provides a method and related apparatus for handling remote video loss, so as to improve the stability of the canvas layout. The specific solution is as follows:

[0005] The first aspect of this application provides a method for handling missing remote video, the method comprising:

[0006] Determine whether the network is in a frame missing state based on the network status and the data verification results of the remote video frames;

[0007] If it is determined that there is a frame missing state, a target placeholder image is obtained, the size of which matches the size of the far-end video window;

[0008] The target placeholder image is written to the remote video window, and the target placeholder image and the local video image are pixel-fused in the remote video window.

[0009] In one possible implementation, determining whether a frame is missing based on network status and the data verification result of the remote video frame includes:

[0010] Receive remote video frames and simultaneously acquire network transmission parameters, including packet loss rate, bandwidth, and network round-trip delay.

[0011] Record the receiving timestamp of the remote video frame and obtain the data verification result of the remote video frame. At the same time, determine the waiting time threshold based on the packet loss rate and the bandwidth.

[0012] The waiting time is determined based on the receiving timestamp of the first type of remote video frame. At the same time, the number of the second type of remote video frames is counted. The first type of remote video frame is the frame that is closest to the current time and whose verification result is successful. The second type of remote video frame is at least one consecutive frame that is closest to the current time and whose verification result is unsuccessful.

[0013] A frame missing state is determined when at least two of the three conditions are met. The three conditions include the waiting time reaching the waiting time threshold, the number of the second type of remote video frames reaching the corresponding number threshold, the average packet loss rate within the first time window reaching the corresponding packet loss rate threshold, and the network round-trip delay reaching the corresponding delay threshold.

[0014] In one possible implementation, determining the waiting time threshold based on the packet loss rate and the bandwidth includes:

[0015] Determine the fluctuation value of the bandwidth within the second time window;

[0016] Obtain a base duration threshold, and adjust the base duration threshold according to the packet loss rate and the fluctuation value corresponding to the bandwidth to obtain the waiting duration threshold.

[0017] In one possible implementation, obtaining the target placeholder image includes:

[0018] Obtain the original placeholder image, wherein the original placeholder image is in YUV420P format;

[0019] The size of the remote video window is obtained according to the layout configuration file;

[0020] The original placeholder image is centered, cropped, and scaled to obtain the target placeholder image, the size of which is the same as the size of the remote video window.

[0021] In one possible implementation, the remote video missing processing method further includes:

[0022] If it is determined that there is no missing frame and the target placeholder image is written into the remote video window, the writing of the target placeholder image is stopped, the received remote video frame is written into the remote video window, and the screen of the remote video window is smoothly transitioned from the target placeholder image to the remote video frame.

[0023] A second aspect of this application provides a remote video missing processing device, the remote video missing processing device comprising:

[0024] The missing frame detection module is used to determine whether a frame is missing based on the network status and the data verification results of the remote video frame.

[0025] The missing frame processing module is used to obtain a target placeholder image when it is determined that there is a missing frame, the size of which matches the size of the remote video window; write the target placeholder image to the remote video window; and perform pixel fusion of the target placeholder image and the local video image under the remote video window.

[0026] In one possible implementation, the missing detection module is specifically used for:

[0027] The system receives remote video frames and simultaneously acquires network transmission parameters, including packet loss rate, bandwidth, and network round-trip delay. It records the reception timestamp of the remote video frames and acquires the data verification result of the remote video frames. Simultaneously, it determines a waiting time threshold based on the packet loss rate and the bandwidth. The system determines the waiting time based on the reception timestamp of the first type of remote video frames and simultaneously counts the number of the second type of remote video frames. The first type of remote video frames are the frames closest to the current time that have a successful verification result, and the second type of remote video frames are at least one consecutive frame closest to the current time that has a failed verification result. If at least two of three conditions are met, the system is determined to be in a frame missing state. These three conditions include the waiting time reaching the waiting time threshold, the number of the second type of remote video frames reaching a corresponding number threshold, the average packet loss rate within the first time window reaching a corresponding packet loss rate threshold, and the network round-trip delay reaching a corresponding delay threshold.

[0028] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the remote video missing processing method described in the first aspect or any implementation thereof.

[0029] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0030] The memory is used to store computer programs;

[0031] The processor is used to execute the computer program so that the electronic device can implement the remote video missing processing method of the first aspect or any implementation thereof.

[0032] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the remote video missing processing method described in the first aspect or any implementation thereof.

[0033] By employing the above technical solution, this application provides a method and related apparatus for handling missing remote video frames, comprising: determining whether a frame is missing based on network status and data verification results of the remote video frame; if a missing remote video frame is determined, acquiring a target placeholder image, the size of which matches the remote video window; writing the target placeholder image to the remote video window, and performing pixel fusion of the target placeholder image and the local video frame within the remote video window. This application, on the one hand, comprehensively detects whether a remote video frame is missing by considering network status and data verification results, reducing the false positive and false negative rates; on the other hand, it performs pixel fusion of the target placeholder image and the local video frame within the remote video window, thereby eliminating the sense of disjointedness and abrupt changes in the image, eliminating the need for users to readjust the layout, and improving the stability of the canvas layout. Attached Figure Description

[0034] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0035] Figure 1 A flowchart illustrating a remote video missing processing method provided in an embodiment of this application;

[0036] Figure 2 This is a partial flowchart illustrating a remote video missing processing method provided in an embodiment of this application;

[0037] Figure 3 This is another schematic flowchart of a remote video missing processing method provided in an embodiment of this application;

[0038] Figure 4 This is another schematic flowchart of a remote video missing processing method provided in an embodiment of this application;

[0039] Figure 5 A schematic diagram of a remote video missing processing device provided in an embodiment of this application;

[0040] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0041] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0042] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0043] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0044] To facilitate understanding of this application, the relevant concepts involved in this application will be explained below:

[0045] YUV420P: A video pixel format that stores luminance (Y) and chrominance (U / V) separately. The sampling rate of the chrominance component is 1 / 4 that of the luminance component. It effectively reduces the amount of data while ensuring a certain image quality and is often used in video encoding and decoding and image processing.

[0046] Remote video loss: refers to the phenomenon in audio and video call scenarios where the local end cannot receive valid remote video frames due to network interruption (such as network link disconnection, sudden bandwidth drop causing data transmission stagnation) or software streaming abnormality (such as streaming process crash or encoding module failure).

[0047] Placeholder image (also known as default image): An image provided by the system to replace the remote video feed when the remote video is missing. In this application, placeholder images can be divided into static base images (such as preset prompt images) and dynamic customized images (such as user-uploaded images). Their core function is to ensure the overall layout stability of the local video display area and avoid blank screens.

[0048] 1×2 layout: A common pattern for video display layout, which divides the display window into two equal parts horizontally. The left area is used to display the video of the local user, and the right area is used to display the video of the remote user. It is a typical layout for two-user audio and video calls.

[0049] Native YUV format adaptation: This technique directly converts the pre-stored placeholder image to YUV color space and adapts the pixel format (such as matching the YUV420P format for call scenarios), without requiring intermediate RGB format conversion, thus reducing data processing latency and image quality loss.

[0050] Currently, most mainstream video calls only monitor the time interval between remote video frame receptions. When the interval exceeds a fixed threshold (e.g., 3 seconds), it is determined that the remote video is missing. The local remote video window then displays a black screen or freezes the last frame, lacking a unified layout rendering with the local video. In addition, remote video missing detection often relies on a single time threshold, which is susceptible to network fluctuations leading to false positives. The placeholder image acquisition and processing methods also suffer from poor compatibility and high latency, ultimately resulting in a poor user experience.

[0051] To address the aforementioned issues, this application provides a method for handling missing remote video frames, which can accurately detect missing remote video frames, efficiently acquire adapted placeholder images, and maintain overall stable screen layout. The method for handling missing remote video frames according to this application will be described in detail below with reference to the accompanying drawings.

[0052] See Figure 1 , Figure 1 This is a flowchart illustrating a method for handling missing remote video data provided in an embodiment of this application. Figure 1 As shown in the figure, the remote video missing processing method provided in this application embodiment may include steps S101 to S103, which are described in detail below.

[0053] S101 determines whether the network is in a frame missing state based on the network status and the data verification results of the remote video frame.

[0054] In this embodiment, considering the impact of network fluctuations (such as brief network lag causing frame reception delays without actual interruption), the current frame missing state can be determined by combining the network status and the data verification results of the received remote video frames. Specifically, if the network status is abnormal and the data verification results of multiple consecutive remote video frames fail, it can be determined that the current state is frame missing. The data verification results of the remote video frames are obtained by verifying the integrity of the frame header, the data length, and whether decoding was successful.

[0055] In one possible implementation, the waiting time threshold can be dynamically adjusted based on network status, and combined with the data verification results of network status and remote video frames to accurately detect whether the current state is in a frame missing state, thereby improving the accuracy of remote video missing detection. See also Figure 2 , Figure 2This is a partial flowchart illustrating a remote video missing processing method provided in an embodiment of this application. Figure 2 As shown in the embodiment of this application, a remote video missing processing method is provided. Step S101, "determine whether it is in a frame missing state based on the network status and the data verification result of the remote video frame", may include steps S201 to S204. These steps are described in detail below.

[0056] S201 receives remote video frames and simultaneously acquires network transmission parameters, including packet loss rate, bandwidth, and network round-trip delay.

[0057] In this embodiment of the application, while receiving remote video frames, network transmission parameters are obtained in real time through RTCP (RTP Control Protocol). These network transmission parameters include the packet loss rate, bandwidth, and round-trip time (RTT) at the current time.

[0058] S202, record the receiving timestamp of the remote video frame and obtain the data verification result of the remote video frame. At the same time, determine the waiting time threshold based on the packet loss rate and bandwidth.

[0059] In this embodiment of the application, the receiving timestamp of the remote video frame is recorded. The time of the receiving timestamp can be accurate to milliseconds. In addition, the data verification result of the remote video frame is obtained. Specifically, when the integrity of the frame header and the data length of the remote video frame pass the verification and the decoding is successful, the data verification result is successful. Conversely, when the integrity of the frame header or the data length of the remote video frame fails the verification or the decoding is unsuccessful, the data verification result is unsuccessful.

[0060] At the same time, the base duration threshold for the current call scenario is dynamically adjusted based on the packet loss rate and bandwidth at the current time, thereby obtaining the waiting duration threshold at the current time.

[0061] To ensure that the waiting time threshold adapts to the current network conditions, it can be accurately calculated based on bandwidth fluctuations and packet loss rate, avoiding misjudgments or omissions in complex scenarios when using a fixed threshold. See also... Figure 3 , Figure 3 This is another schematic flowchart illustrating a remote video missing processing method provided in an embodiment of this application. Figure 3 As shown in the embodiment of this application, a remote video missing processing method is provided. In step S202, "determine the waiting time threshold based on the packet loss rate and bandwidth", which may include steps S301 to S302. These steps are described in detail below.

[0062] S301, determine the bandwidth fluctuation value within the second time window.

[0063] In this embodiment of the application, a time window (i.e., a second time window) can be set to statistically analyze the bandwidth fluctuation. The maximum, minimum and average values ​​of the bandwidth within the second time window are statistically analyzed, and the fluctuation value of the bandwidth within the second time window is calculated. Specifically, the fluctuation value = (maximum value - minimum value) / average value.

[0064] S302, obtain the basic duration threshold, and adjust the basic duration threshold according to the fluctuation value corresponding to the packet loss rate and bandwidth to obtain the waiting duration threshold.

[0065] In this embodiment of the application, a basic duration threshold for the current call scenario is obtained, and then the basic duration threshold is adjusted according to the following formula (1) to obtain a waiting duration threshold:

[0066] (1);

[0067] in, Indicates the waiting time threshold; Indicates the base duration threshold; This indicates the packet loss rate at the current time. This represents the fluctuation value corresponding to the bandwidth; and These are weighting coefficients, which can be calibrated based on experimental data. For example... , This ensures that the impact of packet loss rate and bandwidth fluctuations on the base duration threshold is within a reasonable range.

[0068] S203, determine the waiting time based on the receiving timestamp of the first type of remote video frame, and at the same time, count the number of the second type of remote video frames. The first type of remote video frame is the remote video frame that is closest to the current time and whose verification result is successful. The second type of remote video frame is at least one consecutive remote video frame that is closest to the current time and whose verification result is unsuccessful.

[0069] In this embodiment of the application, a first type of remote video frame and a second type of remote video frame can be determined from the received remote video frames. The first type of remote video frame is the frame that is closest to the current time and whose verification result is successful, and the second type of remote video frame is at least one consecutive frame that is closest to the current time and whose verification result is unsuccessful.

[0070] Furthermore, the time difference between the current time and the receiving timestamp of the first type of remote video frame is used as the waiting time to count the number of the second type of remote video frames.

[0071] S204. If at least two of the three conditions are met, the system is determined to be in a frame missing state. The three conditions include: the waiting time reaches the waiting time threshold, the number of second-type remote video frames reaches the corresponding number threshold, the average packet loss rate within the first time window reaches the corresponding packet loss rate threshold, and the network round-trip delay reaches the corresponding delay threshold.

[0072] The current time can be determined to be in a frame missing state if at least two of the following three conditions are met:

[0073] Condition 1: The waiting time reaches the waiting time threshold. That is, the waiting time is greater than or equal to the waiting time threshold.

[0074] Condition 2: The number of second-category remote video frames reaches the corresponding threshold. That is, the number of second-category remote video frames is greater than or equal to the corresponding threshold (e.g., 5).

[0075] Condition 3: The average packet loss rate within the first time window reaches the corresponding packet loss rate threshold (e.g., 20%), and the network round-trip latency reaches the corresponding latency threshold. That is, the average packet loss rate within the first time window is greater than or equal to the corresponding packet loss rate threshold, and the network round-trip latency at the current time reaches the corresponding latency threshold (e.g., 300ms). Here, the first time window is the time window for calculating the average packet loss rate.

[0076] S102, if it is determined that there is a frame missing state, obtain the target placeholder map, the size of the target placeholder map is matched with the size of the remote video window.

[0077] In this embodiment, when it is determined that the current time is in a frame missing state, the target placeholder image to be written can be read from a placeholder image in YUV420P format pre-stored in the local system or a placeholder image in YUV420P format defined by the user. The size of the target placeholder image matches the size of the remote video window. It should be noted that when the user uploads a custom placeholder image, if the placeholder image is not in YUV420P format, it can be converted to YUV420P format before being saved locally, avoiding the inefficiency problem caused by conversion during subsequent use. Therefore, this application provides a dual-path placeholder image acquisition and native YUV format adaptation processing solution, reducing placeholder image processing latency and improving format compatibility with call scenarios.

[0078] In one possible implementation, to ensure the target placeholder image fits the remote video window, the original local placeholder image can be captured and scaled to obtain the target placeholder image. See [link to relevant documentation]. Figure 4 , Figure 4 This is another schematic flowchart illustrating a remote video missing processing method provided in an embodiment of this application. Figure 4As shown in the embodiment of this application, a remote video missing processing method is provided. In step S102, "obtaining the target placeholder map" may include steps S401 to S403, which are described in detail below.

[0079] S401, retrieve the original placeholder image. The original placeholder image is in YUV420P format.

[0080] In this embodiment of the application, a placeholder image can be read locally as the original placeholder image, and the format of the original placeholder image is YUV420P.

[0081] S402, obtain the size of the remote video window based on the layout configuration file.

[0082] In this embodiment, the video display layout can be obtained from the layout configuration file, and the size of the remote video window can be determined based on the video display layout and the resolution of the entire display window. Taking a 1×2 layout as an example, assuming the resolution of the entire display window is 1920×1080, the coordinate range of the local video window in the 1×2 layout is [0,0]-[960,1080], and the coordinate range of the remote video window is [960,0]-[1920,1080], thus the size of the remote video window can be obtained as 960×1080.

[0083] S403, center-crops and scales the original placeholder image to obtain the target placeholder image, the size of the target placeholder image is the same as the size of the far video window.

[0084] In this embodiment, when the size of the original placeholder image differs from the size of the remote video window, the original placeholder image can be centered, cropped, and scaled to obtain a target placeholder image with the same size as the remote video window. For example, if the original placeholder image is 1280×720 and the remote video window is 960×1080, then the original placeholder image is centered and cropped according to the aspect ratio of the remote video window, using the center point as a reference. This ensures that the aspect ratio of the cropped image is the same as that of the remote video window. For instance, if the remote video window is 960×1080 with an aspect ratio of 8:9, the original placeholder image is centered and cropped to 640×720. Finally, the cropped result of the original placeholder image is scaled to obtain the target placeholder image, making the size of the target placeholder image the same as that of the remote video window.

[0085] S103, write the target placeholder image to the remote video window, and perform pixel fusion of the target placeholder image and the local video image in the remote video window.

[0086] In this embodiment, the target placeholder image is written to the output buffer of the remote video window according to the coordinate mapping result. Then, pixel fusion processing is performed on the target placeholder image and the local video image (i.e., the video frame of the local video window) under the remote video window. Specifically, the splicing edge of the remote video window and the local video window (such as the middle dividing line of a 1×2 layout) is determined. A 5-pixel wide area is taken on both sides of the splicing edge. The transparency of the local video image within the 5-pixel width and the transparency of the target placeholder image within the 5-pixel width are superimposed according to the gradient. For example, the transparency of the local video image within the 5-pixel width gradually changes from 100% to 50%, and the transparency of the target placeholder image within the 5-pixel width gradually changes from 50% to 100%, thereby eliminating the harshness of the splicing edge.

[0087] In one possible implementation, writing the target placeholder image to the output buffer of the remote video window can be stopped immediately after the remote video frame is recovered, and a smooth transition algorithm is used to switch the target placeholder image to the remote video frame, avoiding visual discomfort caused by abrupt changes in the image. To this end, the remote video missing processing method provided in this application embodiment further includes the following steps:

[0088] If it is determined that there is no missing frame and the target placeholder image is written into the remote video window, stop writing the target placeholder image, write the received remote video frame into the remote video window, and smoothly transition the screen of the remote video window from the target placeholder image to the remote video frame.

[0089] In this embodiment, when it is determined that the current time is not in a frame missing state and the target placeholder image is being written into the output buffer of the remote video window, the writing of the target placeholder image is stopped, and the remote video frame received at the current time is written into the output buffer of the remote video window. The image of the remote video window is smoothly transitioned from the target placeholder image to the remote video frame through a smooth transition algorithm (for example, gradually changing the transparency of the target placeholder image from 100% to 0% within 100ms, while gradually changing the transparency of the remote video frame from 0% to 100%).

[0090] In addition, in practical applications, after the remote video frame is recovered, the relevant historical data used to determine the frame missing status can be cleared and the dynamic waiting time threshold can be reset to prepare for the next possible frame missing detection.

[0091] Based on the above description, the remote video missing processing method provided in this application accurately identifies frame missing states through multi-dimensional detection and dynamic thresholding, distinguishing between "brief network lag" and "true video missing." Experimental verification shows that the false positive rate is reduced from 15% in existing solutions to below 3%, and the false negative rate is reduced from 8% to below 1%, significantly improving detection reliability. Furthermore, in this application, the system pre-stores YUV420P placeholder images without format conversion, with a call latency of <10ms; user-defined placeholder images avoid intermediate steps through native YUV conversion, with a processing latency of <50ms, which is more than 50% lower than existing RGB conversion solutions (latency 100-200ms). This results in low placeholder image processing latency and strong compatibility and image quality consistency. Furthermore, this application utilizes coordinate mapping and cropping / scaling to ensure that placeholder images accurately match the layout of the remote video window, avoiding image asymmetry. Edge blending and smooth transitions eliminate visual disjointedness and abrupt changes, eliminating the need for users to readjust to the layout. Contextualized placeholder images (e.g., distinguishing between network interruption and camera shutdown) help users quickly identify the cause of missing elements, reducing confusion. Finally, this application boasts strong scalability and adapts to various scenarios. The placeholder image library supports flexible expansion (e.g., adding customized corporate placeholder images or holiday-themed placeholder images), and the dynamic watermark function meets corporate branding needs. Dynamic threshold adjustment can be adapted to different network environments (e.g., 5G / 4G / Wi-Fi) through parameter calibration. Layout composition can also be expanded to support 2×2, 1×3, and other multi-user layouts, covering various scenarios such as two-person calls, multi-person conferences, and voice chat.

[0092] The above describes a remote video missing processing method provided by the embodiments of this application. The following will describe the apparatus for performing the above-described remote video missing processing method.

[0093] See Figure 5 , Figure 5 This is a schematic diagram of a remote video missing processing device provided in an embodiment of this application. Figure 5 As shown in the figure, an embodiment of this application provides a remote video missing processing device, comprising:

[0094] The missing frame detection module 501 is used to determine whether the frame is missing based on the network status and the data verification results of the remote video frame.

[0095] The missing frame processing module 502 is used to obtain a target placeholder image when it is determined that there is a missing frame. The size of the target placeholder image matches the size of the remote video window. The target placeholder image is written to the remote video window, and pixel fusion is performed between the target placeholder image and the local video image in the remote video window.

[0096] In one possible implementation, the missing detection module 501 is specifically used for:

[0097] The system receives remote video frames and simultaneously acquires network transmission parameters, including packet loss rate, bandwidth, and network round-trip delay. It records the reception timestamp of the remote video frames and obtains the data verification results. Simultaneously, it determines a waiting time threshold based on the packet loss rate and bandwidth. The system determines the waiting time based on the reception timestamp of the first type of remote video frames and counts the number of the second type of remote video frames. The first type of remote video frame is the one closest to the current time that has a successful verification result. The second type of remote video frame is at least one consecutive frame closest to the current time that has a failed verification result. If at least two of the following three conditions are met, the system is determined to be in a frame missing state: the waiting time reaches the waiting time threshold, the number of the second type of remote video frames reaches the corresponding number threshold, the average packet loss rate within the first time window reaches the corresponding packet loss rate threshold, and the network round-trip delay reaches the corresponding delay threshold.

[0098] In one possible implementation, the missing detection module 501, used to determine the waiting time threshold based on the packet loss rate and bandwidth, is specifically used for:

[0099] Determine the bandwidth fluctuation value within the second time window; obtain the base duration threshold, and adjust the base duration threshold according to the packet loss rate and the bandwidth fluctuation value to obtain the waiting duration threshold.

[0100] In one possible implementation, the missing data processing module 502 for obtaining the target placeholder image is specifically used for:

[0101] Obtain the original placeholder image in YUV420P format; obtain the size of the remote video window based on the layout configuration file; center-crop and scale the original placeholder image to obtain the target placeholder image, whose size is the same as that of the remote video window.

[0102] In one possible implementation, the missing data processing module 502 is also used for:

[0103] If it is determined that there is no missing frame and the target placeholder image is written into the remote video window, stop writing the target placeholder image, write the received remote video frame into the remote video window, and smoothly transition the screen of the remote video window from the target placeholder image to the remote video frame.

[0104] It should be noted that the detailed functions of each module in the embodiments of this application can be found in the corresponding disclosure of the above-mentioned remote video missing processing method embodiments, and will not be repeated here.

[0105] This application also provides an electronic device in its embodiments. See also... Figure 6 , Figure 6This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device in this embodiment may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0106] like Figure 6 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0107] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0108] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the remote video missing processing methods provided in this application.

[0109] This application also provides a computer-readable storage medium carrying one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the remote video missing processing methods provided in this application.

[0110] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0112] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0113] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for handling missing remote video, characterized in that, The remote video missing processing method includes: Receive remote video frames and simultaneously acquire network transmission parameters, including packet loss rate, bandwidth, and network round-trip delay. Record the receiving timestamp of the remote video frame and obtain the data verification result of the remote video frame. The data verification result of the remote video frame is obtained by verifying the integrity of the frame header, the data length, and whether the decoding is successful. At the same time, determine the waiting time threshold based on the packet loss rate and the bandwidth. The waiting time is determined based on the receiving timestamp of the first type of remote video frame. At the same time, the number of the second type of remote video frames is counted. The first type of remote video frame is the frame that is closest to the current time and whose verification result is successful. The second type of remote video frame is at least one consecutive frame that is closest to the current time and whose verification result is unsuccessful. If at least two of the three conditions are met, the system is determined to be in a frame missing state. The three conditions include the waiting time reaching the waiting time threshold, the number of the second type of remote video frames reaching the corresponding number threshold, the average packet loss rate within the first time window reaching the corresponding packet loss rate threshold, and the network round-trip delay reaching the corresponding delay threshold. If it is determined that there is a frame missing state, a target placeholder image is obtained, the size of which matches the size of the far-end video window; The target placeholder image is written to the remote video window, and the target placeholder image and the local video image are pixel-fused in the remote video window.

2. The remote video missing processing method according to claim 1, characterized in that, The step of determining the waiting time threshold based on the packet loss rate and the bandwidth includes: Determine the fluctuation value of the bandwidth within the second time window; Obtain a base duration threshold, and adjust the base duration threshold according to the packet loss rate and the fluctuation value corresponding to the bandwidth to obtain the waiting duration threshold.

3. The remote video missing processing method according to claim 1, characterized in that, The process of obtaining the target placeholder image includes: Obtain the original placeholder image, wherein the original placeholder image is in YUV420P format; The size of the remote video window is obtained according to the layout configuration file; The original placeholder image is centered, cropped, and scaled to obtain the target placeholder image, the size of which is the same as the size of the remote video window.

4. The remote video missing processing method according to claim 1, characterized in that, The remote video missing processing method also includes: If it is determined that there is no missing frame and the target placeholder image is written into the remote video window, the writing of the target placeholder image is stopped, the received remote video frame is written into the remote video window, and the screen of the remote video window is smoothly transitioned from the target placeholder image to the remote video frame.

5. A remote video missing processing device, characterized in that, The remote video missing processing device includes: The missing frame detection module is used to determine whether a frame is missing based on the network status and the data verification results of the remote video frame. The missing frame processing module is used to obtain a target placeholder image when it is determined that there is a missing frame, the size of which matches the size of the remote video window; write the target placeholder image to the remote video window; and perform pixel fusion between the target placeholder image and the local video frame under the remote video window. Specifically, the missing detection module is used for: The system receives remote video frames and simultaneously acquires network transmission parameters, including packet loss rate, bandwidth, and network round-trip delay. It records the reception timestamp of the remote video frames and acquires the data verification results, obtained by verifying frame header integrity, data length, and successful decoding. A waiting time threshold is determined based on the packet loss rate and bandwidth. The waiting time is determined based on the reception timestamps of the first type of remote video frames. Simultaneously, the number of the second type of remote video frames is counted. The first type of remote video frames are the frames closest to the current time that have a successful verification result, and the second type of remote video frames are at least one consecutive frame closest to the current time that has a failed verification result. If at least two of three conditions are met, the system is determined to be in a frame missing state. These three conditions include the waiting time reaching the waiting time threshold, the number of the second type of remote video frames reaching a corresponding quantity threshold, the average packet loss rate within a first time window reaching a corresponding packet loss rate threshold, and the network round-trip delay reaching a corresponding delay threshold.

6. A computer program product, characterized in that, Includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the remote video missing processing method as described in any one of claims 1 to 4.

7. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the remote video missing processing method as described in any one of claims 1 to 4.

8. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the remote video missing processing method as described in any one of claims 1 to 4.

Images