Teaching live lag abnormal rapid response system applied to online classroom

Abstract

The application relates to the technical field of image communication, in particular to a teaching live broadcast lag abnormality quick response system applied to an online classroom. The system comprises a data acquisition module, a lag monitoring module and a picture evaluation module. The data acquisition module is used for acquiring video frames and a packet loss rate in live broadcast. The lag monitoring module is used for judging whether lag occurs. The picture evaluation module is used for acquiring a flower screen image, dividing the flower screen image into a slightly disturbed image and a serious flower screen image based on color distribution characteristics of each flower screen image and the packet loss rate, dividing the slightly disturbed image into a replaceable image and a to-be-repaired image based on the similarity between the slightly disturbed image and other images, and performing image repair on the to-be-repaired image and replacing the replaceable image and the serious flower screen image, so that the live broadcast lag is restored to normal. According to the application, part of the flower screen images are repaired and the other flower screen images are replaced, the repair efficiency is improved, and the quick response of the teaching live broadcast lag is realized.

Description

Technical Field

[0001] This application relates to the field of image communication technology, specifically to a rapid response system for live teaching broadcasts experiencing buffering issues in online classrooms. Background Technology

[0002] With the continuous development of internet technology, live streaming has gradually become a common practice. Simultaneously, users' demands for live streaming quality are also increasing, focusing not only on image clarity but also on the smoothness of playback. However, in practical applications, network environments are significantly affected by factors such as location and time of day, with frequent bandwidth fluctuations that can easily lead to live streaming buffering and other anomalies. Therefore, building an anomaly handling system capable of rapidly responding to live streaming buffering and achieving timely and effective recovery of the live stream is of great significance for improving the overall quality of live streaming services.

[0003] When live stream stutters, current technologies typically address the resulting screen tearing by using image restoration algorithms to repair all affected video frames. However, since stuttering often stems from insufficient network bandwidth, repairing and replaying all stuttered images not only easily causes significant timing delays in the actual live stream but also increases the computational burden of image restoration, thus affecting overall processing efficiency and resulting in poor stuttering recovery. Summary of the Invention

[0004] To address the aforementioned technical issues, the purpose of this application is to provide a rapid response system for live online teaching events experiencing buffering or stuttering. The specific technical solution adopted is as follows: This application proposes a rapid response system for live teaching interruptions in online classrooms, the system comprising: Data acquisition module: Real-time acquisition of video frames and packet loss rate during live streaming; The stuttering monitoring module determines whether stuttering occurs in the live stream within each unit of time based on the packet loss rate. Screen evaluation module: If stuttering occurs within the current unit of time, obtain the screen distortion image, and construct the screen distortion severity index of each screen distortion image based on the randomness of the color distribution of each screen distortion image and the packet loss rate at the current time, so as to classify the screen distortion images into micro-interference images and severe screen distortion images. Each micro-interference image is divided into several connected components. Based on the texture features of each connected component, the similarity between the texture features of all adjacent connected components of each connected component, and the difference between the texture features of each connected component and its adjacent connected components, a screen distortion confidence score is constructed to characterize the probability that each connected component belongs to the screen distortion region, thereby obtaining the screen distortion connected components. Based on the similarity between each micro-interference image and all other micro-interference images within the same unit of time after removing the distorted connected components, and the similarity between the two nearest non-distorted images before and after it, a substitution index for each micro-interference image is constructed, and then the micro-interference images within the current unit of time are divided into replaceable images and images to be repaired. Image Repair Module: Performs image repair on all images to be repaired within the current unit of time, and replaces all replaceable images and severely distorted images to obtain a repaired video stream, thereby restoring normal operation of the live stream.

[0005] Preferably, the specific process for determining whether the live stream video experiences stuttering within each unit of time is as follows: If the packet loss rate is greater than or equal to the preset packet loss threshold within any unit of time, it is determined that the live broadcast video has experienced stuttering within that unit of time.

[0006] Preferably, the distorted image refers to a video frame image acquired when the packet loss rate is greater than or equal to a preset packet loss threshold.

[0007] Preferably, the method for constructing the severity index of each distorted image is as follows: Each distorted image is converted to HSV image format, and the range of hue values ​​is divided into a preset number of equidistant intervals; The ratio between the number of hue values ​​in each hue value interval and the total number of pixels in the image is recorded as the probability distribution of each hue value. This is used to calculate the information entropy between the hue values ​​of each distorted image. Statistical analysis of the packet loss rate at each acquisition time of the distorted screen image; The severity index of each distorted image is positively correlated with the information entropy and the packet loss rate.

[0008] Preferably, the specific process of dividing the distorted screen image into a slightly distorted image and a severely distorted screen image is as follows: Images with a screen distortion severity index less than or equal to a preset severity threshold are categorized as micro-interference images; conversely, those with a severity index greater than or equal to a preset severity threshold are categorized as severe screen distortion images.

[0009] Preferably, the method for constructing the confidence level of the screen distortion is as follows: An r×r window centered on each pixel is denoted as the neighborhood window of each pixel, and the neighborhood window is used as the input of the texture feature algorithm to output the texture value of each pixel; where r is a preset side length. The variance among the texture values ​​of all pixels in each connected component is denoted as the discrete texture value of each connected component. All connected components that are directly adjacent to each connected component in each micro-interference image are denoted as the adjacent connected components of each connected component. Calculate the absolute difference between the texture discrete values ​​of any two adjacent connected components in each connected component, and find the minimum value among them; Calculate the mean of the absolute differences between the texture discrete values ​​of each connected component and all its adjacent connected components; The confidence level of the screen distortion in each connected component is positively correlated with the texture discrete value and the mean value, and negatively correlated with the minimum value.

[0010] Preferably, the connected component of the distorted screen refers to the connected component whose distorted screen confidence is greater than or equal to a preset segmentation threshold.

[0011] Preferably, the method for constructing the substitution index of each micro-interference image is as follows: Take the union of all pixels corresponding to the distorted connected components in any two frames of micro-interference images, and remove the pixels in the union in both frames of micro-interference images. Then calculate the similarity between the two frames of micro-interference images after removing the pixels, and denot it as the similarity factor between the two frames of micro-interference images. Calculate the similarity factor between each micro-interference image and all other micro-interference images in the same unit of time, and count the maximum value among them; Calculate the similarity between the nearest non-flawed image in the previous frame and the nearest non-flawed image in the next frame for each micro-interference image, and denot it as the proximity similarity of each micro-interference image. The substitution index of each micro-interference image is positively correlated with the maximum value and the proximity similarity.

[0012] Preferably, the specific process of dividing the micro-interference image within the current unit time into replaceable images and images to be repaired is as follows: Cluster the substitution index of all micro-interference images within the current unit of time; If the number of clusters is 1, then when the mean of the replacement index of all micro-interference images is greater than the preset replacement threshold, all micro-interference images are recorded as replaceable images; otherwise, all micro-interference images are recorded as images to be repaired. If the number of clusters is greater than 1, all micro-interference images corresponding to the cluster with the largest mean replacement index are recorded as replaceable images, and the micro-interference images corresponding to other clusters are recorded as images to be repaired.

[0013] Preferably, the specific process of replacing all replaceable images and severely distorted images is as follows: The system uses the nearest neighbor of the previous frame of each frame that has been repaired or is normal within the current unit of time to replace the corresponding replaceable image or severely distorted image.

[0014] This application has the following beneficial effects: This application addresses the problem of low repair efficiency in existing technologies due to insufficient consideration of the substitutability of distorted images. It constructs a distorted image severity index by analyzing the degree of distortion in each frame, thereby quantifying the extent of damage to the teaching video caused by distorted images in different images. Furthermore, it constructs a distorted image confidence score by analyzing the texture features of each connected component in the micro-interference image and the texture similarity with adjacent connected components, assessing whether each connected component belongs to a distorted area. Finally, it constructs a substitution index by analyzing the similarity between normal areas in each micro-interference image and other images, evaluating the substitutability of micro-interference images in the video stream. This avoids inefficient repair of duplicate or redundant images, allowing for direct replacement of substitutable and severely distorted images, while only repairing other distorted images. This improves repair efficiency while maintaining image continuity, enabling rapid response to teaching livestream stuttering. Attached Figure Description

[0015] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a block diagram of a rapid response system for teaching live streaming interruptions in online classrooms, provided as an embodiment of this application. Figure 2 This is a flowchart illustrating the steps for monitoring and repairing live streaming stuttering issues, as provided in one embodiment of this application. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by this application to achieve the intended purpose of the invention, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of the rapid response system for teaching live streaming interruptions in online classrooms proposed according to this application. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0019] The specific solution of the rapid response system for teaching live broadcast interruptions in online classrooms provided in this application is described below with reference to the accompanying drawings.

[0020] Please see Figure 1 The diagram illustrates a block diagram of a rapid response system for teaching live streaming stuttering issues in online classrooms, provided by an embodiment of this application. The system includes: a data acquisition module, a stuttering monitoring module, a screen evaluation module, and a screen repair module.

[0021] Data acquisition module: Real-time acquisition of video frames and packet loss rate during live streaming.

[0022] The stuttering response system is deployed on a live streaming server equipped with parallel image processing units (such as GPUs and NPUs) to achieve real-time parallel processing of high frame rate video streams using hardware acceleration.

[0023] During the live teaching session, this application uses OBS Studio live streaming software to acquire video frames in real time. In this embodiment, the frame rate of the live video is 60 FPS. Simultaneously, a buffer loss rate system is used to collect packet loss rate data in real time. In this embodiment, the packet loss rate collection frequency is synchronized with the video frame rate and set to 60Hz.

[0024] Lag monitoring module: Based on packet loss rate, it determines whether there is lag in the live broadcast within each unit of time.

[0025] The system monitors the packet loss rate data through a stuttering response system. Specifically, in this embodiment, the system measures the data per second and determines whether stuttering occurs within that unit of time. A preset packet loss threshold is set (2% in this embodiment). If the packet loss rate is greater than or equal to the preset packet loss threshold within the current unit of time, it is determined that stuttering has occurred in the live broadcast within the current unit of time.

[0026] If a stuttering event occurs within the current unit of time, all images captured within that unit of time will be used as input to the image evaluation module for subsequent image analysis and repair.

[0027] Image Evaluation Module: If stuttering occurs within the current unit of time, obtain the distorted image. Based on the randomness of the color distribution of each distorted image and the packet loss rate at the current time, construct a distorted image severity index for each distorted image, thereby classifying the distorted images into minor interference images and severe distorted images. Divide each minor interference image into several connected components. Based on the texture features of each connected component, the similarity between the texture features of all adjacent connected components of each connected component, and the difference between the texture features of each connected component and its adjacent connected components, construct a distorted image confidence score to characterize the probability that each connected component belongs to the distorted region, thereby obtaining the distorted connected components. Based on the similarity between each minor interference image and all other minor interference images within the current unit of time after removing the distorted connected components, and the similarity between the two most recent non-distorted images before and after it, construct a replacement index for each minor interference image, thereby classifying the minor interference images within the current unit of time into replaceable images and images to be repaired.

[0028] Because packet loss rates vary under different bandwidth conditions, the severity of screen tearing in live streams also varies. Severe screen tearing often results in the loss of effective teaching scene characteristics, making repair not only difficult and inefficient, but also potentially disrupting the structural continuity of the live stream if the repair is ineffective. Therefore, removing severely torn footage helps improve repair efficiency while maintaining image continuity and effectively avoids further delays in the actual live stream.

[0029] Unlike live streams of games, movies, or sports events where the visuals change frequently, live teaching sessions primarily consist of teacher explanations, blackboard writing, and presentation slides, with relatively little overall visual variation. In such scenarios, the background is usually relatively fixed, the color types of text and graphics are limited, the color distribution is concentrated, and the color information entropy is at a relatively stable and low level.

[0030] If the screen distortion is mild, the live broadcast may show local distortion or brief interference, but the overall image structure will still retain its original features. However, in cases of severe screen distortion, the edges, contours, and regional relationships in the image will be significantly disrupted, the color distribution of the image will become disordered, and the information entropy will increase significantly.

[0031] Video frames acquired when the packet loss rate is greater than or equal to a preset packet loss threshold are all recorded as distorted images. The analysis is based on the i-th distorted image within the current unit of time. The i-th distorted image is converted to HSV image format, and the range of hue values ​​(H) (0-359) is divided into a preset number (180 in this embodiment) of equidistant intervals. The ratio between the number of hue values ​​within each interval and the total number of pixels in the image is recorded as the probability distribution of each hue value. This is used to calculate the information entropy between the hue values ​​of the i-th distorted image, reflecting the degree of disorder in color distribution and the level of chaos in the image structure. A larger value indicates more random color changes and a more unstable image structure. The calculation of information entropy is a well-known technique, and the specific process will not be elaborated further.

[0032] The information entropy of the hue values ​​of each frame of the distorted screen image transmitted to the screen repair module in the current unit time is calculated in the same way, and the information entropy of the hue value of the i-th frame of the distorted screen image is normalized to the maximum value to avoid the problem that the information entropy is too large and other subsequent parameters have a smaller effect.

[0033] Meanwhile, the packet loss rate corresponding to the acquisition time of the i-th frame of the distorted image is obtained to reflect the degree of network anomaly impact on the image during transmission; the higher the packet loss rate, the more severe the data transmission distortion, and the higher the possibility of severe distorted images.

[0034] In a preferred embodiment, a screen distortion severity index is constructed for each frame of the distorted image based on the randomness of the color distribution and the packet loss rate at that time. This index characterizes the degree of distortion in each frame. The method for constructing the screen distortion severity index is as follows: calculating the information entropy between the hue values ​​of each distorted image; statistically analyzing the packet loss rate corresponding to the acquisition time of each distorted image; the screen distortion severity index is positively correlated with both the information entropy and the packet loss rate. This positive correlation means that the dependent variable increases (decreases) as the independent variable increases (decreases).

[0035] In this embodiment, the product of the normalized information entropy of the hue value of the i-th frame of the distorted image and the packet loss rate at the corresponding time is denoted as the distortion severity index of the i-th frame of the distorted image. The distortion severity index can comprehensively reflect the degree of disorder in the color distribution of the i-th frame of the distorted image and the degree of network fluctuation during transmission; the larger the value, the higher the degree of interference from the distortion in the i-th frame of the distorted image, and the worse the image quality.

[0036] Calculation principle: Using information entropy to reflect the degree of disorder in data distribution is a common existing technique. Therefore, this application calculates the information entropy between the hue values ​​of each pixel to reflect the degree of color disorder in the distorted image. Packet loss rate can directly reflect the data loss during network transmission. Therefore, this application further introduces packet loss rate to quantify the impact of network anomalies on the image. Finally, the degree of distortion in the live broadcast image is comprehensively analyzed by calculating the product.

[0037] Furthermore, to avoid inefficient repair of severely distorted images, which would affect the structural continuity and overall efficiency of the live stream, severely distorted images are removed. Specifically, the severity index of each distorted image transmitted to the image repair module within the current unit of time is calculated; a preset severity threshold is set. In this embodiment, based on the statistical data of the severity index of historical severely distorted images, the preset severity threshold is set to 0.2; distorted images with a severity index less than or equal to the preset severity threshold are recorded as micro-interference images, representing live stream images that are only slightly affected by distorted images and still retain a relatively complete image structure; distorted images with a severity index greater than the preset severity threshold are recorded as severely distorted images, and no further image evaluation is performed on these images.

[0038] Furthermore, considering that in live-streaming teaching scenarios, to ensure users can clearly and completely view all text information, the video content often remains stable, leading to a large number of duplicate frames with identical content. Therefore, deduplication of these duplicate frames can be performed, and image repair operations can be applied to only one frame. This improves overall repair efficiency while maintaining image consistency and readability.

[0039] However, since the screen distortion phenomenon causes random changes in the image content, it is easy to produce large errors if we directly judge whether two frames are duplicate frames based on the content of the whole screen, and further processing is required.

[0040] The analysis will be based on the u-th frame of micro-interference image within the current unit of time.

[0041] The u-th frame of the micro-interference image is converted into a grayscale image. Then, an adaptive thresholding segmentation algorithm (such as the Otsu thresholding method or the Canny algorithm; this embodiment uses the Otsu thresholding method) is used to convert the grayscale image into a binary mask image. The resulting binary mask image is then used as input to a connected component generation algorithm to generate connected components. The connected component generation algorithm is not limited to region growing algorithms or seed filling methods. The connected component generation algorithm is a well-known technique, and its specific process will not be elaborated further.

[0042] Taking the v-th connected component as an example, we evaluate whether it is an extra image region caused by screen tearing. Specifically, the extra regions caused by screen tearing usually exhibit characteristics of disordered structure and random texture; while normal areas such as background or text in the teaching live broadcast image have relatively consistent internal texture features.

[0043] Therefore, an r×r window (where r is a preset side length, and in this embodiment r is 3) centered on each pixel is denoted as the neighborhood window of each pixel, and the neighborhood window is used as the input to the texture feature algorithm to output the texture value of each pixel. Then, the dispersion of the texture values ​​of all pixels in the v-th connected region is calculated and denoted as the texture discrete value of the v-th connected region. The texture discrete value can reflect the fluctuation of the texture distribution within the v-th connected region; the larger the value, the more drastic the texture change, the less stable the structure within the region, and thus the more likely it is caused by screen tearing. Calculating the dispersion to reflect whether the data changes drastically is a common existing technique. This application constructs a texture discrete value based on this, thereby reflecting the drastic degree of texture change within the connected region.

[0044] Texture feature algorithms are not limited to LBP and MLBP algorithms; LBP algorithm is used in this embodiment. The calculation of dispersion is not limited to variance, standard deviation, and coefficient of variation; variance is used in this embodiment.

[0045] Furthermore, since the distorted connected components are additional regions introduced by anomalies, their appearance often disrupts the original continuous and complete teaching screen structure. At the same time, since the originally continuous regions in the teaching live broadcast screen have consistent texture characteristics, if a certain region is abnormally divided by the v-th connected component, the connected components on both sides of the v-th connected component after the division will still maintain similar characteristics.

[0046] In the u-th frame of the micro-interference image, all connected components directly adjacent to the v-th connected component are denoted as the adjacent connected components of the v-th connected component. The absolute difference between the texture discrete values ​​of any two adjacent connected components of the v-th connected component is calculated, and the minimum value is denoted as the first difference of the v-th connected component. The first difference reflects the similarity of texture fluctuation features between adjacent connected components; the smaller the value, the more similar the texture fluctuation features between adjacent connected components, and thus the greater the probability that the v-th connected component is an anomaly. Specifically, when the v-th connected component has only one adjacent connected component, the first difference of the v-th connected component is set to 0, indicating that the texture fluctuation features within the neighborhood of the v-th connected component are consistent. Calculating differences to reflect the degree of data difference is a common existing technique; this application constructs the first difference based on this, thereby reflecting the consistency level of texture fluctuation features between adjacent connected components.

[0047] The mean of the absolute differences between the texture discrete values ​​of the v-th connected component and all its adjacent connected components is calculated and denoted as the first mean of the v-th connected component. The first mean reflects the degree of deviation between the v-th connected component and the overall uniform texture features of the surrounding area. The larger the value, the more obvious the texture difference between the v-th connected component and its adjacent areas, and thus the more likely it is caused by screen tearing.

[0048] Following the calculation steps for the texture discrete value, first difference, and first mean of the v-th connected component, the texture discrete value, first difference, and first mean of all connected components in the u-th frame micro-interference image are calculated. The maximum value normalization method is then used to normalize all texture discrete values, first differences, and first means to eliminate the influence of inconsistent dimensions on subsequent calculation results.

[0049] In a preferred embodiment, a screen distortion confidence score is constructed based on the texture features of each connected component, the similarity between the texture features of all adjacent connected components of each connected component, and the difference between the texture features of each connected component and its adjacent connected components. This score characterizes the probability that each connected component belongs to a screen distortion region. The method for constructing the screen distortion confidence score is as follows: obtaining the texture discrete values ​​of each connected component; recording all connected components directly adjacent to each connected component in each micro-interference image as adjacent connected components of each connected component; calculating the absolute difference between the texture discrete values ​​of any two adjacent connected components of each connected component, and calculating the minimum value; calculating the mean of the absolute differences between the texture discrete values ​​of each connected component and all its adjacent connected components; the screen distortion confidence score of each connected component is positively correlated with the texture discrete values ​​and the mean value, and negatively correlated with the minimum value. The negative correlation means that the dependent variable decreases (increases) as the independent variable increases (decreases).

[0050] In this embodiment, the confidence score of the screen distortion of the v-th connected component is denoted as... Its specific expression is: In the formula, Let be the confidence level of the screen distortion in the v-th connected component; , These are the normalized values ​​of the texture discrete values ​​of the v-th connected component and the normalized value of the first mean, respectively. This is the normalized value of the first difference of the v-th connected component; To preset the parameter tuning coefficients and avoid the denominator being 0, the value range is (0.05, 0.1). The influence of the value on the calculation result is negligible. In this embodiment, 0.08 is used.

[0051] The confidence score for screen distortion can comprehensively reflect the probability that the v-th connected component belongs to the screen distortion region. The larger the value, the more obvious the texture disorder of the v-th connected component is, the more obvious the degree of destruction of the consistency of the surrounding picture is, and the greater the texture difference between it and the surrounding picture is. Then the connected component is more likely to be an additional region caused by the screen distortion anomaly.

[0052] Furthermore, since the normal screen area of ​​the teaching live broadcast does not have the texture disorder features presented by the connected components of the screen distortion, its corresponding screen distortion confidence will be low; while because there are abnormal areas in the micro-interference image that are additionally generated by the screen distortion interference, the connected components in the image can be distinguished based on the screen distortion confidence, so as to effectively identify and locate the abnormal areas introduced by the screen distortion phenomenon.

[0053] Calculate the screen distortion confidence score of all connected components within the u-th frame of the micro-interference image, and use this score as input to the Otsu thresholding method. Output the maximum inter-class variance, and denote this value as the preset segmentation threshold. Connected components with a screen distortion confidence score greater than or equal to the preset segmentation threshold are denoteed as screen distortion connected components. The Otsu thresholding method is a well-known technique, and its specific process will not be elaborated further.

[0054] Based on the identification method of the distorted connected components in the u-th frame micro-interference image, identify all distorted connected components in the micro-interference images of other frames.

[0055] Furthermore, taking the micro-interference images of frame u and frame w as an example, the union of all corresponding pixels in the connected components of the distorted screen in the micro-interference images of frame u and frame w is taken, and the pixels in the union are uniformly removed in the micro-interference images of frame u and frame w, so as to ensure that the effective area range for comparison of the two frames remains consistent.

[0056] Subsequently, the similarity between the u-th frame and the w-th frame (after pixel removal) is calculated using an image similarity algorithm, and denoted as the similarity factor between the u-th and w-th frames. The similarity factor reflects the degree of consistency of the effective image content of the two frames after excluding screen distortion interference; the larger the value, the more similar the content of the non-distorted areas of the two frames.

[0057] Image similarity algorithms include, but are not limited to, the mask-based mean squared error (MSE) algorithm or the masked structural similarity (SSIM) algorithm. This application makes no limitation on these algorithms. In this embodiment, the mask-based mean squared error algorithm is used. Image similarity algorithms are well-known technologies, and the specific process will not be described in detail.

[0058] It should be noted that if the number of pixels remaining after removing the combined distorted areas in frame u or frame w is lower than a preset threshold (its value ranges from 5% to 10% of the total number of pixels in a single frame; in this embodiment, the preset threshold is set to 7% of the total number of pixels in a single frame), it means that there is too little remaining valid content, and the similarity calculation loses statistical significance. In this case, the similarity factor between the two frames is directly set to 0.

[0059] The similarity factors between the u-th frame micro-interference image and all other frames of micro-interference images within the same unit time are calculated sequentially, and the maximum value among them is recorded as the maximum similarity of the u-th frame micro-interference image. The maximum similarity reflects the maximum degree of consistency between the u-th frame and other micro-interference images in the effective image content.

[0060] Furthermore, for a video sequence, if the content of the two frames before and after a certain video frame is the same, then that video frame and the two frames before and after it are repeated frames.

[0061] Among all acquired frame images, the nearest non-flawed image before and after the u-th frame of micro-interference is obtained. An image similarity algorithm is then used to calculate the similarity between the two non-flawed images, denoted as the proximity similarity of the u-th frame of micro-interference. The proximity similarity reflects the continuity between the u-th frame of micro-interference and the preceding and following normal images; a higher value indicates a higher probability that the u-th frame of micro-interference is a repeating frame between two non-flawed images.

[0062] It should be noted that if there is no non-frosted image after the u-th micro-interference image, the similarity between the u-th micro-interference image and the previous non-frosted image is recorded as the proximity similarity of the u-th micro-interference image.

[0063] In a preferred embodiment, a substitution index for each micro-interference image is constructed based on the similarity between each micro-interference image and all other micro-interference images within the same unit time (after removing the distorted connected components), and the similarity between each image and the two nearest non-distorted frames. This index characterizes the degree to which each micro-interference image can be replaced by adjacent or other frames in a video sequence. The method for constructing the substitution index is as follows: calculate the similarity factor between each micro-interference image and all other micro-interference images within the same unit time, and count the maximum value; calculate the similarity between the nearest non-distorted frame and the nearest non-distorted frame of each micro-interference image, denoted as the proximity similarity of each micro-interference image; the substitution index of each micro-interference image is positively correlated with both the maximum value and the proximity similarity.

[0064] In this embodiment, the substitution index of the u-th frame micro-interference image is denoted as Its specific expression is: In the formula, Let be the substitution index of the micro-interference image in the u-th frame. , , respectively, represent the maximum similarity and the nearest neighbor similarity of the micro-interference image in the u-th frame; As a normalization function, this embodiment uses the minimum-maximum normalization method for normalization, where the minimum and maximum values ​​are obtained statistically based on the corresponding parameters of all micro-interference images.

[0065] In the above formula, by constructing a maximum similarity, the maximum similarity between frame u and other micro-interference images within the same time period can be reflected in terms of effective image content; by constructing a proximity similarity, the structural continuity and content consistency between frame u and its neighboring normal frames can be reflected; finally, by fusing the two, the substitutability of the micro-interference image of frame u can be comprehensively reflected. The resulting substitution index reflects the degree to which the micro-interference image of frame u can be replaced by adjacent or other frames in the video sequence; the larger the value, the higher the consistency of frame u with other images in terms of effective content, and thus the greater the possibility of it being replaced by other frames without affecting the overall viewing continuity.

[0066] Furthermore, following the calculation method for the substitution index of the u-th frame micro-interference image described above, the substitution index of all micro-interference images within the current unit time is calculated. All substitution indices are then used as input to a clustering algorithm for clustering. The clustering algorithm is not limited to k-means or DPC algorithms; this embodiment uses the k-means algorithm, and the optimal number of clusters is obtained through the elbow rule.

[0067] If the number of clusters is 1, then when the mean of the replacement index of all micro-interference images is greater than the preset replacement threshold, all micro-interference images are recorded as replaceable images; otherwise, all micro-interference images are recorded as images to be repaired. In this embodiment, the preset replacement threshold is 1.4.

[0068] If the number of clusters is greater than 1, all micro-interference images corresponding to the cluster with the largest mean replacement index are recorded as replaceable images, and the micro-interference images corresponding to other clusters are recorded as images to be repaired.

[0069] Image Repair Module: Performs image repair on all images to be repaired within the current unit of time, and replaces all replaceable images and severely distorted images to obtain a repaired video stream, thereby restoring normal operation of the live stream.

[0070] The system employs a bypass buffering mechanism. When a stutter is detected within the current unit of time, the front-end locks and displays the last normal frame, while the back-end asynchronously processes the stuttered video stream. After processing, the stuttered video stream is pushed to the playback buffer for accelerated continuous playback until the latest non-stuttered section is reached, thus ensuring continuous timestamps and a smooth, uninterrupted viewing experience. The specific process for processing the stuttered video stream involves: inputting all images to be repaired within the current unit of time into an image inpainting algorithm for repair, obtaining the repaired images; then, replacing the corresponding replaceable or severely distorted images with their nearest-neighbor, already repaired or normal images from the previous frame. This method improves the efficiency of image inpainting by repairing the video stream within the current unit of time.

[0071] Image inpainting algorithms include, but are not limited to, TSM (Temporal Shift Module) based inpainting, 3D convolution based inpainting, and optical flow-guided inpainting; image inpainting algorithms are well-known technologies and will not be elaborated here.

[0072] The flowchart for monitoring and repairing live stream buffering issues is as follows: Figure 2 As shown.

[0073] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0074] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0075] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A rapid response system for live online teaching interruptions, characterized in that: The system includes: Data acquisition module: Real-time acquisition of video frames and packet loss rate during live streaming; The stuttering monitoring module determines whether stuttering occurs in the live stream within each unit of time based on the packet loss rate. Screen evaluation module: If stuttering occurs within the current unit of time, obtain the screen distortion image, and construct the screen distortion severity index of each screen distortion image based on the randomness of the color distribution of each screen distortion image and the packet loss rate at the current time, so as to classify the screen distortion images into micro-interference images and severe screen distortion images. Each micro-interference image is divided into several connected components. Based on the texture features of each connected component, the similarity between the texture features of all adjacent connected components of each connected component, and the difference between the texture features of each connected component and its adjacent connected components, a screen distortion confidence score is constructed to characterize the probability that each connected component belongs to the screen distortion region, thereby obtaining the screen distortion connected components. Based on the similarity between each micro-interference image and all other micro-interference images within the same unit of time after removing the distorted connected components, and the similarity between the two nearest non-distorted images before and after it, a substitution index for each micro-interference image is constructed, and then the micro-interference images within the current unit of time are divided into replaceable images and images to be repaired. Image restoration module: Uses image restoration algorithms to restore all images to be restored within the current unit of time, and replaces all replaceable images and severely distorted images to obtain the restored video stream, thereby restoring the live broadcast from stuttering to normal. The specific process of dividing the micro-interference image within the current unit time into replaceable images and images to be repaired is as follows: Cluster the substitution index of all micro-interference images within the current unit of time; If the number of clusters is 1, then when the mean of the replacement index of all micro-interference images is greater than the preset replacement threshold, all micro-interference images are recorded as replaceable images; otherwise, all micro-interference images are recorded as images to be repaired. If the number of clusters is greater than 1, all micro-interference images corresponding to the cluster with the largest mean replacement index are recorded as replaceable images, and the micro-interference images corresponding to other clusters are recorded as images to be repaired.

2. The rapid response system for teaching live streaming interruptions applied to online classrooms as described in claim 1, characterized in that, The specific process for determining whether the live stream video experiences stuttering within each unit of time is as follows: If the packet loss rate is greater than or equal to the preset packet loss threshold within any unit of time, it is determined that the live broadcast video has experienced stuttering within that unit of time.

3. The rapid response system for teaching live streaming interruptions applied to online classrooms as described in claim 1, characterized in that, The "distorted image" refers to a video frame image captured when the packet loss rate is greater than or equal to a preset packet loss threshold.

4. The rapid response system for teaching live streaming interruptions applied to online classrooms as described in claim 1, characterized in that, The method for constructing the severity index of each distorted image is as follows: Each distorted image is converted to HSV image format, and the range of hue values ​​is divided into a preset number of equidistant intervals; The ratio between the number of hue values ​​in each hue value interval and the total number of pixels in the image is recorded as the probability distribution of each hue value. This is used to calculate the information entropy between the hue values ​​of each distorted image. Statistical analysis of the packet loss rate at each acquisition time of the distorted screen image; The severity index of each distorted image is positively correlated with the information entropy and the packet loss rate.

5. The rapid response system for teaching live streaming interruptions applied to online classrooms as described in claim 1, characterized in that, The specific process of dividing the distorted screen image into a slightly interfering image and a severely distorted screen image is as follows: Images with a screen distortion severity index less than or equal to a preset severity threshold are categorized as micro-interference images; conversely, those with a severity index greater than or equal to a preset severity threshold are categorized as severe screen distortion images.

6. The rapid response system for teaching live streaming interruptions applied to online classrooms as described in claim 1, characterized in that, The method for constructing the confidence level of the screen flickering is as follows: An r×r window centered on each pixel is denoted as the neighborhood window of each pixel, and the neighborhood window is used as the input of the texture feature algorithm to output the texture value of each pixel; where r is a preset side length. The variance among the texture values ​​of all pixels in each connected component is denoted as the discrete texture value of each connected component. All connected components that are directly adjacent to each connected component in each micro-interference image are denoted as the adjacent connected components of each connected component. Calculate the absolute difference between the texture discrete values ​​of any two adjacent connected components in each connected component, and find the minimum value among them; Calculate the mean of the absolute differences between the texture discrete values ​​of each connected component and all its adjacent connected components; The confidence level of the screen distortion in each connected component is positively correlated with the texture discrete value and the mean value, and negatively correlated with the minimum value.

7. The rapid response system for teaching live streaming interruptions applied to online classrooms as described in claim 1, characterized in that, The connected component of the distorted screen refers to the connected component whose distorted screen confidence is greater than or equal to a preset segmentation threshold.

8. The rapid response system for teaching live streaming interruptions applied to online classrooms as described in claim 1, characterized in that, The method for constructing the substitution index of each micro-interference image is as follows: Take the union of all pixels corresponding to the distorted connected components in any two frames of micro-interference images, and remove the pixels in the union in both frames of micro-interference images. Then calculate the similarity between the two frames of micro-interference images after removing the pixels, and denot it as the similarity factor between the two frames of micro-interference images. Calculate the similarity factor between each micro-interference image and all other micro-interference images in the same unit of time, and count the maximum value among them; Calculate the similarity between the nearest non-flawed image in the previous frame and the nearest non-flawed image in the next frame for each micro-interference image, and denot it as the proximity similarity of each micro-interference image. The substitution index of each micro-interference image is positively correlated with the maximum value and the proximity similarity.

9. The rapid response system for teaching live streaming interruptions in online classrooms as described in claim 1, characterized in that, The specific process for replacing all replaceable images and severely distorted images is as follows: The system uses the nearest neighbor of the previous frame of each frame that has been repaired or is normal within the current unit of time to replace the corresponding replaceable image or severely distorted image.

Images