A computer vision-based real-time video stream performance evaluation method and system
By inserting frame numbers and timestamps into real-time audio and video streams, and combining computer vision and deep learning models, the problem of incomplete real-time audio and video evaluation in existing technologies is solved, achieving automated, standardized, and efficient video stream performance evaluation.
Patent Information
- Application Number
- CN202610893460.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-22
- Publication Date
- 2026-11-03
AI Technical Summary
Existing real-time audio and video quality assessment methods suffer from incomplete indicators, high cost and low efficiency of manual testing, inability to quantify transmission performance and video quality, and high requirements for non-developers due to the high demands placed on LOG point-based methods.
A real-time video stream performance evaluation method based on computer vision is adopted. The video streams of the push and pull devices are acquired simultaneously through an image acquisition device, and frame numbers and timestamps are inserted. A deep learning model is used to automatically evaluate the smoothness, real-time performance, stability and picture quality.
It enables automated, standardized, and efficient evaluation of real-time audio and video quality, reducing subjective misjudgments and improving the accuracy and efficiency of the evaluation.
Smart Images
Figure CN123033073A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, and in particular relates to a method and system for evaluating the performance of real-time video streams based on computer vision. Background Technology
[0002] Real-time audio and video applications are widespread, involving multiple stages such as acquisition, encoding, transmission, decoding, and display; they are characterized by high real-time requirements, large data volume, high scene complexity, and high user experience requirements. In order to improve the user experience, a large number of tests need to be conducted on real-time audio and video. In the testing process, the following problems still exist: (1) The evaluation indicators of real-time audio and video quality are not comprehensive; (2) Manual testing is costly, and the cost of collecting and judging indicators is high and inefficient; (3) The observation method cannot quantify transmission performance and video picture quality, and cannot make an accurate assessment of transmission quality; (4) The LOG point-based data collection method requires the embedding of points at specific times in the source code, and LOG analysis and point-based data collection have high requirements for non-developers. Summary of the Invention
[0003] To address the aforementioned technical problems, this invention provides a real-time video stream performance evaluation method and system based on computer vision; aiming to monitor and analyze the quality and performance of images or video streams in real time. By introducing computer vision, deep learning models, and image processing algorithms, this invention statistically evaluates the fluency, real-time performance, stability, and picture quality of real-time audio and video, significantly reducing subjective misjudgments and human intervention in traditional manual inspection, and achieving automation, standardization, and efficiency in the evaluation process.
[0004] The first aspect of this invention proposes a real-time video stream performance evaluation method based on computer vision. The method includes: simultaneously acquiring video streams played by a push-streaming device and a pull-streaming device using an image acquisition device to obtain image data; and evaluating the played video stream based on four aspects: smoothness, real-time performance, stability, and picture quality, according to the acquired image data from the push-streaming device and the pull-streaming device.
[0005] Specifically: On the push end, a frame number and timestamp are inserted into each frame of the video stream to be played, and the video stream is encapsulated into a streaming media file before being pushed; on the pull end, after receiving the streaming media file, the video stream is extracted and played.
[0006] Smoothness assessment refers to calculating the average frame rate, frame drop rate, number of stutters, stutter duration, and stutter rate based on the video stream played by the streaming device captured by the image acquisition device, and evaluating smoothness based on the average frame rate, frame drop rate, number of stutters, stutter duration, and stutter rate.
[0007] Real-time performance evaluation refers to calculating the time delay by comparing the image data acquired from the push-stream and pull-stream devices to assess real-time performance.
[0008] Stability assessment refers to calculating the success rate of playback and the interruption rate based on the video stream played by the streaming device acquired by the image acquisition device, and evaluating stability based on the success rate of playback and the interruption rate.
[0009] Image quality assessment refers to: calculating the image anomaly rate based on the video stream played by the streaming device acquired by the image acquisition device, calculating the image fidelity by comparing the image data of the streaming device and the streaming device, and evaluating the image quality based on the image anomaly rate and the image fidelity.
[0010] In the method, the timestamp is equal to the product of the streaming end's encoded frame rate and the current frame number.
[0011] In the fluency assessment: Calculate the average frame rate: Perform frame extraction on the video stream played by the streaming device, extracting n image data frames sequentially. Calculate the frame rate for each image data frame, and then calculate the average frame rate of the n image data frames. The formula for calculating the average frame rate is: These represent the frame numbers of the last and first frames of the i-th image data frame, respectively. , The table contains the timestamps of the last and first frames of the i-th segment of image data. Calculate the frame drop rate: Perform OCR recognition on the video stream played by the streaming device. If adjacent frames are not continuous, the number of dropped frames is accumulated. Calculate the number of stutters: During the streaming process at the streaming end, if the display time of consecutive frame numbers exceeds a threshold, it is determined that a stutter has occurred. The threshold is the reciprocal of the encoding frame rate at the streaming end. Calculate the stutter duration: The stutter duration is equal to the difference between the display time of consecutive frame numbers and the threshold value; Calculate the stuttering rate: The stuttering rate is equal to the proportion of stuttering time to the total video stream duration.
[0012] In the real-time assessment: The acquired image data from the streaming device and the pulling device are processed by frame extraction and OCR recognition to form the streaming dictionary and the pulling dictionary, respectively. Using the frame number identified by OCR as the key and the timestamp identified by OCR as the value, the dictionary for both the push stream and the pull stream is sorted based on the frame number. Based on key-value pairs, the dictionary of the streaming end is traversed. For each frame of the streaming end, the corresponding frame of the pushing end is found. The difference in timestamps between the two is calculated as the time delay of the frame. The average time delay of all frames is calculated to obtain the average delay time, so as to evaluate the real-time performance.
[0013] In the stability assessment: Calculate the start-up success rate: Perform frame extraction processing on the video stream played by the acquisition end device, and extract n image data frames in sequence; for each image data frame, if the first m frames are successfully rendered, it is determined that the start-up is successful; the start-up success rate is equal to the number of successful start-ups / the total number of start-ups. Calculate the interruption rate: Perform frame extraction on the video stream played by the acquisition device and extract n image data frames in sequence; for each image data frame, determine the interruption if the frame number stops incrementing or there is no frame number during playback; the interruption rate is equal to the ratio between the number of interruptions and the total number of interruptions.
[0014] In the image quality assessment: Calculate the image anomaly rate: Perform frame extraction on the video stream played by the acquisition device. For each m frames of the extracted image data frames, use them as a group to determine the number of frames with image anomalies. The image anomaly rate is equal to the ratio between the number of frames with image anomalies and the total number of frames of the extracted image data frames. Image fidelity calculation: Peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are calculated by comparing the image data of the push-streaming device and the pull-streaming device. The peak signal-to-noise ratio (PSNR) is calculated as follows: For mean squared error, MAX I It is the maximum pixel value of the image frame at the streaming end. These are the x and y coordinates of the current pixel. It is the pixel value of the current pixel in the image frame at the streaming end. It is the pixel value of the current pixel in the image frame at the streaming end. and These are the image width and height, respectively. The Structural Similarity Index (SSIM) is calculated as follows: and These are the average brightness values of the push-stream image x and the pull-stream image y, respectively. and The images are the variances of the push-end image x and the pull-end image y, respectively. It is the covariance between the push-end image x and the pull-end image y. and It is the adjustment coefficient.
[0015] A second aspect of this invention proposes a real-time video stream performance evaluation system based on computer vision. The system comprises an image acquisition device, a streaming device, a streaming device, and a processing unit. The image acquisition device simultaneously acquires video streams played by both the streaming device and the streaming device to obtain image data. The processing unit evaluates the played video stream based on four aspects: smoothness, real-time performance, stability, and image quality, according to the acquired image data from the streaming device and the streaming device.
[0016] Specifically: On the push end, each frame of the video stream to be played is inserted with a frame number and a timestamp, and is encapsulated into a streaming media file, which is then pushed by the push end device; after receiving the streaming media file, the pull end device extracts the video stream and plays it.
[0017] Smoothness assessment refers to calculating the average frame rate, frame drop rate, number of stutters, stutter duration, and stutter rate based on the video stream played by the streaming device captured by the image acquisition device, and evaluating smoothness based on the average frame rate, frame drop rate, number of stutters, stutter duration, and stutter rate.
[0018] Real-time performance evaluation refers to calculating the time delay by comparing the image data acquired from the push-stream and pull-stream devices to assess real-time performance.
[0019] Stability assessment refers to calculating the success rate of playback and the interruption rate based on the video stream played by the streaming device acquired by the image acquisition device, and evaluating stability based on the success rate of playback and the interruption rate.
[0020] Image quality assessment refers to: calculating the image anomaly rate based on the video stream played by the streaming device acquired by the image acquisition device, calculating the image fidelity by comparing the image data of the streaming device and the streaming device, and evaluating the image quality based on the image anomaly rate and the image fidelity.
[0021] In the streaming end, each frame of the video stream to be played is inserted with a frame number and a timestamp; where the timestamp is equal to the product of the streaming end's encoded frame rate and the current frame number.
[0022] In summary, this invention inserts frame numbers and timestamps into the frames at the streaming end during the streaming process, uses a high-speed camera (image acquisition device) to simultaneously capture images at both the streaming end and the pulling end, and transmits the captured images to an automated analysis process; after analysis, it provides data on smoothness, real-time performance, stability, and image quality, thereby evaluating the performance of the real-time video stream. Attached Figure Description
[0023] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0024] Figure 1 This is a flowchart illustrating a real-time video stream performance evaluation method based on computer vision according to an embodiment of the present invention. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0026] The first aspect of this invention proposes a real-time video stream performance evaluation method based on computer vision, the method comprising (e.g.) Figure 1 As shown): The video stream played by the push and pull devices is simultaneously acquired using an image acquisition device to obtain image data; based on the acquired image data from the push and pull devices, the played video stream is evaluated based on four aspects: smoothness, real-time performance, stability, and picture quality.
[0027] Specifically: On the push end, a frame number and timestamp are inserted into each frame of the video stream to be played, and the video stream is encapsulated into a streaming media file before being pushed; on the pull end, after receiving the streaming media file, the video stream is extracted and played.
[0028] In some embodiments, the method provided by the present invention is used to automatically determine the quality of real-time audio and video and generate statistical data. The method requires a high-speed high-definition camera (image acquisition device) to simultaneously acquire image data from both the push-streaming device and the pull-streaming device.
[0029] Smoothness assessment refers to calculating the average frame rate, frame drop rate, number of stutters, stutter duration, and stutter rate based on the video stream played by the streaming device captured by the image acquisition device, and evaluating smoothness based on the average frame rate, frame drop rate, number of stutters, stutter duration, and stutter rate.
[0030] Real-time performance evaluation refers to calculating the time delay by comparing the image data acquired from the push-stream and pull-stream devices to assess real-time performance.
[0031] Stability assessment refers to calculating the success rate of playback and the interruption rate based on the video stream played by the streaming device acquired by the image acquisition device, and evaluating stability based on the success rate of playback and the interruption rate.
[0032] Image quality assessment refers to: calculating the image anomaly rate based on the video stream played by the streaming device acquired by the image acquisition device, calculating the image fidelity by comparing the image data of the streaming device and the streaming device, and evaluating the image quality based on the image anomaly rate and the image fidelity.
[0033] In the method, the timestamp is equal to the product of the streaming end's encoded frame rate and the current frame number.
[0034] In some embodiments, frame numbers and timestamps are inserted in real-time during real-time streaming processing, and the recorded stream is also pre-processed, including inserting frame numbers and timestamps. Preprocessing and data acquisition are required before evaluating real-time performance, smoothness, stability, and picture quality. The streaming end needs to insert a frame number and timestamp (timestamp optional) for each frame. The streaming data is divided into real-time data acquisition and locally recorded data. For real-time encoded videos, a white background with black text and a frame number and timestamp can be inserted at the bottom when frame data is acquired (other colors can also be used to improve OCR recognition rate). For recorded videos, preprocessing is required before streaming. After decoding each frame, a frame number and timestamp are inserted, and then the video is repackaged into a streaming media file according to the original encoding parameters.
[0035] The format is “Frame:number Time:hh.mm.ss.zzz”; where Frame represents a frame, number is the frame number of the frame, which increments by 1 for each frame, Time is the time, i.e. the current timestamp, hh represents hours, mm represents minutes, ss represents seconds, and zzz represents milliseconds.
[0036] When the streaming end only writes the frame number and not the timestamp, the timestamp can be calculated if the frame rate is known; it is the product of the streaming end's encoded frame rate and the current frame number.
[0037] In some embodiments, optical character recognition (OCR) technology is used to convert the extracted images into text to improve accuracy. A convolutional neural network (CNN) model is used for text detection. Before recognition, the image is segmented to extract the subtitles at the bottom of the image, thereby reducing the recognition range. After recognition, regular expressions are used to filter frames that match the frame number and timestamp.
[0038] In the fluency assessment: Calculate the average frame rate: Perform frame extraction on the video stream played by the streaming device, extracting n image data frames sequentially. Calculate the frame rate for each image data frame, and then calculate the average frame rate of the n image data frames. The formula for calculating the average frame rate is: These represent the frame numbers of the last and first frames of the i-th image data frame, respectively. , The table contains the timestamps of the last and first frames of the i-th segment of image data. Calculate the frame drop rate: Perform OCR recognition on the video stream played by the streaming device. If adjacent frames are not continuous, the number of dropped frames is accumulated. Calculate the number of stutters: During the streaming process at the streaming end, if the display time of consecutive frame numbers exceeds a threshold, it is determined that a stutter has occurred. The threshold is the reciprocal of the encoding frame rate at the streaming end. Calculate the stutter duration: The stutter duration is equal to the difference between the display time of consecutive frame numbers and the threshold value; Calculate the stuttering rate: The stuttering rate is equal to the proportion of stuttering time to the total video stream duration.
[0039] In some embodiments, the smoothness assessment uses parameters such as frame rate, frame drop rate, number of stutters, and stutter duration to evaluate the smoothness of the streaming video captured by a high-speed camera. The higher the frame rate, the lower the frame drop rate, the lower the stutter rate, and the better the smoothness.
[0040] During frame extraction, n data frames are taken sequentially, the frame rate of each segment is calculated, and finally the average frame rate is calculated. By simultaneously calculating and outputting the frame rate of each segment and the final average frame rate during the sequential frame extraction process, it is easier to evaluate the overall smoothness and the smoothness of each segment.
[0041] The phrase frame drop rate is calculated by performing OCR recognition on the captured video. If adjacent frames are not continuous, the number of dropped frames is accumulated.
[0042] Regarding the number of stutters, stuttering can be defined as the duration of consecutive frame numbers displayed during the streaming process exceeding a threshold. Theoretically, this threshold can be set to... FR is the encoding frame rate of the streaming end. The threshold standard can be changed. It can be defined as 200ms, which is a noticeable stuttering time; or 42ms, which is less than 24 frames, and the discontinuity of the picture can be felt.
[0043] Regarding the duration of stuttering, stuttering occurs when the display time of consecutive frame numbers exceeds a threshold during the streaming process. In this case, the stuttering duration is the time difference between consecutive frame numbers minus... The formula for stuttering rate is: In the real-time assessment: The acquired image data from the streaming device and the pulling device are processed by frame extraction and OCR recognition to form the streaming dictionary and the pulling dictionary, respectively. Using the frame number identified by OCR as the key and the timestamp identified by OCR as the value, the dictionary for both the push stream and the pull stream is sorted based on the frame number. Based on key-value pairs, the dictionary of the streaming end is traversed. For each frame of the streaming end, the corresponding frame of the pushing end is found. The difference in timestamps between the two is calculated as the time delay of the frame. The average time delay of all frames is calculated to obtain the average delay time, so as to evaluate the real-time performance.
[0044] In some embodiments, real-time performance is determined by comparing the time difference between the pushing and pulling ends, requiring frame extraction of the video. After OCR recognition preprocessing, an ordered dictionary is formed, using the frame numbers and timestamps of both the pushing and pulling ends: a push dictionary (`push_dict`) and a pull dictionary (`pull_dict`). The OCR-recognized frame number serves as the key, and the recognition timestamp as the value. The two dictionaries are sorted by frame number. The pull dictionary (`pull_dict`) is iterated through. For each frame from the pulling end, the corresponding push frame is found; the difference is the latency. The average of all differences is the average latency. A smaller average latency indicates better real-time performance.
[0045] In the stability assessment: Calculate the start-up success rate: Perform frame extraction processing on the video stream played by the acquisition end device, and extract n image data frames in sequence; for each image data frame, if the first m frames are successfully rendered, it is determined that the start-up is successful; the start-up success rate is equal to the number of successful start-ups / the total number of start-ups. Calculate the interruption rate: Perform frame extraction on the video stream played by the acquisition device and extract n image data frames in sequence; for each image data frame, determine the interruption if the frame number stops incrementing or there is no frame number during playback; the interruption rate is equal to the ratio between the number of interruptions and the total number of interruptions.
[0046] In some embodiments, stability includes two components: streaming start-up success rate and streaming interruption rate. The streaming start-up success rate and streaming interruption rate are automatically calculated; a higher start-up success rate indicates better quality, and a lower streaming interruption rate indicates better quality.
[0047] Statistical prerequisites are established, and a UI automation testing module is deployed for the operating system. The operation process is simultaneously recorded using a high-speed camera.
[0048] To calculate the success rate of video playback, frames are extracted from the recorded video from the high-speed camera streaming end. A successful playback is defined as the first m frames being rendered completely. OCR recognition is used to inspect the extracted images; if the first m frames are displayed completely, it is recorded as a success. The number of successful playbacks is then counted. The automated UI testing tool has recorded the total number of times the video started playing. Then we have: For dropout rate statistics, frames are extracted from the recorded video from the high-speed camera streaming end. A dropout is defined as a situation where the frame number stops incrementing or there are no frame numbers during complete playback. The number of dropouts is counted. The automated UI testing tool has recorded the total number of times the video started playing. Calculate the interruption rate: In the image quality assessment: Calculate the image anomaly rate: Perform frame extraction on the video stream played by the acquisition device. For each m frames of the extracted image data frames, use them as a group to determine the number of frames with image anomalies. The image anomaly rate is equal to the ratio between the number of frames with image anomalies and the total number of frames of the extracted image data frames. Image fidelity calculation: Peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are calculated by comparing the image data of the push-streaming device and the pull-streaming device. The peak signal-to-noise ratio (PSNR) is calculated as follows: For mean squared error, MAX I It is the maximum pixel value of the image frame at the streaming end. These are the x and y coordinates of the current pixel. It is the pixel value of the current pixel in the image frame at the streaming end. It is the pixel value of the current pixel in the image frame at the streaming end. and These are the image width and height, respectively. The Structural Similarity Index (SSIM) is calculated as follows: and These are the average brightness values of the push-stream image x and the pull-stream image y, respectively. and The images are the variances of the push-end image x and the pull-end image y, respectively. It is the covariance between the push-end image x and the pull-end image y. and It is the adjustment coefficient.
[0049] In some embodiments, the image quality assessment is divided into two dimensions: image anomalies and image restoration effect. Image anomalies include issues such as screen tearing, green screen, and pixelation. The image anomaly rate is calculated as the proportion of frames with anomalies to the total number of frames. Image restoration effect is assessed by extracting the same frame from the push and pull streams and calculating objective metrics such as PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index). PSNR measures the quality of video reconstruction; a higher value indicates less distortion. SSIM measures the difference between two images based on brightness, contrast, and structure; the closer the difference value is to 1, the better the image quality.
[0050] Image anomaly detection requires a large-scale model for judgment, necessitating the pre-training of a visual model. This visual model is a large-scale model specifically trained for image anomaly detection. The training of this large-scale model involves collecting both normal and abnormal images, normalizing, rotating, and flipping them to enhance their clarity, and then labeling them: normal images are labeled 0, and abnormal images are labeled 1. This labeled data is then fed into a Convolutional Neural Network (CNN) model for training. After evaluation and optimization, the model is deployed for use by automated devices.
[0051] For assessing the anomaly rate, the video recorded by the streaming device is processed by frame extraction. The extracted frames are arranged by frame number and grouped into m-frame sets, which are then fed into the visual model to determine if any are abnormal. The number of abnormal frames is then accumulated to obtain the final result. The total number of frames recorded during the traversal is... The anomaly rate is calculated. For frames extracted from the streaming client, a pre-trained model is used to identify anomalies and the percentage of anomaly frames is statistically analyzed. The lower the percentage of anomalies, the better the quality. Third-party models can also be used for anomaly detection.
[0052] For image fidelity evaluation, during playback at the pull end, frames are saved at fixed time intervals. The saved frame sequence and the push end video data are input into the processing unit for image fidelity evaluation. For the saved pull end frame data, frame numbers are identified, and an ordered array containing frame numbers and frame paths is recorded. For the ordered array saved at the pull end (pull dictionary), the push end (pull dictionary) is traversed to extract the frames corresponding to the frame numbers and frame paths, forming an ordered array at the push end. For pull and push streams of the same frame number, the PSNR and SSIM of the two frames are calculated.
[0053] In some embodiments, visual reports are automatically generated.
[0054] In some embodiments, smoothness and real-time performance evaluation only requires camera video data, stability evaluation requires UI automated testing tools and video data recorded by the streaming end, and image quality evaluation requires a pre-trained model, the original video from the streaming end, and frame-sampling data from the streaming end.
[0055] A second aspect of this invention proposes a real-time video stream performance evaluation system based on computer vision. The system comprises an image acquisition device, a streaming device, a streaming device, and a processing unit. The image acquisition device simultaneously acquires video streams played by both the streaming device and the streaming device to obtain image data. The processing unit evaluates the played video stream based on four aspects: smoothness, real-time performance, stability, and image quality, according to the acquired image data from the streaming device and the streaming device.
[0056] Specifically: On the push end, each frame of the video stream to be played is inserted with a frame number and a timestamp, and is encapsulated into a streaming media file, which is then pushed by the push end device; after receiving the streaming media file, the pull end device extracts the video stream and plays it.
[0057] Smoothness assessment refers to calculating the average frame rate, frame drop rate, number of stutters, stutter duration, and stutter rate based on the video stream played by the streaming device captured by the image acquisition device, and evaluating smoothness based on the average frame rate, frame drop rate, number of stutters, stutter duration, and stutter rate.
[0058] Real-time performance evaluation refers to calculating the time delay by comparing the image data acquired from the push-stream and pull-stream devices to assess real-time performance.
[0059] Stability assessment refers to calculating the success rate of playback and the interruption rate based on the video stream played by the streaming device acquired by the image acquisition device, and evaluating stability based on the success rate of playback and the interruption rate.
[0060] Image quality assessment refers to: calculating the image anomaly rate based on the video stream played by the streaming device acquired by the image acquisition device, calculating the image fidelity by comparing the image data of the streaming device and the streaming device, and evaluating the image quality based on the image anomaly rate and the image fidelity.
[0061] In the streaming end, each frame of the video stream to be played is inserted with a frame number and a timestamp; where: the timestamp is equal to the product of the streaming end's encoded frame rate and the current frame number.
[0062] In summary, this invention inserts frame numbers and timestamps into the frames at the streaming end during the streaming process, uses a high-speed camera (image acquisition device) to simultaneously capture images at both the streaming end and the pulling end, and transmits the captured images to an automated analysis process; after analysis, it provides data on smoothness, real-time performance, stability, and image quality, thereby evaluating the performance of the real-time video stream.
[0063] Please note that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. The above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A real-time video stream performance evaluation method based on computer vision, characterized in that, The method includes: The image acquisition device simultaneously acquires video streams played by both the push-streaming and pull-streaming devices to obtain image data. Based on the image data acquired from the push and pull devices, the played video stream is evaluated in four aspects: smoothness, real-time performance, stability, and picture quality. Specifically: On the push end, a frame number and timestamp are inserted into each frame of the video stream to be played, and the video stream is encapsulated into a streaming media file before being pushed; on the pull end, after receiving the streaming media file, the video stream is extracted and played. Among them, the smoothness assessment refers to: calculating the average frame rate, frame drop rate, number of stutters, stutter duration, and stutter rate based on the video stream played by the streaming device acquired by the image acquisition device, and evaluating the smoothness based on the average frame rate, frame drop rate, number of stutters, stutter duration, and stutter rate. Among them, real-time performance evaluation refers to: calculating the time delay by comparing the image data of the push-streaming device and the pull-streaming device to evaluate real-time performance; Among them, stability assessment refers to: calculating the start-up success rate and interruption rate based on the video stream played by the streaming device acquired by the image acquisition device, and assessing stability based on the start-up success rate and interruption rate; Among them, the picture quality assessment refers to: calculating the picture anomaly rate based on the video stream played by the pull-end device acquired by the image acquisition device, calculating the picture fidelity by comparing the image data of the push-end device and the pull-end device, and evaluating the picture quality based on the picture anomaly rate and the picture fidelity.
2. The real-time video stream performance evaluation method based on computer vision according to claim 1, characterized in that, In the method, the timestamp is equal to the product of the streaming end's encoded frame rate and the current frame number.
3. The real-time video stream performance evaluation method based on computer vision according to claim 2, characterized in that, In the fluency assessment: Calculate the average frame rate: Perform frame extraction on the video stream played by the streaming device, extracting n image data frames sequentially. Calculate the frame rate for each image data frame, and then calculate the average frame rate of the n image data frames. The formula for calculating the average frame rate is: These represent the frame numbers of the last and first frames of the i-th image data frame, respectively. , The table contains the timestamps of the last and first frames of the i-th segment of image data. Calculate the frame drop rate: Perform OCR recognition on the video stream played by the streaming device. If adjacent frames are not continuous, the number of dropped frames is accumulated. Calculate the number of stutters: During the streaming process at the streaming end, if the display time of consecutive frame numbers exceeds a threshold, a stutter is determined to have occurred. The threshold is the reciprocal of the encoding frame rate at the streaming end. Calculate the stutter duration: The stutter duration is equal to the difference between the display time of consecutive frame numbers and the threshold value; Calculate the stuttering rate: The stuttering rate is equal to the proportion of stuttering time to the total video stream duration.
4. The real-time video stream performance evaluation method based on computer vision according to claim 3, characterized in that, In real-time assessment: The acquired image data from the streaming device and the pulling device are processed by frame extraction and OCR recognition to form the streaming dictionary and the pulling dictionary, respectively. Using the frame number identified by OCR as the key and the timestamp identified by OCR as the value, the dictionary for both the push stream and the pull stream is sorted based on the frame number. Based on key-value pairs, the dictionary of the streaming end is traversed. For each frame of the streaming end, the corresponding frame of the pushing end is found. The difference in timestamps between the two is calculated as the time delay of the frame. The average time delay of all frames is calculated to obtain the average delay time, so as to evaluate the real-time performance.
5. The real-time video stream performance evaluation method based on computer vision according to claim 4, characterized in that, In stability assessment: Calculate the start-up success rate: Perform frame extraction processing on the video stream played by the acquired streaming device, and extract n image data frames in sequence; For each image data frame, if the first m frames are successfully rendered, the playback is considered successful; the playback success rate is equal to the number of successful playbacks / the total number of playbacks. Calculate the interruption rate: Perform frame extraction processing on the video stream played by the streaming device and extract n image data frames in sequence; For each image data frame, the case where the frame number stops incrementing or there is no frame number during playback is determined as a stream interruption; the stream interruption rate is equal to the ratio between the number of stream interruptions and the total number of stream interruptions.
6. The real-time video stream performance evaluation method based on computer vision according to claim 5, characterized in that, In image quality assessment: Calculate the abnormality rate of the video: Perform frame extraction processing on the video stream played by the acquired streaming device. For the image data frames obtained by frame extraction, each m frames is used as a group to determine the number of frames with abnormalities. The frame anomaly rate is equal to the ratio between the number of frames with frame anomalies and the total number of frames in the image data obtained by frame extraction. Image fidelity calculation: Peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are calculated by comparing the image data of the push-streaming device and the pull-streaming device. The peak signal-to-noise ratio (PSNR) is calculated as follows: For mean squared error, MAX I It is the maximum pixel value of the image frame at the streaming end. These are the x and y coordinates of the current pixel. It is the pixel value of the current pixel in the image frame at the streaming end. It is the pixel value of the current pixel in the image frame at the streaming end. and These are the image width and height, respectively. The Structural Similarity Index (SSIM) is calculated as follows: and These are the average brightness values of the push-stream image x and the pull-stream image y, respectively. and The images are the variances of the push-end image x and the pull-end image y, respectively. It is the covariance between the push-end image x and the pull-end image y. and It is the adjustment coefficient.
7. A real-time video stream performance evaluation system based on computer vision, characterized in that, The system includes an image acquisition device, a streaming end device, a streaming pull end device, and a processing unit; wherein: The image acquisition device simultaneously acquires video streams played by both the push-streaming and pull-streaming devices to obtain image data. The processing unit evaluates the played video stream based on four aspects: smoothness, real-time performance, stability, and picture quality, according to the image data acquired from the push and pull devices. Specifically: at the push end, each frame of the video stream to be played is inserted with a frame number and a timestamp, and is encapsulated into a streaming media file, which is then pushed by the push end device; after receiving the streaming media file, the pull end device extracts the video stream and plays it. Among them, the smoothness assessment refers to: calculating the average frame rate, frame drop rate, number of stutters, stutter duration, and stutter rate based on the video stream played by the streaming device acquired by the image acquisition device, and evaluating the smoothness based on the average frame rate, frame drop rate, number of stutters, stutter duration, and stutter rate. Among them, real-time performance evaluation refers to: calculating the time delay by comparing the image data of the push-streaming device and the pull-streaming device to evaluate real-time performance; Among them, stability assessment refers to: calculating the start-up success rate and interruption rate based on the video stream played by the streaming device acquired by the image acquisition device, and assessing stability based on the start-up success rate and interruption rate; Among them, the picture quality assessment refers to: calculating the picture anomaly rate based on the video stream played by the pull-end device acquired by the image acquisition device, calculating the picture fidelity by comparing the image data of the push-end device and the pull-end device, and evaluating the picture quality based on the picture anomaly rate and the picture fidelity.
8. The real-time video stream performance evaluation system based on computer vision according to claim 7, characterized in that, At the streaming end, each frame of the video stream to be played is inserted with a frame number and a timestamp; where: the timestamp is equal to the product of the streaming end's encoded frame rate and the current frame number.