Video quality evaluation system and method, electronic equipment and storage medium

Abstract

The invention provides a video quality evaluation system and method, an electronic device and a storage medium, and relates to the technical field of data processing. Key frames and features of an original video, namely a video to be processed, are extracted through a sending end, and the key frames and the features are sent to a receiving end together with a coded video through a cloud end; a receiving end obtains corresponding key frames and features thereof in the decoded video based on the same extraction mode, and further evaluates quality loss by comparing the features of the key frames at the two ends; quality evaluation is carried out based on key frame features instead of full video data, resource consumption of calculation and transmission is remarkably reduced, and meanwhile, due to the fact that quality loss evaluation covers the whole process of encoding, decoding and transmission, the video quality evaluation result is more comprehensive and objective.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a video quality assessment system and method, electronic device and storage medium. Background Technology

[0002] With the development of three-dimensional (3D) video technology, how to effectively evaluate video quality has become an important issue. In order to accurately and efficiently evaluate the overall quality of video, existing technologies usually render 3D video into images or videos and use image or video quality evaluation algorithms to evaluate the quality of the rendering results, or use a temporal graph neural network (GNN) to directly use 3D video as input and regress quality scores to evaluate video quality.

[0003] However, the dynamic mesh video data in 3D video technology is massive and usually needs to go through a series of complex processing steps before it is finally presented to the user. Existing video quality assessment technologies mostly assess the quality of the video itself and require point-by-point calculation of the entire video, which leads to low objectivity of the video quality assessment results and high resource consumption.

[0004] Therefore, improving the objectivity of video quality assessment and reducing its resource consumption are urgent problems to be solved. Summary of the Invention

[0005] This disclosure provides a video quality assessment system and method, electronic device, and storage medium. Its main objective is to address how to improve the objectivity of video quality assessment and reduce the resource consumption of video quality assessment.

[0006] According to a first aspect of this disclosure, a video quality assessment system is provided, comprising: a transmitter, a receiver, and a cloud. The sending end is configured to extract at least one first keyframe from the video to be processed, extract the first keyframe features of the first keyframe, and encode the video to be processed to obtain the encoded video. The cloud is configured to receive the encoded video, the first keyframe, and the first keyframe features from the sender, and then forward them to the receiver. The receiving end is configured to decode the encoded video to obtain the decoded video, extract at least one second keyframe corresponding to the first keyframe from the decoded video, and extract the second keyframe features of the second keyframe. The receiving end is also configured to perform quality loss assessment based on the first keyframe features and the second keyframe features to obtain a quality loss assessment result, wherein the method of extracting the second keyframe and the second keyframe features is the same as the method of extracting the first keyframe and the first keyframe features.

[0007] According to a second aspect of this disclosure, a video quality assessment method is provided, wherein the video quality assessment method is applied to a transmitting end in the video quality assessment system of the first aspect described above, comprising: Extract at least one first keyframe from the video to be processed, extract the first keyframe features of the first keyframe, and encode the video to be processed to obtain the encoded video. The encoded video, the first keyframe, and the first keyframe features are forwarded to the receiving end via the cloud so that the receiving end can perform a quality loss assessment based on the encoded video, the first keyframe, and the first keyframe features.

[0008] According to a third aspect of this disclosure, a video quality assessment method is provided, wherein the video quality assessment method is applied to a receiving end in the video quality assessment system of the first aspect described above, comprising: The received end forwards the encoded video, the first keyframe, and the features of the first keyframe via the cloud. The encoded video is decoded to obtain the decoded video. At least one second keyframe corresponding to the first keyframe is extracted from the decoded video, and the second keyframe features of the second keyframe are extracted. Quality loss assessment is performed based on the features of the first keyframe and the features of the second keyframe to obtain the quality loss assessment result. The method of extracting the second keyframe and the features of the second keyframe is the same as the method of extracting the first keyframe and the features of the first keyframe.

[0009] According to a fourth aspect of this disclosure, a video quality assessment method is provided, wherein the video quality assessment method is applied to the cloud within the video quality assessment system of the first aspect described above, comprising: Receive the encoded video, the first keyframe, and the first keyframe features sent by the sending end, and forward the encoded video, the first keyframe, and the first keyframe features to the receiving end; It receives decoded video from the receiving end and video to be processed from the sending end. The decoded video is obtained by the receiving end decoding the encoded video. Based on a pre-defined full-volume calculation model, quality loss is calculated and processed using the decoded video and the video to be processed to obtain the full-volume quality loss assessment result.

[0010] According to a fifth aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and A memory that is communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, such that the at least one processor is able to perform the methods of the second aspect, the third aspect, or the fourth aspect described above.

[0011] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods of the second aspect, the third aspect, or the fourth aspect described above.

[0012] According to a seventh aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method of the second aspect, the third aspect, or the fourth aspect described above.

[0013] The video quality assessment system, method, electronic device, and storage medium disclosed herein extract keyframes and their features from the original video (i.e., the video to be processed) at the sending end, and send them along with the encoded video to the receiving end via the cloud. The receiving end obtains the corresponding keyframes and their features from the decoded video using the same extraction method, and then assesses the quality loss by comparing the keyframe features at both ends. By performing quality assessment based on keyframe features rather than the full video data, the resource consumption for computation and transmission is significantly reduced. At the same time, since the quality loss assessment covers the entire encoding, decoding, and transmission process, the video quality assessment results are more comprehensive and objective.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0015] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a schematic diagram of the structure of a video quality assessment system provided in an embodiment of the present disclosure; Figure 2 This is a schematic diagram illustrating the overall process of video quality assessment provided in an embodiment of the present disclosure; Figure 3 A flowchart illustrating a video quality assessment method provided in an embodiment of this disclosure; Figure 4 A flowchart illustrating another video quality assessment method provided in this embodiment of the present disclosure; Figure 5A flowchart illustrating another video quality assessment method provided in this embodiment of the present disclosure; Figure 6 This is a schematic block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0016] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0017] The following description, with reference to the accompanying drawings, describes a video quality assessment system and method, electronic device, and storage medium according to embodiments of the present disclosure.

[0018] Figure 1 This is a schematic diagram of the structure of a video quality assessment system provided in an embodiment of the present disclosure.

[0019] like Figure 1 As shown, the system includes: a transmitter, a receiver, and a cloud platform. The sending end is configured to extract at least one first keyframe from the video to be processed, extract the first keyframe features of the first keyframe, and encode the video to be processed to obtain the encoded video. The cloud is configured to receive the encoded video, the first keyframe, and the first keyframe features from the sender, and then forward them to the receiver. The receiving end is configured to decode the encoded video to obtain the decoded video, extract at least one second keyframe corresponding to the first keyframe from the decoded video, and extract the second keyframe features of the second keyframe. The receiving end is also configured to perform quality loss assessment based on the first keyframe features and the second keyframe features to obtain a quality loss assessment result, wherein the method of extracting the second keyframe and the second keyframe features is the same as the method of extracting the first keyframe and the first keyframe features.

[0020] In this context, the sending end refers to the starting device or node in the video data processing flow, typically located on the data acquisition or generation side. Examples include a 3D reconstruction workstation equipped with a depth camera, a dynamic mesh sequence generation server, or an edge computing device with encoding capabilities. The sending end is responsible for processing the raw video data to be processed. This raw video data requires quality assessment, such as dynamic mesh sequences or 3D video data. The data format of the video to be processed includes, but is not limited to, point clouds, mesh model sequences, or other data forms representing dynamic 3D scenes.

[0021] The first keyframe refers to a frame selected from the video to be processed using a specific algorithm that represents significant changes in video content over a period of time or has high information representativeness. Various strategies can be employed for selecting the first keyframe, such as fixed-time interval sampling, motion-alteration-based detection, or rendering image residual analysis, to ensure that the selected frame effectively represents the content features of its adjacent time periods. The first keyframe feature refers to a high-dimensional abstract representation extracted from the first keyframe using a feature extraction network or algorithm. The first keyframe feature can capture key visual or structural information within the frame, such as grid structure features extracted using a graph neural network or projected image features extracted using a convolutional network. The purpose of feature extraction is to transform high-dimensional, computationally complex raw data into low-dimensional feature vectors rich in semantic information, thereby significantly reducing subsequent computation.

[0022] In addition, the sending end is also responsible for encoding and compressing the video to be processed, using codecs such as DRACO to generate the encoded video. The encoded video refers to the video data stream after compression and encoding, and its data volume is significantly reduced compared to the original video to be processed, which facilitates network transmission and storage.

[0023] The cloud, as a relay, coordination, and complex data computing node, is usually composed of a cluster of remote servers with high bandwidth and storage capacity. The cloud plays the role of data routing and temporary storage, ensuring reliable data exchange between the sending and receiving ends, especially in long-distance and cross-network transmission scenarios, effectively connecting distributed edge devices.

[0024] The receiving end is typically located on the user side or the service side, such as a mobile terminal or an edge rendering server. It is the device that ultimately decodes, renders, and presents the video data to the user. Decoded video refers to the video data reconstructed after the encoding and decoding process. Its content should be as consistent as possible with the original video to be processed, but distortion or quality loss may be introduced during the encoding and transmission process.

[0025] Subsequently, the receiving end needs to extract at least one second keyframe from the decoded video that corresponds to the first keyframe selected by the sending end. A second keyframe is a frame in the decoded video sequence that corresponds to the original first keyframe selected by the sending end in terms of time or content. To ensure the validity and consistency of the comparison, the selection logic for the second keyframe is consistent with that of the sending end. The second keyframe feature refers to the feature representation calculated from the second keyframe using the exact same feature extraction algorithm and network structure as the sending end.

[0026] Quality loss assessment is a quantitative indicator used to characterize the degree of quality loss suffered by a video after the entire process of encoding, transmission, and decoding. Quality loss assessment can be achieved by calculating the difference between features of two keyframes (e.g., distance or similarity in feature space) and inputting it into a regression network (e.g., multilayer perceptron) to obtain an absolute score or a relative loss amount.

[0027] This disclosure significantly reduces the resource requirements of the sending and receiving ends and network bandwidth by performing feature extraction and comparison only at the keyframe level, rather than processing the entire video data. Furthermore, because keyframe extraction and feature extraction are performed in exactly the same way at both the sending and receiving ends (the term "way" is a general term encompassing the specific algorithms, model structures, and parameter configurations used to complete keyframe selection and feature extraction), it ensures that the extracted features reside in the same comparable feature space, thereby guaranteeing the accuracy and consistency of the quality loss assessment results.

[0028] In one possible implementation of this disclosure, the sending end is specifically configured to extract at least one first keyframe from the video to be processed using a first keyframe selection algorithm. The receiving end is specifically configured to extract at least one second keyframe corresponding to the first keyframe from the decoded video according to the same second keyframe selection algorithm as the first keyframe selection algorithm. The first keyframe selection algorithm and the second keyframe selection algorithm include at least one of the following: extracting frames at fixed intervals, extracting frames based on the displacement distance of key points, or extracting frames based on the residual of the rendered image.

[0029] In the embodiments of this disclosure, the first keyframe selection algorithm refers to a method at the sending end that identifies and extracts representative first keyframes from a video sequence to be processed through a pre-set, non-manually-intervened automated program or calculation rules. The purpose of the first keyframe selection algorithm is to filter out frames rich in information and representative of the video content state within their respective time periods from a lengthy video stream, thereby significantly reducing the amount of data that needs to be processed subsequently, while ensuring the accuracy of quality assessment. Correspondingly, at the receiving end, the second keyframe selection algorithm is completely consistent with the first keyframe selection algorithm in terms of functional objectives, logical steps, and core parameter settings. This ensures that for the same video content, even after encoding, transmission, and decoding processes, the receiving end can still locate and extract a second keyframe that highly corresponds in time and content to the original first keyframe at the sending end, laying the foundation for subsequent accurate feature comparison.

[0030] The first keyframe selection algorithm and the second keyframe selection algorithm can be implemented using various strategies, including at least the following embodiments: 1. Frames are extracted at fixed intervals. Without considering the duration or amplitude of motion, frames at fixed intervals are designated as keyframes. This ensures that a sufficient number of keyframes are generated periodically to support subsequent algorithms such as quality loss calculation. Fixed-interval frame extraction refers to extracting frames as keyframes from a video sequence at fixed time or frame intervals, regardless of the drastic changes in the video content. For example, it can be set to extract one frame every N frames (N being an integer greater than 1). The advantages are its simple algorithm, minimal computational overhead, and ability to ensure a uniform distribution of keyframes along the timeline, thus providing a stable and predictable sampling basis for the entire video sequence and ensuring the coverage of quality assessment.

[0031] 2. Frames are extracted based on keypoint displacement distances. In a moving scene, if the keypoint displacement distances in consecutive frames are small, it can be assumed that the overall change in these consecutive frames is not significant and the quality is similar. Therefore, frames with large keypoint displacement distances can be considered to have abrupt motion changes. By setting these abrupt motions as keyframes, it can be seen as a quality assessment of each motion segment. Frame extraction based on keypoint displacement distance is suitable for video content containing significant motion, such as human movements or object displacements in dynamic mesh sequences. The core principle is to first identify and track keypoints in the video sequence; keypoints are points in video frames that have significant features and are easy to track, such as the 3D coordinates of joints in a dynamic mesh or the 2D pixel coordinates of feature points in an image. By calculating the displacement distance of keypoints between consecutive frames and comparing it to a preset threshold, when the displacement distance exceeds the preset threshold, it is determined that the frame has undergone a significant motion change, and thus it is extracted as a keyframe. This method can more accurately capture moments of substantial change in video content, making the keyframe set more representative of information, thereby potentially improving the sensitivity of quality loss assessment.

[0032] 3. Extract frames based on the residual of the rendered image, render each frame of the video model as an image, and calculate the residual between two frames to generate a residual image. Use an image-based algorithm to identify the residual image and select keyframes.

[0033] Frame extraction based on residuals from rendered images is suitable for scenarios where 3D video frames (such as mesh models) can be rendered into 2D images. The process involves rendering two consecutive video frames into two separate images, and then calculating the residual between these two images. The residual refers to the difference in corresponding pixels between the two images, directly reflecting the amount of visual content change between the two frames. By analyzing the overall intensity of the residual images (e.g., the sum or average of pixel differences), the degree of change between frames can be quantified. When the residual value exceeds a certain threshold, it indicates a significant change in content, and the corresponding frame is selected as a keyframe. From the perspective of the final rendered output's visual presentation, it has a high correlation with changes perceived by the human visual system.

[0034] When extracting frames based on keypoint displacement distance and frame extraction based on rendered image residuals, if there are long periods without keyframes, keyframes must be selected at fixed intervals to ensure that subsequent processes can proceed.

[0035] The keyframe selection algorithms described above can be used individually or in combination, depending on the specific application scenario, to balance efficiency and accuracy. Through an algorithmic frame selection strategy, this disclosure achieves objectivity, consistency, and automation in the keyframe extraction process, providing a core guarantee for the reliability and efficiency of the entire quality assessment process.

[0036] In one possible implementation of this disclosure, the sending end is specifically configured to extract the first keyframe features of the first keyframe based on a first feature extraction algorithm.

[0037] The receiving end is specifically configured to extract the second keyframe features of the second keyframe based on the same second feature extraction algorithm as the first feature extraction algorithm.

[0038] In the embodiments of this disclosure, the first feature extraction algorithm refers to the computational process or model architecture employed at the sending end to extract a high-dimensional abstract representation (first keyframe feature) from the first keyframe. The first feature extraction algorithm is not a simple data preprocessing step, but a complex process capable of deeply mining the structural and semantic information contained within the keyframe. Its purpose is to transform the computationally intensive and dimensionally complex original keyframe data into a low-dimensional, dense, and highly condensed feature vector. This transformation process significantly reduces the overhead of subsequent data transmission and computational comparison.

[0039] At the receiving end, the second feature extraction algorithm is strictly identical to the first feature extraction algorithm in terms of network structure, number of layers, parameter settings, and input / output dimensions. This algorithmic consistency is a mechanism to ensure feature comparability, ensuring that for keyframes with identical content but potentially subject to minor distortions introduced during encoding / decoding (i.e., the first keyframe at the sending end and the second keyframe at the receiving end), the difference between the extracted feature vectors truly reflects the quality degradation caused by encoding / decoding transmission, rather than stemming from systematic biases introduced by the different extraction algorithms at both ends.

[0040] The first feature extraction algorithm can be concretized as a pre-trained deep learning model. For example, when the keyframe is a 3D mesh frame, a graph neural network can be used as the feature extractor, which can directly process non-Euclidean space mesh data and generate global features by aggregating the relationship information between vertices and edges. If the keyframe has been rendered as a 2D image, a convolutional neural network can be used as the feature extraction algorithm, leveraging its powerful image feature extraction capabilities to obtain effective feature representations. Regardless of the specific model, deploying identical models, even with the same weights, at both the sending and receiving ends is crucial to ensuring consistency in feature extraction. Through standardization and alignment at the algorithm level, a fair and accurate feature comparison platform is constructed, making quality loss assessment based on feature differences highly credible and practically valuable.

[0041] In one possible implementation of this disclosure, the receiving end is specifically configured as follows: Input at least one first keyframe feature and a second keyframe feature into a preset quality loss model to perform quality loss evaluation processing and obtain at least one quality loss score. In response to at least one quality loss score being less than a preset quality loss threshold, a quality loss prediction process is performed based on a preset prediction method according to at least one quality loss score to obtain a predicted quality loss assessment result. The preset prediction method is determined based on the second keyframe selection algorithm.

[0042] In the embodiments of this disclosure, the preset quality loss model is a pre-trained computational model with fixed parameters that receives one or more pairs of keyframe features as input and outputs a quantified score. The preset quality loss model is typically designed as a function capable of learning and measuring the difference or distortion between two features. For example, a dual-input multilayer perceptron receives a first keyframe feature from the transmitter and a second keyframe feature from the receiver, respectively. This network, through forward propagation, ultimately outputs a scalar value, namely the quality loss score; the quality loss score is a specific numerical value used to quantitatively characterize the degree of quality loss suffered by a single keyframe after the encoding / decoding transmission process. A higher score generally indicates greater quality loss and more severe distortion.

[0043] The receiving end inputs one or more extracted first keyframe features and their corresponding second keyframe features into this preset quality loss model, thereby obtaining one or more quality loss scores corresponding to each keyframe. Subsequently, the system compares each score with a preset quality loss threshold; the preset quality loss threshold is a pre-set threshold value that defines the maximum acceptable single keyframe quality loss boundary of the system, used to determine whether the quality status of the current encoding / decoding transmission channel is at a stable and acceptable level.

[0044] When the comparison results show that the quality loss scores calculated for all input keyframes are less than the preset quality loss threshold, it indicates that the entire video sequence suffers relatively small and uniform quality loss under the current transmission environment, and is in a healthy state. At this point, a computationally intensive full evaluation will not be initiated. Instead, based on a preset pre-estimation algorithm, quality loss prediction processing will be performed according to the quality loss scores of the obtained keyframes, ultimately yielding a predicted quality loss evaluation result. The predicted quality loss evaluation result refers to the estimate of the global quality loss of the entire video sequence obtained through mathematical deduction or weighted calculation based on the loss scores of a subset of sampled keyframes. In other words, under a stable state with relatively small quality loss, the loss of a subset of keyframes can effectively represent and infer the average quality level of the entire video.

[0045] It should be noted that the preset estimation method used is not arbitrarily chosen; its specific form is determined based on the second keyframe selection algorithm. The preset estimation method is tightly coupled with the keyframe selection strategy. For example, if keyframes are selected by extracting frames at fixed intervals, meaning the keyframes are evenly distributed over time, then the preset estimation method can be a simple arithmetic average of the quality loss scores of all keyframes to obtain the global estimated quality loss. If keyframes are selected by extracting frames based on the displacement distance of keypoints, meaning the time period represented by each keyframe may be different, then the preset estimation method will use a weighted average, using the duration represented by each keyframe as the weight for calculation. This ensures the mathematical consistency between the estimation logic and the sampling strategy, making the final estimation result more scientific and accurate.

[0046] In one possible implementation of this disclosure, the receiving end is further configured to transmit the decoded video to the cloud and send a video transmission instruction to the sending end in response to at least one quality loss score having a first preset number of quality loss scores not less than a preset quality loss threshold. The sending end is also configured to send the video to be processed to the cloud in response to video transmission commands; In the cloud, it is also configured to perform quality loss calculation based on the decoded video and the video to be processed according to the preset full calculation model, and obtain the full quality loss assessment result, which includes the estimated quality loss assessment result or the full quality loss assessment result.

[0047] In the embodiments of this disclosure, the first preset quantity is a configurable parameter whose value can be set to one or more. It defines the severity of the condition that triggers the full calculation mode. For example, setting it to 1 means that if the quality loss score of any keyframe exceeds the threshold, the current transmission state is considered to have a potential widespread problem, thus initiating an emergency procedure. After calculating one or more quality loss scores, the receiving end will evaluate them. Once it is found that any score reaching the first preset quantity is not less than (i.e., greater than or equal to) the preset quality loss threshold, it indicates that the encoding / decoding transmission process may have introduced non-negligible, non-local quality impairments. In this case, estimation based solely on keyframes is insufficient to accurately reflect the overall quality status.

[0048] At this point, the receiving end will perform two actions. First, it will transmit the decoded video data to the cloud. Second, it will send a video transmission command to the remote sending end; the video transmission command is a specific control signal or message that notifies and instructs the sending end to perform subsequent coordinated operations.

[0049] Upon receiving the video transmission instruction, the sending end immediately responds by sending the original video data to be processed to the cloud. At this point, the cloud server has collected the original unencoded video from the sending end and the decoded video from the receiving end—the complete data from the input and output of the processing flow.

[0050] Subsequently, the cloud-based system performs quality loss calculations based on a pre-defined full-data computation model, using both the received decoded video and the video to be processed. The pre-defined full-data computation model refers to a quality assessment model or method with higher computational accuracy but also higher resource consumption. Unlike previous lightweight models that only operated on keyframe features, this typically requires precise frame-by-frame, or even pixel-by-pixel (or vertex-by-vertex) comparisons of two complete video sequences. For example, calculating the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) between two video sequences, or calculating the mean square error of vertex coordinates for a 3D mesh sequence. Finally, a full-data quality loss assessment result is output; because it is calculated based on all data, the full-data quality loss assessment result represents the most accurate and comprehensive quantification of quality loss during the entire video sequence's encoding, decoding, and transmission process.

[0051] Ultimately, the meaning of the quality loss assessment result varies depending on the processing path: when the quality is good, it is a preliminary quality loss assessment result calculated by the receiving end, representing the global estimate; when the quality may deteriorate significantly, it is a precise and reliable full quality loss assessment result calculated by the cloud. This allows the system to meet the needs of efficient daily monitoring while providing indisputable accurate measurements at critical moments.

[0052] In one possible implementation of this disclosure, the receiving end is specifically configured as follows: In response to the second keyframe selection algorithm being a fixed-interval frame extraction, the average value of at least one quality loss score is calculated to obtain the estimated quality loss assessment result. In response to the second keyframe selection algorithm extracting frames by keypoint displacement distance, a weighted average of at least one quality loss score is calculated based on the keyframe duration of the second keyframe to obtain the estimated quality loss assessment result. The second keyframe selection algorithm extracts frames based on the residual of the rendered image, determines the target weight based on the motion distance of key points in the second keyframe and the pixel value of the rendered image, and calculates a weighted average of at least one quality loss score based on the target weight to obtain the estimated quality loss assessment result.

[0053] In the embodiments of this disclosure, the processing logic of the receiving end is at least divided into the following cases: 1. When the second keyframe selection algorithm is detected to be using fixed-interval frame extraction, it means that the selected keyframes are uniformly distributed along the timeline. Each keyframe represents a time segment of equal duration, and their importance or representativeness is statistically similar. In this case, the average of the quality loss calculated from all keyframes can be taken as the global quality loss. That is, the preset pre-estimation algorithm is triggered to perform an average calculation on the quality loss scores of all calculated keyframes. This involves adding all scores and dividing by the total number of keyframes to obtain the arithmetic mean, which is then used as the final estimated quality loss assessment result.

[0054] 2. When the second keyframe selection algorithm is detected to extract frames based on keypoint displacement distance, it means that the keyframe selection is based on the moment of motion change, and the duration of the video segments represented by each keyframe may vary. One keyframe may represent a long period of stillness, while another may represent a short period of intense movement. Therefore, simply calculating the arithmetic mean is no longer reasonable. At this point, the preset prediction algorithm is triggered to calculate a weighted average. The weight of each quality loss score is determined by the keyframe duration of its corresponding second keyframe; the keyframe duration refers to the length of the video segment represented by that keyframe until the next keyframe appears. The longer the duration, the greater the weight of the keyframe's score in the overall evaluation. Through time-weighted averaging, the final predicted quality loss assessment result better reflects the actual contribution of segments of different durations to the overall video quality perception, and the prediction result is more accurate.

[0055] Specifically, the interval between two keyframes can be regarded as a tiny motion duration. When calculating the global quality loss, a weighted average is used, which can be implemented using, but is not limited to, the following formula (1): Formula (1) in, For the first The quality loss is calculated for each keyframe. For the first The duration of each keyframe.

[0056] 3. When the second keyframe selection algorithm is detected to be extracting frames based on the residual of the rendered image, more diverse influencing factors need to be introduced. At this time, the preset prediction algorithm is triggered to calculate a weighted average based on the target weight. The target weight is a comprehensive weight coefficient, the value of which is determined by at least the following parameters: one is the motion distance of the keypoints within each keyframe, reflecting the magnitude of the action; the other is the pixel value of the rendered image derived from the residual image (usually referring to the overall intensity or energy of the residual image, such as the sum or square of the absolute values ​​of all pixel differences), reflecting the severity of the visual changes between frames. The greater the motion distance or the higher the residual pixel value, the more important the information change is usually contained, and the greater the impact of its quality loss on the overall visual perception may be, so the calculated target weight value is also higher. The target weight of each keyframe is calculated based on this parameter, and the weight is used to calculate the weighted average of their respective quality loss scores, finally obtaining a more refined and visually sensitive predicted quality loss assessment result.

[0057] That is, based on the weighted average calculation, parameters such as the motion distance of key points and the pixel value of the residual image are introduced as weights when estimating the global quality loss. For example, if the motion distance between a key point in one key frame and its corresponding point in the previous key frame is greater than the motion distance between a key point in another key frame and its corresponding point in the previous key frame, then the weight of that key frame and related frames is increased.

[0058] This disclosure ensures that the most reasonable global quality inference can be derived from a subset of samples under different application scenarios and video content characteristics through an intelligent prediction mechanism that adapts to the front-end selection algorithm.

[0059] In one possible implementation of this disclosure, the receiving end is further configured to perform video quality assessment on the decoded video using a preset quality assessment model to obtain a video quality assessment result, wherein the video quality assessment result includes at least a grid sequence quality assessment, a rendered video quality assessment, and / or a device display assessment.

[0060] In the embodiments of this disclosure, the objective quality of the video directly affects the user's actual experience. This quality is directly influenced by objective factors such as the grid sequence itself, the quality and parameters of the rendering algorithm, and the display device. Therefore, in addition to calculating and evaluating the quality loss caused by the encoding / decoding and transmission processes, the objective quality of the final rendered video (i.e., the decoded video) is also evaluated.

[0061] A pre-defined quality assessment model is a pre-defined and trained set of analytical frameworks or algorithms used for multi-faceted, automated quality analysis of decoded video. Decoded video refers to a dynamic mesh video sequence that has undergone complete encoding / decoding, transmission, and reconstruction at the receiving end, ready for rendering or in the process of rendering. Evaluating this final output provides insight into the comprehensive impact of each stage in the entire chain—from raw data generation to encoding / compression and decoding / rendering—on the end-user experience. Furthermore, the video quality assessment result is not a single score, but a multi-dimensional evaluation set, providing a comprehensive quality diagnostic report.

[0062] Video quality assessment results include, but are not limited to, the following: 1. Grid sequence quality assessment: To evaluate this metric, each frame's mesh model can be treated as a static mesh, and a no-reference static mesh model quality evaluation method can be used to identify structural problems such as holes and burrs in the model. Once the objective quality of the mesh model structure for each frame is obtained, methods such as voting or averaging can be used to obtain the overall mesh sequence quality of the dynamic mesh video.

[0063] In addition to static mesh quality assessment of individual frames, a temporal-GNN structure can also be used: the static mesh of each frame in the dynamic mesh video is treated as a graph structure, the video is treated as a sequence containing temporal relationships, and the temporal-GNN structure is used to perform an objective quality score regression on the overall video.

[0064] 2. Evaluation of rendered video quality: To evaluate this metric, the mesh model is rendered according to the rendering parameters to obtain a 2D video. A video quality evaluation algorithm is then used to identify the video's perceived quality. By rendering the mesh sequence into a video, it's also possible to identify objective issues such as sudden motion changes and screen flickering, thus more accurately simulating the quality of actual viewing for the user.

[0065] 3. Equipment demonstration and evaluation: This indicator needs to be evaluated comprehensively in conjunction with the first two indicators. Since users' actual devices are not necessarily the same, their resolutions, imaging principles, and other aspects vary significantly. Therefore, when displaying on a specific device, operations such as edge cropping based on resolution or distortion based on imaging principles may be performed. Thus, when objectively assessing the quality of the first two indicators, appropriate processing must be applied based on the display device.

[0066] Meanwhile, if network transmission problems such as network interruptions or lag occur in actual business operations, resulting in incomplete data reception and a large number of missing values ​​or content in the rendered video, these issues can also be detected through objective quality identification of the rendered video.

[0067] In essence, mesh sequence quality assessment focuses on the structural integrity and geometric quality of the 3D mesh model itself within each frame of the decoded video. It eliminates the influence of rendering and display devices, directly examining the intrinsic properties of the 3D data. The assessment may include detecting geometric defects such as non-manifold structures, voids (unexpected missing areas on the model surface), and burrs (unsmooth surfaces with jagged or sharp unnatural protrusions), as well as evaluating the uniformity of mesh patch distribution and topological correctness. This assessment aims to ensure that the 3D model itself is of high quality and error-free.

[0068] For rendered video quality assessment, the perspective shifts from the raw 3D data to its 2D visual presentation. This involves synthesizing each frame of a dynamic mesh sequence into a 2D image based on a set of standard or specified rendering parameters (such as lighting models, material maps, and camera viewpoints), thus forming a 2D video stream. Subsequently, mature video quality assessment algorithms are used to analyze this rendered 2D video. The assessment includes, but is not limited to, overall image sharpness, color fidelity, the presence of motion abrupt changes (inconsistent or jumping motion), image flicker (high-frequency unstable changes in brightness or color), and other visual artifacts that may affect viewing comfort. This dimension directly simulates the final image quality observed by the user.

[0069] Device performance evaluation recognizes that high-quality raw data and rendered videos do not always guarantee a superior end-user experience, as the final presentation is heavily constrained by the characteristics of the display device. This evaluation requires considering the physical characteristics of the specific terminal device (e.g., mobile phone screen, high-definition display) (e.g., resolution, color gamut, brightness range, pixel arrangement) and imaging principles (e.g., distortion caused by lenses). For example, imperfections that are difficult to detect on a high-resolution device may become apparent on a low-resolution device; or content optimized for one screen may exhibit color deviations or cropped edges on another. Device performance evaluation involves adjusting and interpreting the evaluation results from the first two dimensions in a device-specific way, predicting the actual performance of content on a specific terminal, thereby achieving true end-to-end quality control.

[0070] This disclosure provides a crucial technical basis for achieving comprehensive user experience assurance through multi-dimensional video quality assessment, focusing not only on fidelity (quality loss) during data transmission but also, more importantly, on the objective quality level of the final product.

[0071] In one possible implementation of this disclosure, the receiving end is further configured to send the video quality assessment result to the cloud; In the cloud, it is also configured to generate parameter adjustment information based on the video quality assessment results and the full quality loss assessment results, and send the parameter adjustment information to the receiving end and the sending end. The parameter adjustment information includes encoder parameter adjustment information and decoder parameter adjustment information. The sending end is also configured to adjust the encoder parameters based on parameter adjustment information from the cloud. The receiving end is also configured to adjust the decoder parameters based on parameter adjustment information from the cloud.

[0072] In the embodiments of this disclosure, after completing the multi-dimensional quality analysis of the decoded video, the receiving end uploads the comprehensive video quality assessment results (including multiple dimensions such as grid sequence quality, rendered video quality, and device display assessment) to the cloud. This provides the cloud with detailed data on the final quality status as perceived from the user's perspective.

[0073] The cloud receives video quality assessment results from the receiving end and combines them with its own previously obtained full-scale quality loss assessment results obtained through computationally intensive processing (which precisely quantifies the distortion introduced by the encoding / decoding transmission link). It then performs comprehensive analysis, reasoning, and decision-making to ultimately generate parameter adjustment information. This parameter adjustment information is a collection of specific, actionable instructions or suggested values, designed to guide the sending and receiving ends on how to adjust their encoding / decoding behavior to improve overall quality. This information is explicitly divided into two parts: encoder parameter adjustment information (for the sending end) and decoder parameter adjustment information (for the receiving end). For example, encoder parameter adjustment information might include adjusting compression ratio, keyframe interval, bitrate, etc.; decoder parameter adjustment information might involve error concealment strategies, post-processing filtering strength, etc. The decision logic for generating this information could be: if the full-scale quality loss is large but the final rendering quality is acceptable, then appropriately reducing the encoding precision can be attempted to save bandwidth; if the Quality of Experience (QoE) index is poor and the final quality is unsatisfactory, then the encoding bitrate should be reduced first to decrease latency.

[0074] Subsequently, the cloud sends the generated parameter adjustment information to both the sending and receiving ends.

[0075] Upon receiving parameter adjustment information from the cloud, the sending end immediately adjusts the encoder parameters of its internally configured encoder based on this information. This means that the sending end no longer operates with fixed parameters, but can dynamically reconfigure its encoder's operating parameters according to instructions fed back from the cloud, such as adjusting the intensity of the compression algorithm, changing the inter-frame prediction structure, or updating the bitrate control strategy.

[0076] Correspondingly, upon receiving the parameter adjustment information, the receiving end will also perform decoder parameter adjustment processing on its internally configured decoder based on this information. This allows the receiving end to adaptively adjust its decoding strategy, such as enhancing error recovery capabilities or enabling different post-processing filters, to better process the received video stream, compensate for transmission losses, and thus further improve the reconstruction quality in the terminal presentation stage.

[0077] Specifically, it can be explained in, but is not limited to, the following ways: At the sending end, after the acquisition and reconstruction of the video to be processed is completed, encoding, keyframe selection, and keyframe feature extraction are performed. The outputs of these processes are then transmitted to the receiving end via the cloud. At the receiving end, decoding and rendering, keyframe feature extraction of the decoded video, and two quality assessments are performed.

[0078] This ensures efficient utilization of computing resources at each end while maintaining a controllable size of the transmitted information. Simultaneously, at the receiving end, the global loss of keyframe feature prediction is compared with a threshold. If the threshold is exceeded, a full-scale quality loss evaluation is required. This evaluation involves point-by-point calculation of existing frames, requiring the sending end to transmit the original unencoded data to the cloud server. The encoding and decoding steps can be completed on the cloud server, yielding the resulting quality loss. This process requires significant computing resources and is therefore performed on the cloud server.

[0079] In addition to quality loss, the evaluation also includes the objective quality assessment of the decoded video at the receiving end; quality evaluation indicators such as QoE and network transmission quality assessment can be sent back to the cloud server. Based on the results of each quality assessment, the cloud server determines whether the encoding and decoding work meets the requirements under the current network environment. If it does not meet business requirements, the parameters of the encoding and decoding algorithm can be adjusted according to the results of each quality assessment. For example, if the decoded video quality meets business requirements, but the QoE indicator does not, the encoding and decoding precision can be appropriately reduced to decrease transmission time, thereby reducing latency while ensuring presentation quality.

[0080] By assessing the quality and providing feedback to the codec to adjust hyperparameters, it can be ensured that the final quality of the overall workflow meets service requirements.

[0081] Furthermore, to facilitate understanding of the system architecture and system interaction flow of the embodiments of this disclosure, the embodiments of this disclosure also provide a schematic diagram of the overall process of video quality assessment, such as... Figure 2 As shown, edge A is the transmitter and edge B is the receiver.

[0082] While encoding the dynamic mesh model sequence at edge A, a keyframe selection algorithm is used to select keyframes in the video stream and extract their features. During transmission, in addition to encoding the video stream, the keyframes / keyframe features are also transmitted.

[0083] On the final display device (edge ​​B), the full video stream is decoded and displayed for normal business operations. Simultaneously, a dynamic grid sequence quality evaluation model is deployed on edge B to evaluate the objective quality of the final displayed video and model sequence.

[0084] Furthermore, at node B, keyframe features are extracted using the same feature extraction algorithm as at node A after the video stream is decoded. The cloud also transmits the original video stream keyframes and keyframe features from node A to node B, and performs quality loss calculations based on the keyframes and features at node B.

[0085] Simultaneously, the system sets a threshold for keyframe quality loss. When calculating the quality loss of the real-time video stream at edge B, the result is compared with this threshold. If the quality loss is not greater than this threshold, it indicates that the quality loss is within an acceptable range, and the quality loss of the keyframes is used to estimate the quality loss of the video stream throughout the entire encoding, decoding, and transmission process. Conversely, if it exceeds this threshold, it indicates that the dynamic mesh video quality loss in the current encoding and transmission environment (decoding at the display end) is significant, and the quality loss of the entire video stream needs to be calculated. The calculation of the full quality loss is performed on a high-performance cloud server.

[0086] For the video decoded and rendered at edge B, this invention also performs an objective quality evaluation. After calculating the quality loss of the entire video stream, the quality assessment results—both the objective quality evaluation and the quality loss calculation—can be fed back to the cloud and edge A, and the hyperparameters of the encoding / decoding algorithm in the current environment can be adjusted to ensure that the final decoded dynamic mesh video meets quality specifications.

[0087] Figure 3 This is a schematic flowchart of a video quality assessment method provided in an embodiment of the present disclosure.

[0088] like Figure 3 As shown, this method is applied to Figure 1 The sending end of the video quality assessment system includes the following steps: Step 301: Extract at least one first keyframe from the video to be processed, extract the first keyframe features of the first keyframe, and encode the video to be processed to obtain the encoded video.

[0089] Step 302: The encoded video, the first keyframe, and the first keyframe features are forwarded to the receiving end via the cloud so that the receiving end can perform a quality loss assessment based on the encoded video, the first keyframe, and the first keyframe features.

[0090] In one possible implementation of the present disclosure, when extracting at least one first keyframe from the video to be processed, it can be implemented in the following ways, but not limited to: extracting at least one first keyframe from the video to be processed by a first keyframe selection algorithm, wherein the first keyframe selection algorithm includes at least one of fixed-interval frame extraction, frame extraction based on keypoint displacement distance, or frame extraction based on rendered image residual.

[0091] In one possible implementation of this disclosure, the following methods may also be used, but are not limited to: in response to a video transmission command sent by the receiving end, sending the video to be processed to the cloud; receiving parameter adjustment information from the cloud, and performing encoder parameter adjustment processing on the configured encoder.

[0092] For details on the specific implementation process of the embodiments of this disclosure, please refer to the detailed description in the above video quality assessment system, and therefore will not be repeated here.

[0093] Figure 4 This is a schematic flowchart of a video quality assessment method provided in an embodiment of the present disclosure.

[0094] like Figure 4 As shown, this method is applied to Figure 1 The receiving end of the video quality assessment system includes the following steps: Step 401: Receive the encoded video, the first keyframe, and the first keyframe features forwarded by the sending end via the cloud.

[0095] Step 402: Decode the encoded video to obtain the decoded video, extract at least one second keyframe corresponding to the first keyframe from the decoded video, and extract the second keyframe features of the second keyframe.

[0096] Step 403: Perform quality loss assessment based on the first keyframe features and the second keyframe features to obtain the quality loss assessment result. The method of extracting the second keyframe and the second keyframe features is the same as the method of extracting the first keyframe and the first keyframe features.

[0097] In one possible implementation of this disclosure, when performing quality loss assessment based on the first keyframe features and the second keyframe features to obtain a quality loss assessment result, the following methods can be used, but are not limited to: inputting at least one first keyframe feature and the second keyframe feature into a preset quality loss model for quality loss assessment processing to obtain at least one quality loss score; in response to at least one quality loss score being less than a preset quality loss threshold, performing quality loss prediction processing based on at least one quality loss score using a preset prediction method to obtain a predicted quality loss assessment result, wherein the preset prediction method is determined based on a second keyframe selection algorithm.

[0098] In one possible implementation of this disclosure, after obtaining at least one quality loss score, the following methods may also be used, but are not limited to: in response to the fact that a first preset number of quality loss scores are not less than a preset quality loss threshold among the at least one quality loss score, the decoded video is transmitted to the cloud so that the cloud can perform quality loss calculation processing based on the decoded video; a video transmission instruction is sent to the sending end so that the sending end can transmit the corresponding video to be processed to the cloud in response to the video transmission instruction.

[0099] In one possible implementation of this disclosure, when performing quality loss prediction processing based on at least one quality loss score to obtain a predicted quality loss assessment result, the following methods can be used, but are not limited to: In response to the second keyframe selection algorithm being a fixed-interval frame extraction, the average value of at least one quality loss score is calculated to obtain a predicted quality loss assessment result. The second keyframe selection algorithm is an algorithm for extracting second keyframes, and includes at least one of fixed-interval frame extraction, frame extraction based on keypoint displacement distance, or frame extraction based on rendered image residuals. In response to the second keyframe selection algorithm being a keypoint displacement distance frame extraction, a weighted average calculation is performed on at least one quality loss score based on the keyframe duration of the second keyframe to obtain a predicted quality loss assessment result. In response to the second keyframe selection algorithm being a frame extraction based on rendered image residuals, a target weight is determined based on the keypoint movement distance in the second keyframe and the rendered image pixel value, and a weighted average calculation is performed on at least one quality loss score based on the target weight to obtain a predicted quality loss assessment result.

[0100] In one possible implementation of this disclosure, the following methods may also be used, but are not limited to: performing video quality assessment on the decoded video using a preset quality assessment model to obtain video quality assessment results, wherein the video quality assessment results include at least grid sequence quality assessment, rendered video quality assessment, and / or device display assessment.

[0101] In one possible implementation of this disclosure, the following method may also be used, but is not limited to: in response to at least one quality loss score having a first preset number of quality loss scores not less than a preset quality loss threshold, sending the video quality assessment result to the cloud; receiving parameter adjustment information generated by the cloud based on the video quality assessment result, and performing decoder parameter adjustment processing on the configured decoder according to the parameter adjustment information.

[0102] For details on the specific implementation process of the embodiments of this disclosure, please refer to the detailed description in the above video quality assessment system, and therefore will not be repeated here.

[0103] Figure 5This is a schematic flowchart of a video quality assessment method provided in an embodiment of the present disclosure.

[0104] like Figure 5 As shown, this method is applied to Figure 1 The cloud-based video quality assessment system includes the following steps: Step 501: Receive the encoded video, the first keyframe, and the first keyframe features sent by the sending end, and forward the encoded video, the first keyframe, and the first keyframe features to the receiving end.

[0105] Step 502: Receive the decoded video from the receiving end and the video to be processed from the sending end, wherein the decoded video is obtained by the receiving end decoding the encoded video.

[0106] Step 503: Based on the preset full-volume calculation model, perform quality loss calculation processing on the decoded video and the video to be processed to obtain the full-volume quality loss assessment result.

[0107] In one possible implementation of this disclosure, the following method may also be used, but is not limited to: receiving video quality assessment results from the receiving end, and generating parameter adjustment information based on the video quality assessment results and the full quality loss assessment results; sending the parameter adjustment information to the receiving end and the sending end, wherein the parameter adjustment information includes encoder parameter adjustment information and decoder parameter adjustment information.

[0108] For details on the specific implementation process of the embodiments of this disclosure, please refer to the detailed description in the above video quality assessment system, and therefore will not be repeated here.

[0109] In summary, the embodiments of this disclosure can achieve the following technical effects: 1. This disclosure extracts keyframes and their features from the original video (i.e., the video to be processed) at the sending end, and sends them together with the encoded video to the receiving end via the cloud. The receiving end obtains the corresponding keyframes and their features from the decoded video using the same extraction method, and then evaluates the quality loss by comparing the keyframe features at both ends. By performing quality evaluation based on keyframe features rather than the full video data, the resource consumption for computation and transmission is significantly reduced. At the same time, since the quality loss evaluation covers the entire encoding, decoding, and transmission process, the video quality evaluation results are more comprehensive and objective.

[0110] 2. This disclosure changes the calculation of encoding and decoding quality loss from the full video stream to the calculation of key frames and features, which effectively reduces the resource burden and improves the overall computational efficiency of quality assessment.

[0111] 3. This disclosure further improves the efficiency of quality assessment by setting a quality loss threshold and estimating the quality loss of the entire video stream based on the quality loss of keyframes. If the quality loss exceeds the threshold and quality loss calculation must be performed on all frames, both the sending and receiving ends transmit the required data to a high-performance cloud server. This ensures that normal decoding and display services on the receiving end are not affected by computing resource limitations when a large amount of computing resources are required.

[0112] 4. This disclosure assesses quality from two aspects: quality loss assessment and video quality assessment. This ensures both the quality of encoding and decoding transmission and the quality of the product presented to the user, making the quality assessment results more objective.

[0113] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0114] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0115] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 602 or a computer program loaded from storage unit 608 into RAM (Random Access Memory) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. I / O (Input / Output) interface 605 is also connected to bus 604.

[0116] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0117] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as video quality assessment methods. For example, in some embodiments, the video quality assessment method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the aforementioned video quality assessment method by any other suitable means (e.g., by means of firmware).

[0118] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0119] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0120] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0121] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0122] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0123] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0124] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0125] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0126] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video quality assessment system, characterized in that, include: Sender, receiver, and cloud The sending end is configured to extract at least one first keyframe from the video to be processed, extract the first keyframe features of the first keyframe, and encode the video to be processed to obtain an encoded video. The cloud is configured to receive the encoded video, the first keyframe, and the first keyframe features from the sending end, and forward them to the receiving end; The receiving end is configured to decode the encoded video to obtain the decoded video, extract at least one second keyframe corresponding to the first keyframe from the decoded video, and extract the second keyframe features of the second keyframe. The receiving end is further configured to perform quality loss assessment based on the first keyframe features and the second keyframe features to obtain a quality loss assessment result, wherein the method of extracting the second keyframe and the second keyframe features is the same as the method of extracting the first keyframe and the first keyframe features.

2. The video quality assessment system according to claim 1, characterized in that, The sending end is specifically configured to extract at least one first keyframe from the video to be processed using a first keyframe selection algorithm. The receiving end is specifically configured to extract at least one second keyframe corresponding to the first keyframe from the decoded video according to a second keyframe selection algorithm that is the same as the first keyframe selection algorithm. The first keyframe selection algorithm and the second keyframe selection algorithm include at least one of the following: extracting frames at fixed intervals, extracting frames based on the displacement distance of key points, or extracting frames based on the residual of the rendered image.

3. The video quality assessment system according to claim 1, characterized in that, The sending end is specifically configured to extract the first keyframe features of the first keyframe based on the first feature extraction algorithm; The receiving end is specifically configured to extract the second keyframe features of the second keyframe based on a second feature extraction algorithm that is the same as the first feature extraction algorithm.

4. The video quality assessment system according to claim 2, characterized in that, The receiving end is specifically configured as follows: At least one of the first keyframe features and the second keyframe feature are input into a preset quality loss model for quality loss evaluation to obtain at least one quality loss score. In response to the fact that at least one quality loss score is less than a preset quality loss threshold, a quality loss prediction process is performed based on the at least one quality loss score using a preset prediction method to obtain a predicted quality loss assessment result, wherein the preset prediction method is determined based on the second keyframe selection algorithm.

5. The video quality assessment system according to claim 4, characterized in that, The receiving end is further configured to transmit the decoded video to the cloud and send a video transmission instruction to the sending end in response to the fact that there is a first preset number of quality loss scores that are not less than the preset quality loss threshold among the at least one quality loss score. The sending end is also configured to send the video to be processed to the cloud in response to the video transmission command; The cloud platform is also configured to perform quality loss calculation based on the decoded video and the video to be processed according to a preset full-volume calculation model to obtain a full-volume quality loss assessment result, wherein the quality loss assessment result includes the estimated quality loss assessment result or the full-volume quality loss assessment result.

6. The video quality assessment system according to claim 4, characterized in that, The receiving end is specifically configured as follows: In response to the second keyframe selection algorithm extracting frames at fixed intervals, the average value of the at least one quality loss score is calculated to obtain the estimated quality loss assessment result. In response to the second keyframe selection algorithm extracting the keypoint displacement distance frame, a weighted average calculation is performed on the at least one quality loss score based on the keyframe duration of the second keyframe to obtain the estimated quality loss assessment result. In response to the second keyframe selection algorithm, which extracts frames based on the residual of the rendered image, determines the target weight based on the motion distance of key points in the second keyframe and the pixel value of the rendered image, and calculates a weighted average of the at least one quality loss score based on the target weight to obtain the estimated quality loss assessment result.

7. The video quality assessment system according to claim 5, characterized in that, The receiving end is further configured to perform video quality assessment on the decoded video using a preset quality assessment model to obtain video quality assessment results, wherein the video quality assessment results include at least grid sequence quality assessment, rendered video quality assessment, and / or device display assessment.

8. The video quality assessment system according to claim 7, characterized in that, The receiving end is also configured to send the video quality assessment result to the cloud; The cloud is also configured to generate parameter adjustment information based on the video quality assessment results and the full quality loss assessment results, and send the parameter adjustment information to the receiving end and the sending end, wherein the parameter adjustment information includes encoder parameter adjustment information and decoder parameter adjustment information; The transmitting end is also configured to perform encoder parameter adjustment processing on the configured encoder based on the parameter adjustment information from the cloud. The receiving end is also configured to perform decoder parameter adjustment processing on the configured decoder based on the parameter adjustment information from the cloud.

9. A video quality assessment method, characterized in that, The method is applied to the transmitting end of the video quality assessment system as described in any one of claims 1-8, comprising: Extract at least one first keyframe from the video to be processed, extract the first keyframe features of the first keyframe, and encode the video to be processed to obtain an encoded video. The encoded video, the first keyframe, and the first keyframe features are forwarded to the receiving end via the cloud so that the receiving end can perform a quality loss assessment based on the encoded video, the first keyframe, and the first keyframe features.

10. The video quality assessment method according to claim 9, characterized in that, The step of extracting at least one first keyframe from the video to be processed includes: At least one first keyframe is extracted from the video to be processed by a first keyframe selection algorithm, wherein the first keyframe selection algorithm includes at least one of fixed-interval frame extraction, frame extraction based on keypoint displacement distance, or frame extraction based on rendered image residual.

11. The video quality assessment method according to claim 9, characterized in that, The method further includes: In response to the video transmission command sent by the receiving end, the video to be processed is sent to the cloud. Receive parameter adjustment information from the cloud and perform encoder parameter adjustment processing on the configured encoder.

12. A video quality assessment method, characterized in that, The method is applied to the receiving end of the video quality assessment system as described in any one of claims 1-8, comprising: The received end forwards the encoded video, the first keyframe, and the features of the first keyframe via the cloud. The encoded video is decoded to obtain a decoded video. At least one second keyframe corresponding to the first keyframe is extracted from the decoded video, and the second keyframe features of the second keyframe are extracted. Quality loss assessment is performed based on the first keyframe features and the second keyframe features to obtain quality loss assessment results. The method of extracting the second keyframe and the second keyframe features is the same as the method of extracting the first keyframe and the first keyframe features.

13. The video quality assessment method according to claim 12, characterized in that, The quality loss assessment based on the first keyframe features and the second keyframe features, and the resulting quality loss assessment, includes: At least one of the first keyframe features and the second keyframe feature are input into a preset quality loss model for quality loss evaluation to obtain at least one quality loss score. In response to the fact that at least one quality loss score is less than a preset quality loss threshold, a quality loss prediction process is performed based on the at least one quality loss score using a preset prediction method to obtain a predicted quality loss assessment result. The preset prediction method is determined based on a second keyframe selection algorithm, which is an algorithm used to extract the second keyframe.

14. The video quality assessment method according to claim 13, characterized in that, After inputting at least one of the first keyframe features and the second keyframe features into a preset quality loss model for quality loss evaluation to obtain at least one quality loss score, the method further includes: In response to the fact that a first preset number of quality loss scores are not less than the preset quality loss threshold among the at least one quality loss score, the decoded video is transmitted to the cloud so that the cloud can perform quality loss calculation processing based on the decoded video. A video transmission command is sent to the sending end, so that the sending end responds to the video transmission command and transmits the corresponding video to be processed to the cloud.

15. The video quality assessment method according to claim 13, characterized in that, The process of performing quality loss prediction based on the preset prediction method according to the at least one quality loss score to obtain the predicted quality loss assessment result includes: In response to the second keyframe selection algorithm being a fixed-interval frame extraction, the average value of the at least one quality loss score is calculated to obtain the estimated quality loss assessment result, wherein the second keyframe selection algorithm includes at least one of the fixed-interval frame extraction, frame extraction based on keypoint displacement distance, or frame extraction based on rendered image residual. In response to the second keyframe selection algorithm extracting the keypoint displacement distance frame, a weighted average calculation is performed on the at least one quality loss score based on the keyframe duration of the second keyframe to obtain the estimated quality loss assessment result. In response to the second keyframe selection algorithm, which extracts frames based on the residual of the rendered image, determines the target weight based on the motion distance of key points in the second keyframe and the pixel value of the rendered image, and calculates a weighted average of the at least one quality loss score based on the target weight to obtain the estimated quality loss assessment result.

16. The video quality assessment method according to claim 12, characterized in that, The method further includes: The decoded video is evaluated using a preset quality assessment model to obtain video quality assessment results, wherein the video quality assessment results include at least grid sequence quality assessment, rendered video quality assessment and / or device display assessment. Send the video quality assessment results to the cloud; The system receives parameter adjustment information generated by the cloud based on the video quality assessment results, and adjusts the decoder parameters of the configured decoder according to the parameter adjustment information.

17. A video quality assessment method, characterized in that, The method is applied in the cloud within the video quality assessment system as described in any one of claims 1-8, comprising: The receiver receives the encoded video, the first keyframe, and the first keyframe features sent by the sending end, and forwards the encoded video, the first keyframe, and the first keyframe features to the receiving end. The receiver receives a decoded video from the receiving end and a video to be processed from the sending end, wherein the decoded video is obtained by the receiving end decoding the encoded video. Based on a preset full-volume calculation model, quality loss calculation is performed on the decoded video and the video to be processed to obtain the full-volume quality loss assessment result.

18. The video quality assessment method according to claim 17, characterized in that, The method further includes: Receive video quality assessment results from the receiving end, and generate parameter adjustment information based on the video quality assessment results and the full-volume quality loss assessment results; The parameter adjustment information is sent to the receiving end and the sending end, wherein the parameter adjustment information includes encoder parameter adjustment information and decoder parameter adjustment information.

19. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 9-11, any one of claims 12-16, or any one of claims 17-18.

20. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 9-11, any one of claims 12-16, or any one of claims 17-18.

21. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 9-11, any one of claims 12-16, or any one of claims 17-18.

Images