A method and apparatus for evaluating the quality of an image or video based on approximations, a method and apparatus for training a first model, electronic equipment, a storage medium, and a computer program.

Abstract

The present invention relates to technical fields such as artificial intelligence computer vision (image) or machine learning, and provides a method and related device for evaluating the quality of an image or video based on an approximation. The method uses an online first model to calculate a first approximation that approximates the subjective truth value of the sample to be evaluated based on acquired parameters of the sample to be evaluated, the first model being a model obtained based on an offline second model, the second model being a model obtained using k training samples and the subjective truth values ​​of the k training samples as a training set, the first model being a model obtained by fitting the parameters of the k training samples to the approximation values ​​of the k training samples obtained using the second model, where k is a positive integer, and evaluating the quality of the image or video based on the first approximation. This method can evaluate the quality of the image or video based on the approximation that approximates the subjective truth value fed back in real time, without increasing the cost of server-side hardware and while ensuring the accuracy of the evaluation.

Description

Technical Field

[0001] This application claims priority based on a Chinese patent application with an application number of 202110395015.2 filed on April 13, 2021, and an invention title of "Method and Related Apparatus for Evaluating Image or Video Quality Based on Approximation Values", and incorporates all its contents into the present invention by reference.

[0002] Embodiments of the present invention relate to the technical fields of computer vision (image) of artificial intelligence or machine learning, etc., and specifically relate to evaluating the quality of an image or video based on approximation values.

Background Art

[0003] The quality of an image / video can generally calculate the quality index of the video / image using an algorithm model.

[0004] Generally, the subjective scoring model and calculated value of a video can feedback the subjective quality situation of the video, but it is not ideal in terms of computational complexity and accuracy. For example, simply using the quantization parameter (QP) of the encoder, Peak Signal-to-Noise Ratio (PSNR), and Structural SIMilarity (SSIM) can feedback the objective quality score of a game video. This method has low complexity, but has a low correlation with subjective quality evaluation, has certain limitations, and introduces a certain delay effect. Also, an objective quality score of a game video can be obtained using a model such as Video Multimethod Assessment Fusion (VAMF). This method has high accuracy for PSNR, but has a very high computational complexity and cannot achieve real-time calculation of high frame rate and high resolution videos.

[0005] Furthermore, because cloud gaming encompasses a wide variety of applications, it is necessary to acquire subjective video scores in real time from large amounts of video data. However, the relevant technologies lack models and schemes for providing real-time feedback on subjective quality.

[0006] Therefore, in this field, there is a need for a method that can evaluate the quality of an image or video based on subjective quality feedback in real time. [Overview of the project]

[0007] The present invention provides a method and related apparatus for evaluating the quality of an image or video based on approximate values, which can evaluate the quality of an image or video based on approximate values ​​that approximate subjective truth values ​​fed back in real time, without increasing the cost of server-side hardware and ensuring the accuracy of the evaluation.

[0008] One aspect of the present invention provides a method for evaluating the quality of an image or video based on an approximation, performed by an electronic device having data processing capabilities, comprising: acquiring a sample to be evaluated, which includes a video or image to be evaluated; calculating a first approximation that approximates the subjective truth value of the sample to be evaluated using a first online model based on the parameters of the sample to be evaluated, wherein the first model is a model obtained based on a second offline model, the second model is a model obtained using k training samples and the subjective truth values ​​of the k training samples as a training set, the subjective truth values ​​of the k training samples are obtained in a subjective scoring manner, and the first model is a model obtained by fitting the parameters of the k training samples with reference to the approximation of the k training samples obtained using the second model, where k is a positive integer; and evaluating the quality of the video or image to be evaluated based on the first approximation.

[0009] In another aspect of the present invention, there is a device for evaluating the quality of an image or video based on an approximation, which is deployed in an electronic device having data processing capabilities, and the device includes: an acquisition unit that acquires a sample to be evaluated, which includes a video or image to be evaluated; a calculation unit that calculates a first approximation that approximates the subjective truth value of the sample to be evaluated using an online first model based on the parameters of the sample to be evaluated, wherein the first model is a model obtained based on an offline second model, the second model is a model obtained as a training set of k training samples and the subjective truth values ​​of the k training samples, the subjective truth values ​​of the k training samples are obtained in a subjective scoring manner, and the first model is a model obtained by fitting the parameters of the k training samples with reference to the approximation of the k training samples obtained using the second model, where k is a positive integer; and an evaluation unit that evaluates the quality of the video or image to be evaluated based on the first approximation.

[0010] Another aspect of the present invention provides a method for training a first model, performed by an electronic device having data processing capabilities, comprising: acquiring k training samples, wherein the subjective truth values ​​of the k training samples are acquired in a subjective scoring manner; acquiring a second model using the k training samples and the subjective truth values ​​of the k training samples as a training set; acquiring approximations that approximate the subjective truth values ​​of the k training samples using the second model, with the k training samples as input; and acquiring the first model by fitting the parameters of the k training samples with reference to the approximations of the k training samples.

[0011] In another aspect of the present invention, a training device for a first model, deployed in an electronic device having data processing capabilities, is provided, comprising: a first acquisition unit that acquires k training samples, wherein the subjective truth values ​​of the k training samples are acquired by a subjective scoring method; a first training unit that acquires a second model using the k training samples and the subjective truth values ​​of the k training samples as a training set; a second acquisition unit that takes the k training samples as input and acquires approximate values ​​that approximate the subjective truth values ​​of the k training samples using the second model; and a second training unit that uses the approximate values ​​of the k training samples as a reference and fits the parameters of the k training samples to acquire the first model.

[0012] In another aspect of the present invention, an electronic device is provided which includes a processor capable of executing a computer program and a computer-readable storage medium in which the computer program is stored, wherein the computer program, when executed by the processor, implements a method for evaluating the quality of an image or video based on the above-mentioned approximations or a method for training the above-mentioned first model.

[0013] In another embodiment of the present invention, a computer-readable storage medium for storing a computer program is provided, wherein the computer program causes the computer to perform a method for evaluating the quality of an image or video based on the above-mentioned approximations or a method for training the above-mentioned first model.

[0014] In another embodiment of the present invention, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions stored on a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, thereby causing the computer device to perform a method for evaluating the quality of an image or video based on the above approximations or a method for training the above first model.

[0015] In embodiments of the present invention, based on the parameters of the sample to be evaluated, a first approximation that approximates the subjective truth value of the sample to be evaluated is calculated using an online first model, and the quality of the video or image to be evaluated is further evaluated based on the first approximation. With the online first model, the first approximation can be calculated and fed back in real time, so the quality of the image or video can be evaluated based on the approximation that approximates the subjective truth value that is fed back in real time. Furthermore, the first model is a model obtained based on an offline second model, the second model is a model obtained using k training samples and the subjective truth values ​​of the k training samples as a training set, the subjective truth values ​​of the k training samples are obtained using a subjective scoring method, and the first model is a model obtained by fitting the parameters of the k training samples with reference to the approximation of the k training samples obtained using the second model, where k is a positive integer. In other words, by using the second model obtained through training to acquire the first model through fitting, the accuracy of the evaluation of the first model can be ensured without increasing the cost of the server-side hardware.

[0016] As described above, the method according to the present invention can evaluate the quality of an image or video based on an approximate value that approximates a subjective truth value that is fed back in real time, without increasing the cost of server-side hardware and ensuring the accuracy of the evaluation. [Brief explanation of the drawing]

[0017] To more clearly explain the technology relating to the embodiments of the present invention, the following is a brief introduction of the drawings necessary for describing the embodiments. Note that the following drawings are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these without any creative work. [Figure 1]This is a schematic diagram of the interface of a subjective scoring platform according to an embodiment of the present invention. [Figure 2] This is a schematic flowchart of a method for training a quality evaluation model based on randomly selected samples according to an embodiment of the present invention. [Figure 3] This is a schematic flowchart of a method for evaluating the quality of an image or video based on approximate values ​​according to an embodiment of the present invention. [Figure 4] This is a schematic block diagram of the operating principle of the first model according to an embodiment of the present invention. [Figure 5] This is a schematic block diagram of the training and evaluation principles of the first model according to an embodiment of the present invention. [Figure 6] This is a schematic block diagram of the optimization principle of the first model according to an embodiment of the present invention. [Figure 7] This is a schematic block diagram of a service system including a first model according to an embodiment of the present invention. [Figure 8] This is a schematic flowchart of the training method for the first model according to an embodiment of the present invention. [Figure 9] This is a schematic block diagram of an apparatus for evaluating the quality of an image or video based on approximate values ​​according to an embodiment of the present invention. [Figure 10] This is a schematic block diagram of a training device for a first model according to an embodiment of the present invention. [Figure 11] This is a schematic block diagram of an electronic device according to an embodiment of the present invention. [Modes for carrying out the invention]

[0018] The following describes the art in embodiments of the present invention clearly and completely with reference to the drawings of the embodiments, but it is clear that the embodiments described are only some embodiments of the present invention and not all embodiments. All other embodiments obtained by those skilled in the art without creative work based on embodiments of the present invention are within the scope of the present invention.

[0019] The technology related to the present invention may relate to artificial intelligence technology.

[0020] Here, artificial intelligence (AI) is a theory, method, technology, and application system that simulates and extends human intelligence, senses the environment, acquires knowledge, and uses knowledge to obtain optimal results by using a digital computer or a machine controlled by a digital computer. In other words, artificial intelligence is an integrated technology of computer science, aiming to understand the essence of intelligence and produce new intelligent machines that react in the same way as human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, and endows machines with the functions of perception, reasoning, and decision-making.

[0021] Note that artificial intelligence technology is an integrated discipline with a wide range of fields, including both hardware-level technologies and software-level technologies. Artificial intelligence basic technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interactive systems, mechatronics, etc. Artificial intelligence software technology mainly includes several aspects such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0022] With the research and progress of artificial intelligence technology, artificial intelligence technology has carried out research and applications in many fields. For example, in general smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, driverless, autonomous driving, drones, robots, intelligent healthcare, intelligent customer service, etc. With the development of technology, it is believed that artificial intelligence technology will be applied in more fields and play an increasingly important value.

[0023] Embodiments of the present invention may relate to computer vision (CV) technology in artificial intelligence technology. Computer vision is the science of studying how machines "see," and more specifically, it means using cameras and computers as substitutes for human eyes to perform machine vision such as target recognition, tracking, and measurement, as well as graphic processing, further adapting computer processing to human eye observation, or transmitting detected images to instruments. As a scientific field, it studies theories and technologies related to computer vision and aims to build artificial intelligence systems that can acquire information from images and multidimensional data. Computer vision technology generally includes technologies such as image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / action recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, synchronous localization, and map building, and further includes biofeature recognition technologies such as general face recognition and fingerprint recognition.

[0024] Embodiments of the present invention may relate to machine learning (ML) in artificial intelligence technology. ML is a multidisciplinary, interdisciplinary field encompassing various disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It focuses on studying how computers can simulate or replicate human learning behavior, acquire new knowledge or skills, reorganize existing knowledge structures, and constantly improve their own performance. Machine learning is at the heart of artificial intelligence, the fundamental route to giving intelligence to computers, and its applications extend to all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, trust networks, reinforcement learning, transition learning, inductive learning, and demonstration learning.

[0025] To facilitate the scheme of the present invention, the following will explain terms related to the present invention.

[0026] (1) Image or video quality evaluation: This means sensing, measuring, and evaluating distortion in an image or video frame in a subjective or objective manner. Subjective scoring methods are generally expressed as mean opinion score (MOS) or difference mean opinion score (DMOS), and are also called subjective scoring methods. Objective scoring methods generally employ algorithmic models to calculate video / image quality indicators, and are also called methods that use quality evaluation models to output quality scores.

[0027] (2) Subjective scoring platform: A labeling platform that provides subjective scores for images and videos. Subjective scoring means that evaluators / labelers score an image or video based on its quality, aesthetics, etc.

[0028] Figure 1 is a schematic diagram of the interface of a subjective scoring platform according to an embodiment of the present invention.

[0029] As shown in Figure 1, the interface of a subjective scoring platform may include videos to be scored and scoring options. For example, in a 5-point scoring system, the scoring options would correspond to five levels: "Excellent," "Good," "Average," "Poor," and "Very Poor." A single video to be scored is typically scored by multiple evaluators, each selecting a scoring level, and the final quality score for the video is obtained by averaging the scores of all evaluators. Other subjective scoring methods may also be employed, such as pairing comparison, where evaluators view two images or videos and choose which is better.

[0030] (3) Active learning: A subjective scoring platform selects a portion of training samples from a large image and video library and provides them to evaluators for subjective scoring. The process by which evaluators subjectively score training samples to obtain subjective scores is called obtaining subjective scores for training samples using a subjective scoring method, and samples to which subjective scores are labeled are called labeled samples. Active learning can proactively select training samples that are the most difficult for the current model to distinguish or contain the most information, based on a specific selection policy, and provide them to evaluators for scoring. This ensures the performance of the model while efficiently reducing the amount of samples that need to be labeled.

[0031] (4) Passive learning: The model is trained using randomly selected samples.

[0032] Figure 2 is a schematic flowchart of a method for training a quality evaluation model based on randomly selected samples according to an embodiment of the present invention.

[0033] As shown in Figure 2, the training process for the quality assessment model is a "waterfall" algorithm development process, in which several (n) training samples are randomly selected from a large database, provided to a subjective scoring platform for evaluation by evaluators (i.e., evaluator 1 to evaluator t), and then the model is trained after the evaluation. In this method of training quality assessment models, the random selection of samples makes it easy to select many low-value training samples, especially in large image and video libraries where there is a lot of similar and redundant data. Also, the number of training samples to be selected must be predetermined and is not easily controlled. Furthermore, because subjective scoring and model training are completely separated, in the "waterfall" development process, if the quality of the dataset after subjective scoring is not high, scoring is required again, which is time-consuming and labor-intensive, and the error tolerance rate is low.

[0034] (5) The mean opinion score (MOS) is the final quality score of the training sample described above, and the specific value of this score may be obtained based on the average of the scores of all evaluators. For example, the subjective truth value according to the present invention may be the MOS.

[0035] (6) Fitting: Fitting is the process of connecting a series of points on a plane with a smooth curve. Since there are countless possibilities for the connected curve, there are various fitting methods. The fitted curve can generally be represented as a function, and different fitting names are used depending on the function. Common fitting methods include least-squares curve fitting. When the function to be determined is linear, it is called linear fitting or linear regression (mainly in statistics); otherwise, it is called nonlinear fitting or nonlinear regression. The expression may also be a segment function, in which case it is called spline fitting. For example, the first model according to the present invention may be a model obtained by fitting. The prediction result of this first model may be an approximation that approximates the subjective truth value.

[0036] (7) Training: A model can be obtained by training using a dataset of images / videos after subjective scoring as input. For example, the second model according to the present invention may be a model obtained through training. The prediction result of this second model may be an approximation that approximates the subjective truth value.

[0037] (8) Image Quality Assessment (IQA): This is one of the fundamental techniques in image processing, and is mainly studied through the analysis of image characteristics to evaluate the quality of images (image distortion level). Image quality assessment plays an important role in image processing systems, such as algorithm analysis comparison and system performance evaluation. In recent years, with the widespread research in the field of digital images, research on image quality assessment has attracted increasing attention from researchers, and many image quality assessment indicators and methods have been proposed and improved.

[0038] (9) Video Quality Assessment (VQA): This is one of the fundamental techniques in video processing, and it mainly involves studying the characteristics of video to evaluate the quality of the video (objective quality of the video).

[0039] Generally, subjective video scoring models and calculations can provide feedback on the subjective quality of a video, but they are not ideal in terms of computational complexity and accuracy. For example, objective quality scores for game videos can be fed back by simply using encoder quantization parameters (QP), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM). This method is not complex, but it has low correlation with subjective quality evaluations, is subject to certain limitations, and introduces a certain delay effect. Alternatively, objective quality scores for game videos can be obtained using models such as Video Multimethod Assessment Fusion (VAMF). This method is highly accurate for PSNR, but its computational complexity is very high, and it cannot achieve real-time calculation of high frame rate, high-resolution video.

[0040] Furthermore, because cloud gaming encompasses a wide variety of applications, it is necessary to acquire subjective video scores in real time from large amounts of video data. However, the relevant technologies lack models and schemes for providing real-time feedback on subjective quality.

[0041] Accordingly, the present invention provides a method for evaluating the quality of an image or video based on an approximate value that approximates a subjective truth value fed back in real time, without increasing the cost of server-side hardware and ensuring the accuracy of the evaluation.

[0042] Figure 3 is a schematic flowchart of method 100 for evaluating image or video quality based on approximations according to an embodiment of the present invention. The technology according to the embodiment of the present invention may be implemented by any electronic device having data processing capabilities. For example, the electronic device may function as a server. The server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms. The servers may be connected directly or indirectly by wired or wireless communication methods, and the present invention is not limited to these. For convenience of explanation, a service system will be described below as an example.

[0043] As shown in Figure 3, the method 100 may include some or all of the following steps.

[0044] Step S110: Obtain a sample to be evaluated, which includes a video or image to be evaluated.

[0045] Step S120: Based on the parameters of the sample to be evaluated, a first approximation is calculated using an online first model to approximate the subjective truth value of the sample to be evaluated.

[0046] Here, the first model is a model obtained based on an offline second model, the second model is a model obtained using k training samples and the subjective truth values ​​of the k training samples as the training set, the subjective truth values ​​of the k training samples are obtained using a subjective scoring method, and the first model is a model obtained by fitting the parameters of the k training samples with reference to the approximate values ​​of the k training samples obtained using the second model, where k is a positive integer.

[0047] Step S130: The quality of the video or image to be evaluated is evaluated based on the first approximation.

[0048] In cloud game execution mode, the game runs on the server side, and then the rendered game video screen is captured using a screen capture method, compressed by a video encoder, and then transmitted to the user client over the network. Based on this, the user client can evaluate the quality of the image or video based on an approximation that approximates the subjective truth value fed back in real time, using a method that evaluates the quality of the image or video based on an approximation.

[0049] In embodiments of the present invention, based on the parameters of the sample to be evaluated, a first approximation that approximates the subjective truth value of the sample to be evaluated is calculated using an online first model, and the quality of the video or image to be evaluated is further evaluated based on the first approximation. With the online first model, the first approximation can be calculated and fed back in real time, so the quality of the image or video can be evaluated based on the approximation that approximates the subjective truth value that is fed back in real time. Furthermore, the first model is a model obtained based on an offline second model, the second model is a model obtained using k training samples and the subjective truth values ​​of the k training samples as a training set, the subjective truth values ​​of the k training samples are obtained using a subjective scoring method, and the first model is a model obtained by fitting the parameters of the k training samples with reference to the approximation of the k training samples obtained using the second model, where k is a positive integer. In other words, by using the second model obtained through training to acquire the first model through fitting, the accuracy of the evaluation of the first model can be ensured without increasing the cost of the server-side hardware.

[0050] As described above, the method according to the present invention can evaluate the quality of an image or video based on an approximate value that approximates the subjective truth value, which is fed back in real time, without increasing the cost of server-side hardware and ensuring the accuracy of the evaluation. In other words, an approximate value that approximates the subjective truth value of the sample to be evaluated can be calculated online by a first model, and the video quality of a cloud game can be evaluated using the approximate value calculated by the first model. That is, the subjective quality of the video can be evaluated in real time while playing an online game. For example, the overall quality of a cloud game can be monitored online.

[0051] In the embodiments of the present invention, the user may use a set of cloud game video quality evaluation schemes similar to the application scenario on the user client to obtain subjective truth values ​​in a subjective scoring manner as training samples for the second model. For example, the interface of a subjective scoring platform as shown in Figure 1 may be adopted, or the flow as shown in Figure 2 may be adopted to train the second model based on randomly selected samples, but the present invention is not limited to these.

[0052] In some embodiments, the parameters of the sample to be evaluated include at least one of the following: feedback parameters of a network module, configuration parameters of a cloud game module, and computation parameters of an encoding / decoding module. In one embodiment, the type of parameters of the sample to be evaluated may be defined based on a classification of Software Development Kits (SDKs). For example, an SDK may include, but is not limited to, a network SDK, a cloud game SDK, and an encoding / decoding SDK.

[0053] Figure 4 is a schematic block diagram of the operating principle of the first model according to an embodiment of the present invention.

[0054] As shown in Figure 4, the first model takes parameters 1 to n of the input sample to be evaluated (for example, parameters are selected according to a scenario, which includes, but is not limited to, the encoding parameters of the sample to be evaluated, such as the encoded frame parameters and those related to the code stream, such as the quantization parameter (QP), MV, frame rate, frame length, frame complexity parameter, and frame type. The frame complexity parameter may be, for example, the Sum of Absolute Transformed Difference (SATD). The frame rate means the number of images played per second, for example, 24 frames means 24 images are played per second, 60 frames means 60 images are played per second, and so on. The frame length means the length of the data frame). By calculating the input parameters 1 to n, a first approximation can be obtained that approximates the subjective truth value of the sample to be evaluated.

[0055] In some embodiments, based on the parameters of the sample to be evaluated, the method 100 may further include the following steps prior to step S120.

[0056] Based on the parameters of the k training samples, the first model is used to calculate approximate values ​​of the k training samples. The first model is evaluated based on the approximate values ​​of the k training samples obtained using the second model and the approximate values ​​of the k training samples calculated using the first model. For each of the k training samples, if the difference between the approximate value calculated using the first model and the approximate value obtained using the second model is less than or equal to a first predetermined threshold, the evaluation result of the first model is determined to be positive; otherwise, the evaluation result of the first model is determined to be negative. For example, if the difference between the approximate value calculated using the first model and the approximate value obtained using the second model for any one of the k training samples is greater than the first predetermined threshold, the evaluation result of the first model is determined to be negative.

[0057] Simply put, the performance or accuracy of the first model may be evaluated using the approximate values ​​obtained from the second model.

[0058] In some embodiments, based on the parameters of the sample to be evaluated, the method 100 may further include the following steps prior to step S120.

[0059] The first model is evaluated based on the subjective truth values ​​of the k training samples and the approximate values ​​of the k training samples calculated using the first model. For each of the k training samples, if the difference between the approximate value calculated using the first model and the subjective truth value is less than or equal to a second predetermined threshold, the evaluation result of the first model is determined to be positive; otherwise, the evaluation result of the first model is determined to be negative. For example, if the difference between the approximate value calculated using the first model and the subjective truth value for any one of the k training samples is greater than the second predetermined threshold, the evaluation result of the first model is determined to be negative.

[0060] Simply put, the performance or accuracy of the first model may be evaluated using subjective truth values.

[0061] Figure 5 is a schematic block diagram of the training and evaluation principles of the first model according to an embodiment of the present invention.

[0062] As shown in Figure 5, the fitting process for the first model involves obtaining subjective truth values ​​A of image frames in the decoded sequence set using a subjective scoring method, as indicated by the dashed arrows in Figure 5. The obtained subjective truth values ​​A are used as a training set for training the second model, and the first model is obtained based on the second model and the set of parameters to be input. The optimization process for the first model may involve evaluating the first model based on the approximate subjective truth value B output by the second model and the approximate subjective truth value C output by the first model, as indicated by the solid arrows, or it may involve evaluating the first model based on subjective truth value A and the approximate subjective truth value C output by the first model.

[0063] The training process for the second model may include the following steps:

[0064] (1) Obtain the decoded sequence set.

[0065] Because cloud games come in many varieties, they differ in complexity and subjective perception. The present invention may perform scenario classification on cloud game videos so that a second submodel in a second model and a first submodel in a first model can be trained for a single scenario. Furthermore, the scenario-classified cloud game videos may be subdivided based on game scenes so that a second submodel in a second model and a first submodel in a first model can be trained for a single game scene in a single scenario. Embodiments of the present invention are not limited to specific classifications of scenarios and game scenes. For example, a scenario may be the type of device for playing a video or image to be evaluated. A game scene may be a scene from the game screen, such as a combat scene or a non-combat scene. Game classification may be performed on cloud game videos, and a certain type of video classified (e.g., an acquired source sequence set) may be encoded and compressed using an encoding configuration scheme (adapted to the frame rate, code rate, resolution, etc., of the cloud game features) to obtain an encoded and compressed sequence set. The encoded and compressed sequence set performs lossy compression on the captured source sequence set, resulting in a loss of detail in the video image quality. The encoded and compressed sequence set is then decoded to obtain the decoded sequence set. Preferably, the decoded sequence set may be the k training samples mentioned above.

[0066] (2) The subjective truth value A of the image frames in the decoded sequence set is obtained using a subjective scoring method.

[0067] Because cloud gaming scenarios differ from subjective video scenarios in real-time communication, cloud gaming scenarios can relate to mobile terminals such as cell phones, fixed PCs, and television terminals. Furthermore, the player's requirements for the experience differ from those of users in real-time communication. In this invention, subjective truth values ​​A of image frames in a decoded sequence set may be obtained using a subjective scoring method, using a standard system for subjective evaluation of cloud game video. For example, an encoded and compressed sequence set is played back by decoding by a third party, and subjective truth values ​​A of image frames in the decoded sequence set are obtained manually (i.e., by an evaluator) according to a subjective scoring method, in accordance with a standard system for subjective evaluation of cloud game video.

[0068] (3) A second model is trained based on the subjective truth value A of the image frames in the decoded sequence set.

[0069] A sequence set may be created by a third party decoding an encoded and compressed sequence set, and a second model may be trained using this decoded sequence set as the training set for the second model. Using the trained second model, an approximate value B of the subjective truth value of the image frames in the decoded sequence set (approximating the subjective truth value A as closely as possible) may be obtained. The present invention does not require the performance complexity of the second model, but only relatively high accuracy; that is, the second model may be very complex and very accurate.

[0070] (4) Obtain the first model based on the second model.

[0071] After obtaining an approximate subjective truth value B for the image frames in the decoded sequence set using a trained second model, a fitting calculation may be performed based on the approximate subjective truth value B for the image frames in the decoded sequence set and the parameters of the image frames in the decoded sequence set in order to obtain a first model. Fitting is the process of connecting a series of points on a plane with a smooth curve. Since there are countless possibilities for the connected curve, there are various fitting methods. The fitted curve can generally be represented as a function, and different fitting names are used depending on the function. Common fitting methods include least-squares curve fitting. When the function to be determined is linear, it is called linear fitting or linear regression (mainly in statistics); otherwise, it is called nonlinear fitting or nonlinear regression. The expression may also be a segment function, in which case it is called spline fitting. For example, the first model according to the present invention may be a model obtained by fitting. The prediction result of this first model may be an approximation that approximates the subjective truth value.

[0072] In some embodiments, method 100 may further include the following steps:

[0073] If the evaluation result of the first model is positive, the first model is integrated into the service system. If the evaluation result of the first model is negative, the first model is refitted until the evaluation result of the first model becomes positive.

[0074] In other words, if the first model achieves the desired evaluation result, the first model may be integrated into the service system; if the first model does not achieve the desired result, the first model must be refitted until it achieves the desired result.

[0075] In some embodiments, method 100 may further include the following steps:

[0076] The statistical module reports a first approximation to the second model. The second model is used to obtain a second approximation of the sample to be evaluated. If the difference between the first and second approximations is greater than a third predetermined threshold, the first model is optimized using the parameters of the sample to be evaluated and the second approximation. If the difference between the first and second approximations is less than or equal to the third predetermined threshold, it is determined that there is no need to optimize the first model.

[0077] Simply put, the first approximation can be used to optimize the first model, thereby improving the accuracy of the first model.

[0078] In some embodiments, before deciding whether to optimize the first model based on the first and second approximations, subjective truth values ​​of the samples to be evaluated are obtained, and if the difference between the second approximation and the subjective truth values ​​of the samples to be evaluated is greater than a fourth predetermined threshold, the second model is optimized using the samples to be evaluated and their subjective truth values.

[0079] Simply put, the first approximation can be used to optimize the second model, thereby improving the accuracy of the second model.

[0080] Figure 6 is a schematic block diagram of the optimization principle of the first model according to an embodiment of the present invention.

[0081] As shown in Figure 6, first, a first model is obtained to calculate an approximation that approximates the subjective truth value, and then the first model is placed on the cloud game server system. Based on this, the ingestion frame (sample to be evaluated) is obtained, the ingestion frame is encoded, and the encoded frame is obtained. Then, by encoding the parameters and code stream information from the encoded frame, the parameters of the sample to be evaluated that need to be input into the first model are obtained, and after inputting the parameters of the sample to be evaluated into the first model, the first model performs calculations based on the parameters of the sample to be evaluated and obtains a first approximation of the sample to be evaluated. In this case, the first approximation may be fed back to the platform statistics module via the data statistics module, and the platform statistics module may decide whether or not to optimize the first model or the second model based on the first approximation.

[0082] Figure 7 is a schematic block diagram of a service system including a first model according to an embodiment of the present invention.

[0083] As shown in Figure 7, the fitted first model is integrated into the encoding / decoding module in the service system, and input parameters such as parameter P1, parameter P2, and parameter P3 can be obtained using the service system. Parameter P1 represents a feedback parameter of the network module, parameter P2 represents a configuration parameter of the cloud game module, and parameter P3 represents a computation parameter of the encoding / decoding module. Next, the first model calculates a first approximation of the sample to be evaluated based on parameters P1, P2, and P3. In one possible embodiment, the first approximation is reported to the platform statistics module of the service system via a data statistics module, and the platform statistics module may determine whether a second model and the first model need to be optimized based on the first approximation. For example, the platform statistics module may perform a statistical classification and determine whether a first or second submodel corresponding to the classification needs to be optimized. After optimizing the second model, the first model may be optimized based on the optimized second model, but the embodiments of the present invention are not specifically limited thereto. By optimizing the first model, the accuracy of the approximation can be improved.

[0084] In some embodiments, the first model includes a first submodel corresponding to each of a plurality of scenarios, the second model includes a second submodel corresponding to each of a plurality of scenarios, and the plurality of scenarios includes a first scenario in which the sample to be evaluated is located. Based on this, step S120 above may include the steps of determining a first submodel corresponding to a first scenario and calculating a first approximation using the first submodel corresponding to the first scenario based on the parameters of the sample to be evaluated.

[0085] In other words, different first submodels can be fitted based on the parameters of samples in different scenarios, and different second submodels can be obtained based on samples in different scenarios, depending on the training method. This allows for improvements in the accuracy of both the first and second submodels.

[0086] In some embodiments, multiple scenarios include types of devices for playing back videos or images to be evaluated.

[0087] In other alternative embodiments of the present invention, the multiple scenes may include an application program or playback mode to which the video or image to be evaluated belongs, but the present invention is not specifically limited to these.

[0088] In some embodiments, the first model is a semi-referenced or non-referenced model, and the second model is a fully referenced or non-referenced model, wherein the semi-referenced model is obtained by referencing some parameters in the uncompressed image frame and the compressed image frame, the non-referenced model is obtained by referencing only the compressed image frame, and the fully referenced model is obtained by referencing both the uncompressed and compressed image frames.

[0089] The semi-reference model may refer to a model obtained by referencing some parameters in the image frame before encoding and the image frame after encoding; the non-reference model may refer to a model obtained by referencing only the image frame after encoding; and the full-reference model may refer to a model obtained by referencing both the image frame before encoding and the image frame after encoding. However, the present invention is not specifically limited to these.

[0090] As described above, the present invention provides a method for evaluating the quality of an image or video based on an approximation, which can evaluate the subjective quality of the video in real time. Preferably, subjective truth values ​​A of k training samples may be obtained for a given classification by classifying information such as the type and platform of the cloud game. Next, an offline second model obtains an approximation B that approximates the subjective truth values ​​A of the K training samples (approximating A as closely as possible). Then, the obtained approximation B is used as the reference truth value, and the first model is fitted using multiple parameters of the input k training samples (used to calculate an approximation C that approximates the subjective truth value A, and the approximation C can be calculated quickly). This allows the first model to obtain an approximation that approximates the subjective truth values ​​of real-time video of the cloud game online. Preferably, the online first model can be optimized based on the updated second model by updating the second model for subsequently added games.

[0091] Figure 8 is a schematic flowchart of the training method 200 for the first model according to an embodiment of the present invention. The technology according to the embodiment of the present invention may be implemented by any electronic device having data processing capabilities. For example, the electronic device may be implemented as a server. The server may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms. The servers may be connected directly or indirectly by wired or wireless communication methods, and the present invention is not limited to these.

[0092] As shown in Figure 8, Method 200 may include the following steps.

[0093] Step S210: Obtain k training samples. The subjective truth values ​​of these k training samples are obtained using a subjective scoring method.

[0094] Step S220: A second model is obtained using the k training samples and the subjective truth values ​​of the k training samples as the training set.

[0095] Step S230: Using the k training samples as input, the second model is used to obtain approximate values ​​that approximate the subjective truth values ​​of the k training samples.

[0096] Step S240: The parameters of the k training samples are fitted to the k training samples using the k training samples as a reference to obtain the first model.

[0097] Method 200 may further include schemes related to the evaluation and optimization of the first model in Method 100. In other words, schemes related to the evaluation and optimization of the first model in Method 200 may refer to the corresponding schemes in Method 100, and to avoid duplication, their explanation is omitted here.

[0098] The above describes preferred embodiments of the present invention in detail with reference to the accompanying drawings. However, the present invention is not limited to the specific details of the above embodiments, and various simple modifications are possible within the technical scope of the present invention, within the technical spirit of the invention. For example, the individual specific technical features described in the above embodiments can be combined in any suitable way without contradiction, and to avoid unnecessary redundancy, various possible combinations are not described further in this specification. As another example, any combination is possible between various different embodiments of the present invention and will be considered the same as those disclosed by the present invention, as long as it does not contradict the spirit of the invention.

[0099] In the various embodiments of the methods described herein, the magnitude of the sequence number of the processes described above does not indicate the order of execution, and the execution order of each process should be determined by its function and underlying logic, and does not limit the implementation process of the embodiments described herein. Furthermore, in embodiments of the present invention, the term "and / or" refers only to a dependency relationship describing the recipient, and means that three types of relationships may exist. Specifically, "A and / or B" can represent three cases: A only exists, both A and B exist, or B only exists. In this specification, the symbol " / " generally means that the recipients before and after it are in an "or" relationship.

[0100] The above describes the method according to an embodiment of the present invention; the following describes the apparatus according to an embodiment of the present invention.

[0101] Figure 9 is a schematic block diagram of an apparatus 300 for evaluating the quality of an image or video based on approximate values ​​according to an embodiment of the present invention.

[0102] The acquisition unit 310 acquires a sample to be evaluated, which includes a video or image to be evaluated.

[0103] The calculation unit 320 calculates a first approximate value that approximates the subjective truth value of the sample to be evaluated using an online first model, based on the parameters of the sample to be evaluated.

[0104] Here, the first model is a model obtained based on the offline second model, the second model is a model obtained using k training samples and the subjective truth values ​​of the k training samples as the training set, the subjective truth values ​​of the k training samples are obtained using a subjective scoring method, and the first model is a model obtained by fitting the parameters of the k training samples using the approximate values ​​of the k training samples obtained using the second model as a reference, where k is a positive integer.

[0105] The evaluation unit 330 evaluates the quality of the video or image to be evaluated based on the first approximate value.

[0106] In some embodiments, the parameters of the sample to be evaluated include at least one of the following: feedback parameters of the network module, configuration parameters of the cloud game module, and computation parameters of the encoding / decoding module.

[0107] In some embodiments, before the calculation unit 320 calculates a first approximation that approximates the subjective truth value of the sample to be evaluated using an online first model based on the parameters of the sample to be evaluated, the evaluation unit 330 calculates approximations of the k training samples using the first model based on the parameters of the k training samples, evaluates the first model based on the approximations of the k training samples obtained using the second model and the approximations of the k training samples calculated using the first model, and determines that the evaluation result of the first model is positive if the difference between the approximation calculated using the first model and the approximation obtained using the second model for each of the k training samples is less than or equal to a first predetermined threshold, and otherwise determines that the evaluation result of the first model is negative.

[0108] In some embodiments, before the calculation unit 320 calculates a first approximation that approximates the subjective truth value of the sample to be evaluated using an online first model based on the parameters of the sample to be evaluated, the evaluation unit 330 evaluates the first model based on the subjective truth values ​​of k training samples and the approximation of k training samples calculated using the first model. For each of the k training samples, if the difference between the approximation calculated using the first model and the subjective truth value is less than or equal to a second predetermined threshold, the evaluation result of the first model is determined to be positive; otherwise, the evaluation result of the first model is determined to be negative.

[0109] In some embodiments, the evaluation unit 330 integrates the first model into the service system if the evaluation result of the first model is positive, and if the evaluation result of the first model is negative, it refits the first model until the evaluation result of the first model becomes positive.

[0110] In some embodiments, the evaluation unit 330 reports a first approximation to a second model using a statistical module, obtains a second approximation of the sample to be evaluated using the second model, optimizes the first model using the parameters of the sample to be evaluated and the second approximation if the difference between the first approximation and the second approximation is greater than a third predetermined threshold, and determines that there is no need to optimize the first model if the difference between the first approximation and the second approximation is less than or equal to the third predetermined threshold.

[0111] In some embodiments, before the evaluation unit 330 decides whether to optimize the first model based on the first approximation and the second approximation, the evaluation unit 330 obtains the subjective truth value of the sample to be evaluated, and if the difference between the second approximation and the subjective truth value of the sample to be evaluated is greater than a fourth predetermined threshold, the evaluation unit 330 optimizes the second model using the sample to be evaluated and the subjective truth value of the sample to be evaluated.

[0112] In some embodiments, the first model includes a first submodel corresponding to each of a plurality of scenarios, and the plurality of scenarios includes a first scenario in which the sample to be evaluated is located. The calculation unit 320 determines the first submodel corresponding to the first scenario and calculates a first approximation using the first submodel corresponding to the first scenario, based on the parameters of the sample to be evaluated.

[0113] In some embodiments, multiple scenarios include types of devices for playing back videos or images to be evaluated.

[0114] In some embodiments, the first model is a semi-referenced or non-referenced model, and the second model is a fully referenced or non-referenced model, wherein the semi-referenced model is obtained by referencing some parameters in the uncompressed image frame and the compressed image frame, the non-referenced model is obtained by referencing only the compressed image frame, and the fully referenced model is obtained by referencing both the uncompressed and compressed image frames.

[0115] Figure 10 is a schematic block diagram of a training device 400 of a first model according to an embodiment of the present invention.

[0116] As shown in Figure 10, the apparatus 400 may include the following parts.

[0117] The first acquisition unit 410 acquires k training samples. The subjective truth values ​​of the k training samples are acquired using a subjective scoring method.

[0118] The first training unit 420 acquires a second model using k training samples and the subjective truth values ​​of the k training samples as the training set.

[0119] The second acquisition unit 430 takes k training samples as input and uses the second model to acquire approximate values ​​that approximate the subjective truth values ​​of the k training samples.

[0120] The second training unit 440 obtains a first model by fitting the parameters of k training samples to k training samples, using the approximate values ​​of the k training samples as a reference.

[0121] The embodiments of the apparatus and the embodiments of the method can correspond to each other, and similar descriptions can be found in the embodiments of the method. To avoid redundancy, such descriptions are omitted here. Specifically, apparatus 300 can correspond to an entity that performs method 100 of the embodiments of the present invention, and each unit of apparatus 300 can be used to realize the corresponding flow of method 100. Similarly, apparatus 400 can correspond to an entity that performs method 200 of the embodiments of the present invention, and the units of apparatus 400 can be used to realize the flow of method 200. For the sake of brevity, such descriptions are omitted in this specification.

[0122] Furthermore, each unit of the video processing apparatus according to an embodiment of the present invention may be configured by integrating each or all of them into one or more additional units, or one of these units may be subdivided into a plurality of functionally smaller units, which can achieve similar operation without affecting the realization of the technical effects of the embodiment of the present invention. The above units are divided based on logical function, and in practice, the function of one unit may be realized by multiple units, or the function of multiple units may be realized by one unit. In other embodiments of the present invention, apparatus 300 or apparatus 400 may include other units, and in practice, these functions may be supported by other units or realized in cooperation with multiple units. According to another embodiment of the present invention, apparatus 300 or apparatus 400 according to an embodiment of the present invention is a general-purpose computer device including a general-purpose computer including processing elements and storage elements such as a central processing unit (CPU), random access storage medium (RAM), and read-only storage medium (ROM), and is constructed by executing a computer program (including program code) capable of executing the steps relating to each method, and can implement the method relating to an embodiment of the present invention or the training method of a first model for evaluating image video quality based on approximations. A computer program can be implemented by recording it on a computer-readable storage medium, loading it onto an electronic device via the computer-readable storage medium, and executing it there, thereby carrying out the corresponding method of the embodiment of the present invention.

[0123] In other words, the units described above may be implemented in hardware, in software, or in combination of software and hardware. In particular, the steps of the embodiments of the method in the embodiments of the present invention may be performed by integrated logic circuits in hardware and / or instructions in software form within a processor, and the steps of the method disclosed in connection with embodiments of the present invention may be directly embodied either by being performed by a hardware coding device or by a combination of hardware and software within a coding device. Preferably, the software may reside in a mature storage medium of the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically rewritable programmable memory, or registers. The storage medium is located in memory, and the processor reads information from the memory and combines it with its hardware to complete the steps of the embodiments of the method.

[0124] Figure 11 is a schematic block diagram of an electronic device 500 according to an embodiment of the present invention.

[0125] As shown in Figure 11, the electronic device 500 includes at least a processor 510 and a computer-readable storage medium 520. The processor 510 and the computer-readable storage medium 520 may be connected by a bus or other means. The computer-readable storage medium 520 is used to store a computer program 521 containing computer instructions, and the processor 510 executes the computer instructions stored in the computer-readable storage medium 520. The processor 510 is the computing core and control core of the electronic device 500 and can execute one or more computer instructions, specifically, it can load and execute one or more computer instructions to implement a corresponding method process or a corresponding function.

[0126] For example, the processor 510 may be referred to as a Central Processing Unit (CPU). The processor 510 may include, but is not limited to, a general-purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, an discrete gate or transistor logic device, a discrete hardware component, and the like.

[0127] For example, the computer-readable storage medium 520 may be a high-speed RAM memory, or it may be non-volatile memory such as at least one magnetic disk memory. Preferably, it may be at least one computer-readable storage medium located away from the processor 510. Specifically, the computer-readable storage medium 520 includes, but is not limited to, volatile memory and / or non-volatile memory. Here, the non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM) that functions as an external cache. While not limiting, many types of RAM are available, including, for example, static random access memory (Static RAM: SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (Synchronous DRAM: SDRAM), double data rate synchronous dynamic random access memory (DRAM: DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronously connected dynamic random access memory (synch-link DRAM: SLDRAM), and direct memory bus random access memory (Direct Rambus RAM: DR RAM).

[0128] In one embodiment, the electronic device 500 may be a device 300 that evaluates the quality of an image or video based on the approximations shown in Figure 9. Computer instructions are stored in the computer-readable storage medium 520. To perform the corresponding steps in the embodiment of the method shown in Figure 3, the computer instructions stored in the computer-readable storage medium 520 are loaded and executed by the processor 510. In a specific embodiment, the computer instructions in the computer-readable storage medium 520 are loaded by the processor 510 and the corresponding steps are executed, but to avoid redundancy, that description is omitted here.

[0129] In one embodiment, the electronic device 500 may be the training device 400 of the first model shown in Figure 10. Computer instructions are stored in the computer-readable storage medium 520. To perform the corresponding steps in the embodiment of the method shown in Figure 8, the computer instructions stored in the computer-readable storage medium 520 are loaded and executed by the processor 510. In a specific embodiment, the computer instructions in the computer-readable storage medium 520 are loaded by the processor 510 and the corresponding steps are executed, but to avoid redundancy, that description is omitted here.

[0130] In another embodiment of the present invention, the embodiment also provides a computer-readable storage medium (Memory) which is a storage device within an electronic device 500 for storing programs and data. For example, a computer-readable storage medium 520 may be provided. The computer-readable storage medium 520 herein may include an internal storage medium within the electronic device 500, or it may include an extended storage medium supported by the electronic device 500. The computer-readable storage medium provides a storage area for storing the operating system of the electronic device 500. Furthermore, the storage space stores one or more computer instructions configured to be loaded and executed by a processor 510, and these computer instructions may be one or more computer programs 521 (including program code).

[0131] In another embodiment of the present invention, a computer program product or computer program is provided which includes computer instructions stored in a computer-readable storage medium. For example, computer program 521 is provided. In this case, the electronic device 500 may have a processor 510 that reads computer instructions from the computer-readable storage medium 520, and the processor 510 executes the computer instructions to cause the computer to perform a method for evaluating the quality of an image or video based on approximations according to the various preferred embodiments described above, or a method for training a first model.

[0132] In other words, when implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process of an embodiment of the present invention is executed in whole or in part, or the functionality of an embodiment of the present invention is realized. The computer may be a general-purpose computer, a dedicated computer, a computer network, or other programmable device. The computer instructions may be stored on a computer-readable storage medium, or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired (coaxial cable, optical fiber, digital subscriber line, etc. (DSL)) or wireless (e.g., infrared, radio, microwave, etc.)

[0133] Those skilled in the art will recognize that the elements and process steps of the various examples described in relation to the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints. Those skilled in the art may use different methods for each specific application to implement the described functions, but such implementations should not be considered beyond the scope of the invention.

[0134] Finally, while the above describes only specific embodiments of the present invention, the scope of the present invention is not limited thereto, and is intended to be included within the scope of the present invention insofar as those skilled in the art can easily conceive of modifications or substitutions within the technical scope disclosed herein. Accordingly, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for evaluating the quality of an image or video based on an approximation, performed by an electronic device having data processing capabilities, The steps include obtaining a sample to be evaluated, which includes a video or image to be evaluated, A step of calculating a first approximation that approximates the MOS (mean opinion score) or DMOS (difference mean opinion score) of the sample to be evaluated using an online first model, based on the parameters of the sample to be evaluated, wherein the first model is a model obtained based on an offline second model, the first model includes a first submodel corresponding to each of a plurality of scenarios, the plurality of scenarios includes a first scenario in which the sample to be evaluated is located, the second model is a model obtained using k training samples and the MOS or DMOS of the k training samples as a training set, the second model includes a second submodel corresponding to each of the plurality of scenarios, the MOS or DMOS is a numerical value obtained by a subjective scoring method, and the first model is a model obtained by fitting the parameters of the k training samples with reference to the approximation of the k training samples obtained using the second model, where k is a positive integer, A method comprising: evaluating the quality of a video or image to be evaluated based on a first approximation calculated using a first submodel corresponding to a first scenario, wherein the first submodel corresponding to a first scenario is obtained based on a second submodel corresponding to a first scenario.

2. The method according to claim 1, wherein the parameters of the sample to be evaluated include at least one of the following: the feedback parameters of the network module, the configuration parameters of the cloud game module, and the computation parameters of the encoding / decoding module.

3. Before calculating a first approximate value that approximates the MOS or DMOS of the sample to be evaluated using an online first model based on the parameters of the sample to be evaluated, A step of calculating an approximate value of the k training samples using the first model based on the parameters of the k training samples, A step of evaluating the first model based on the approximate values ​​of the k training samples obtained using the second model and the approximate values ​​of the k training samples calculated using the first model, The method according to claim 1, further comprising the step of determining that the evaluation result of the first model is positive if, for each of the k training samples, the difference between the approximate value calculated using the first model and the approximate value obtained using the second model is less than or equal to a first predetermined threshold, and otherwise determining that the evaluation result of the first model is negative.

4. Before calculating a first approximate value that approximates the MOS or DMOS of the sample to be evaluated using an online first model based on the parameters of the sample to be evaluated, A step of evaluating the first model based on the MOS or DMOS of the k training samples and the approximate values ​​of the k training samples calculated using the first model, The method according to claim 1, further comprising the step of determining that the evaluation result of the first model is positive if, for each of the k training samples, the difference between the approximate value calculated using the first model and MOS or DMOS is less than or equal to a second predetermined threshold, and otherwise determining that the evaluation result of the first model is negative.

5. The method according to claim 3 or 4, further comprising the steps of: if the evaluation result of the first model is positive, integrating the first model into a service system; and if the evaluation result of the first model is negative, refitting the first model until the evaluation result of the first model becomes positive.

6. The steps include reporting the first approximate value to the second model using a statistical module, The steps include obtaining a second approximate value of the sample to be evaluated using the second model, If the difference between the first approximate value and the second approximate value is greater than a third predetermined threshold, the first model is optimized using the parameters of the sample to be evaluated and the second approximate value. The method according to claim 1, further comprising the step of determining that it is not necessary to optimize the first model if the difference between the first approximation and the second approximation is less than or equal to a third predetermined threshold.

7. If the difference between the first approximation and the second approximation is greater than a third predetermined threshold, before optimizing the first model using the parameters of the sample to be evaluated and the second approximation, The steps include: acquiring the MOS or DMOS of the sample to be evaluated; The method according to claim 6, further comprising the step of optimizing the second model using the sample to be evaluated and the MOS or DMOS of the sample to be evaluated if the difference between the second approximate value and the MOS or DMOS of the sample to be evaluated is greater than a fourth predetermined threshold.

8. The step of calculating a first approximate value that approximates the MOS or DMOS of the sample to be evaluated using an online first model based on the parameters of the sample to be evaluated is: The steps include determining the first submodel corresponding to the first scenario, The method according to claim 1, comprising the step of calculating a first approximation using a first submodel corresponding to the first scenario, based on the parameters of the sample to be evaluated.

9. The method according to claim 8, wherein the plurality of scenarios include the type of device for playing back the video or image to be evaluated.

10. The aforementioned first model is a semi-reference model or a non-reference model, The second model described above is either a fully referenced model or a non-referenced model. The aforementioned half-reference model is a model obtained by referencing some parameters in the image frame before compression and the image frame after compression. The aforementioned non-reference model is a model obtained by referencing only the compressed image frame, The method according to any one of claims 1 to 9, wherein the full reference model is a model obtained by referencing the image frame before compression and the image frame after compression.

11. A device for evaluating the quality of an image or video based on an approximate value, which is deployed in an electronic device having data processing capabilities, An acquisition unit that acquires a sample to be evaluated, which includes a video or image to be evaluated, A calculation unit that calculates a first approximate value that approximates the MOS (mean opinion score) or DMOS (difference mean opinion score) of the sample to be evaluated using an online first model based on the parameters of the sample to be evaluated, wherein the first model is a model obtained based on an offline second model, the first model includes a first submodel corresponding to each of a plurality of scenarios, the plurality of scenarios includes a first scenario in which the sample to be evaluated is located, the second model is a model obtained using k training samples and the MOS or DMOS of the k training samples as a training set, the second model includes a second submodel corresponding to each of the plurality of scenarios, the MOS or DMOS is a numerical value obtained by a subjective scoring method, and the first model is a model obtained by fitting the parameters of the k training samples with reference to the approximate value of the k training samples obtained using the second model, where k is a positive integer. Apparatus comprising: an evaluation unit for evaluating the quality of a video or image to be evaluated based on a first approximate value calculated using a first submodel corresponding to the first scenario, wherein the first submodel corresponding to the first scenario is obtained based on a second submodel corresponding to the first scenario.

12. A method for training a first model, which is performed by an electronic device having data processing capabilities, A step of acquiring k training samples, wherein the MOS (mean opinion score) or DMOS (difference mean opinion score) of the k training samples is a numerical value acquired using a subjective scoring method, and the step, A step of acquiring a second model using the k training samples and the MOS or DMOS of the k training samples as a training set, wherein the second model includes a second submodel corresponding to each of a plurality of scenarios, and the plurality of scenarios includes a first scenario in which the k training samples are located. The steps include taking the k training samples as input and using the second model to obtain an approximate value that approximates the MOS or DMOS of the k training samples, A method comprising the steps of: obtaining a first model by fitting the parameters of the k training samples with reference to approximations of the k training samples, wherein the first model includes a first submodel corresponding to each of the plurality of scenarios, and the first submodel corresponding to the first scenario is obtained based on the second submodel corresponding to the first scenario.

13. A first model training device, deployed on an electronic device having data processing capabilities, A first acquisition unit that acquires k training samples, wherein the MOS (mean opinion score) or DMOS (difference mean opinion score) of the k training samples is a numerical value acquired by a subjective scoring method, and the first acquisition unit and, A first training unit acquires a second model using the k training samples and the MOS or DMOS of the k training samples as a training set, wherein the second model includes a second submodel corresponding to each of a plurality of scenarios, and the plurality of scenarios includes the first scenario in which the k training samples are located. A second acquisition unit takes the k training samples as input and uses the second model to acquire approximate values ​​that approximate the MOS or DMOS of the k training samples, Apparatus comprising: a second training unit that obtains a first model by fitting the parameters of the k training samples with reference to approximations of the k training samples, wherein the first model includes a first submodel corresponding to each of the plurality of scenarios, and the first submodel corresponding to the first scenario is obtained based on the second submodel corresponding to the first scenario.

14. A processor capable of executing computer programs, A computer-readable storage medium on which a computer program is stored, The computer program, when executed by the processor, is an electronic device that implements a method for evaluating the quality of an image or video based on an approximation value according to any one of claims 1 to 10, or a method for training a first model according to claim 12.

15. A computer-readable storage medium for storing a computer program, wherein the computer program causes a computer to execute a method for evaluating the quality of an image or video based on an approximation according to any one of claims 1 to 10, or a method for training a first model according to claim 12.

16. A computer program for causing a computer to perform a method for evaluating the quality of an image or video based on an approximate value according to any one of claims 1 to 10, or a method for training a first model according to claim 12.

Images