Image processing method, apparatus, device, readable storage medium and program product

Abstract

Embodiments of the present application provide an image processing method, device, equipment, readable storage medium and program product, which can be applied to the fields or scenarios of cloud technology, artificial intelligence, intelligent platform, application software, video quality prediction, etc. The method comprises: obtaining encoding parameters of each to-be-processed video frame in a to-be-processed video frame set, the to-be-processed video frame set being a video frame set of a to-be-evaluated video; calling a first quality evaluation model to perform quality evaluation processing on the encoding parameters of each to-be-processed video frame respectively to obtain quality evaluation parameters of each to-be-processed video frame; and determining a quality evaluation result of the to-be-evaluated video according to the quality evaluation parameters of each to-be-processed video frame, wherein the first quality evaluation model is obtained by training in combination with a second quality evaluation model. Through the embodiments of the present application, the quality evaluation result of the to-be-evaluated video can be automatically predicted, and the accuracy and efficiency of video quality prediction are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to image processing methods, image processing apparatus, computer equipment, computer-readable storage media, and computer program products. Background Technology

[0002] With the continuous development and application of computer technology, the amount of video content on the internet is increasing, and the quality of this content greatly affects the viewing experience. Therefore, people are placing higher demands on the quality of video content. Currently, the quality of video content is evaluated using manual inspection methods. For example, this involves collecting partial image frames from a video, scoring the image quality of each frame, and then averaging the scores to obtain a predicted quality score for the video. However, this method relies on the scorer's inherent experience, leading to strong subjectivity and low accuracy and efficiency in video quality prediction. Therefore, improving the automation level, accuracy, and efficiency of video quality prediction is a pressing issue that needs to be addressed. Summary of the Invention

[0003] This application provides an image processing method, apparatus, device, readable storage medium, and program product that can automatically predict the quality assessment results of a video to be evaluated, thereby improving the accuracy and efficiency of video quality prediction.

[0004] In a first aspect, this application provides an image processing method, the method comprising:

[0005] Obtain the encoding parameters of each video frame to be processed in the set of video frames to be processed, wherein the set of video frames to be processed is the set of video frames of the video to be evaluated;

[0006] The first quality assessment model is invoked to perform quality assessment processing on the encoding parameters of each video frame to be processed, so as to obtain the quality assessment parameters of each video frame to be processed.

[0007] The quality assessment result of the video to be evaluated is determined based on the quality assessment parameters of each video frame to be processed.

[0008] The first quality assessment model is trained in conjunction with the second quality assessment model. During model training, the first quality assessment model is used to determine the first predicted quality assessment parameter of the first sample video frame based on the encoding parameters of the first sample video frame. The second quality assessment model is used to determine the second predicted quality assessment parameter of the first sample video frame based on the first sample video frame and its encoding parameters. The first quality assessment model is obtained by adjusting the model parameters using the target loss parameter determined based on the first and second predicted quality assessment parameters. The second quality assessment model is trained using the second sample video frame, its encoding parameters, and quality labels.

[0009] Secondly, this application provides an image processing apparatus, the apparatus comprising:

[0010] The acquisition module is used to acquire the encoding parameters of each video frame to be processed in the set of video frames to be processed, wherein the set of video frames to be processed is the set of video frames of the video to be evaluated.

[0011] The processing module is used to call the first quality assessment model to perform quality assessment processing on the encoding parameters of each video frame to be processed, so as to obtain the quality assessment parameters of each video frame to be processed.

[0012] The quality assessment module is used to determine the quality assessment result of the video to be assessed based on the quality assessment parameters of each video frame to be processed.

[0013] The first quality assessment model is trained in conjunction with the second quality assessment model. During model training, the first quality assessment model is used to determine the first predicted quality assessment parameter of the first sample video frame based on the encoding parameters of the first sample video frame. The second quality assessment model is used to determine the second predicted quality assessment parameter of the first sample video frame based on the first sample video frame and its encoding parameters. The first quality assessment model is obtained by adjusting the model parameters using the target loss parameter determined based on the first and second predicted quality assessment parameters. The second quality assessment model is trained using the second sample video frame, its encoding parameters, and quality labels.

[0014] Thirdly, this application provides a computer device, including: a processor, a storage device, and a communication interface, wherein the processor, the communication interface, and the storage device are interconnected, wherein the storage device stores executable program code, and the processor is used to call the executable program code to implement the image processing method described above.

[0015] Fourthly, this application provides a computer-readable storage medium storing a computer program, the computer program including program instructions that are executed by a processor to implement the image processing method described above.

[0016] Fifthly, this application provides a computer program product, which includes a computer program or computer instructions, which are executed by a processor to implement the image processing method described above.

[0017] This application obtains the encoding parameters of each video frame to be processed in the video to be evaluated, calls a first quality assessment model to perform quality assessment processing on the encoding parameters of each video frame to be processed, obtains the quality assessment parameters of each video frame to be processed, and then determines the quality assessment result of the video to be evaluated. By directly using the encoding parameters obtained during the encoding process of the video to be evaluated for quality assessment processing, without the need for additional processing such as decoding of the image frames of the video to be evaluated, the efficiency of video quality assessment is improved. Since the input information is simpler than the image, modeling can be completed by using some low-complexity models, which can achieve real-time image quality prediction without using GPU resources and only occupying a small amount of CPU resources, so as to facilitate video quality assessment in real-time scenarios such as cloud gaming, and can achieve real-time image quality assessment with very low computational cost. Compared with the method of manually inspecting the quality of video content, the method proposed in this application can automatically predict the quality assessment result of the video to be evaluated, improving the efficiency of video quality prediction. Meanwhile, the first quality assessment model in this application is obtained by adjusting the model parameters using the first predicted quality assessment parameters determined by the first quality assessment model based on the coding features, and the target loss parameters determined by the second predicted quality assessment parameters determined by the second quality assessment model based on the image features and coding features. By using the second predicted quality assessment parameters provided by the second quality assessment model, which has higher prediction accuracy and higher model complexity, as the supervision information for the first quality assessment model, the training effect of the first quality assessment model is improved, thereby improving the accuracy of video quality prediction. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the architecture of an image processing system provided in an exemplary embodiment of this application;

[0020] Figure 2 This is a schematic flowchart of an image processing method provided in an exemplary embodiment of this application;

[0021] Figure 3 This is a schematic diagram of a video quality assessment process provided in an exemplary embodiment of this application;

[0022] Figure 4 This is a schematic flowchart of another image processing method provided in an exemplary embodiment of this application;

[0023] Figure 5A This is a flowchart illustrating the training process of an image processing model provided in an exemplary embodiment of this application;

[0024] Figure 5B This is a flowchart illustrating the training process of a second quality assessment model provided in an exemplary embodiment of this application;

[0025] Figure 5C This is a flowchart illustrating the training process of a first quality assessment model provided in an exemplary embodiment of this application;

[0026] Figure 6 This is a schematic block diagram of an image processing apparatus provided in an exemplary embodiment of this application;

[0027] Figure 7 This is a schematic block diagram of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0029] It should be noted that the terms "first," "second," etc., used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature specified with "first" or "second" may explicitly or implicitly include at least one of those features.

[0030] The embodiments of this application can be applied to various fields or scenarios such as cloud computing, cloud IoT, cloud gaming, artificial intelligence, vehicle scenarios, smart transportation, assisted driving, and video quality assessment. Several typical fields or scenarios will be introduced below.

[0031] Cloud computing refers to the delivery and usage model of IT infrastructure, meaning obtaining necessary resources in an on-demand and easily scalable manner through a network. In a broader sense, cloud computing also refers to the delivery and usage model of services, meaning obtaining necessary services in an on-demand and easily scalable manner through a network. These services can be IT and software related, internet-related, or other services. Cloud computing is a product of the convergence and development of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing. Driven by the development of the internet, real-time data streams, the diversification of connected devices, and the demands of search services, social networks, mobile commerce, and open collaboration, cloud computing has developed rapidly. Unlike previous parallel and distributed computing, the emergence of cloud computing will fundamentally revolutionize the entire internet model and enterprise management model. This application can store data such as encoding parameters and videos to be evaluated on a cloud server. When these different data are needed, they can be directly retrieved from the cloud server, greatly improving the data retrieval speed.

[0032] Cloud IoT aims to connect the information sensed and commands received by traditional IoT devices to the Internet, truly achieving networking. It also enables massive data storage and computation through cloud computing technology. Due to the nature of IoT, which involves connecting things to each other and sensing the current operating status of each "object" in real time, a large amount of data information is generated in this process. How to aggregate this information and how to sift out useful information from the massive amount of data to support decision-making for future development have become key issues affecting the development of IoT. As a result, IoT cloud based on cloud computing and cloud storage technology has become a powerful support for IoT technology and applications.

[0033] Cloud gaming, also known as gaming on demand, is an online gaming technology based on cloud computing. It enables thin clients with relatively limited graphics processing and data processing capabilities to run high-quality games. In cloud gaming, the game does not reside on the player's terminal but runs on a cloud server. The cloud server renders the game scene as a video and audio stream, which is then transmitted to the player's terminal via the network. The player's terminal does not need powerful graphics processing and data processing capabilities; it only needs basic streaming media playback capabilities and the ability to receive player input commands and send them to the cloud server. This application applies to cloud gaming by using the encoding parameters of the cloud gaming host to perform real-time image quality detection. This allows the server to quickly obtain the image quality status of each cloud gaming host, enabling targeted image enhancement and quality compensation to improve the player's gaming experience.

[0034] Intelligent Traffic Systems (ITS), also known as Intelligent Transportation Systems, effectively integrate advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) into transportation, service control, and vehicle manufacturing. This strengthens the connection between vehicles, roads, and users, thereby forming a comprehensive transportation system that ensures safety, improves efficiency, enhances the environment, and saves energy.

[0035] Intelligent Vehicle Infrastructure Cooperative Systems (IVICS), or simply vehicle-road cooperative systems, represent a development direction for Intelligent Transportation Systems (ITS). IVICS utilizes advanced wireless communication and next-generation Internet technologies to implement comprehensive, real-time dynamic information interaction between vehicles and infrastructure. Based on the collection and fusion of dynamic traffic information across all times and spaces, it conducts active vehicle safety control and cooperative road management, fully realizing effective collaboration between people, vehicles, and roads. This ensures traffic safety, improves traffic efficiency, and ultimately forms a safe, efficient, and environmentally friendly road traffic system. This application can be applied to the aforementioned fields. For example, the method proposed in this application can be used to assess the quality of traffic videos collected by the system, allowing for the removal of low-quality videos and reducing the system's workload. Alternatively, the quality assessment method can be used to obtain low-quality videos and identify the corresponding video acquisition equipment, thereby enabling rapid location of faulty equipment and improving maintenance efficiency.

[0036] Currently, video content quality is assessed manually. For example, this involves collecting partial image frames from a video, scoring each frame's quality, and then averaging these scores to obtain a predicted quality score. However, this method relies on the scorer's experience, leading to strong subjectivity and low accuracy and efficiency in quality prediction. Therefore, this application proposes a reference-free video quality assessment method. This method uses distorted videos or their features as input to learn different types of distorted video features and determine the subjective quality of the current video. This approach makes information acquisition easier in practical video quality assessment, requires less computation, and enables real-time computation on small datasets in scenarios such as cloud gaming, thus achieving online, real-time video quality assessment.

[0037] The referenceless video quality assessment method proposed in this application can use information from the video encoding process as input; it can also use distorted video as input; or it can combine video encoding parameters and video content information as input. However, referenceless video quality assessment algorithms that use texture and content information from distorted video as input require the use of image processing algorithms or deep learning methods to extract spatial and temporal features of the video. The feature extraction process is computationally intensive, slow, and demands high computing resources such as GPUs or CPUs. Especially in scenarios such as cloud gaming, such methods also require additional decoding of the encoded video to obtain image information, increasing the difficulty of real-time video quality assessment. Furthermore, using information from only the video encoding process as input leads to low accuracy in video quality assessment. For example, ITU-TP.1203 primarily uses the coding bitrate as the main indicator to measure quantization loss during the encoding process. However, for different game types, bitrate is insufficient to measure the actual loss of image quality. In some complex 3D games, encoding at a bitrate of 20M still results in a subjective feeling of significant loss in image quality; while in some simple 2D games, encoding at a bitrate of 20M yields an image quality that is subjectively visually lossless.

[0038] Based on this, this application proposes a no-reference video quality assessment method based on multiple coding parameters. This method does not require additional decoding of each frame; it only needs to utilize multiple coding parameters calculated during the coding process to predict the video's quality score (e.g., Mean Opinion Score (MOS)). This enables real-time video quality assessment with minimal computational cost, facilitating real-time monitoring of image quality in scenarios such as cloud gaming and timely detection of online anomalies. Compared to manual detection methods, this application can automatically predict the quality assessment result of the video to be evaluated, improving the accuracy and efficiency of video quality prediction.

[0039] Specifically, taking NVIDIA graphics cards as an example, this application can use encoding parameters such as the sum of absolute transformed differences (SATD), quantizer parameter (QP), motion vector (MV), frame length (FL), frame rate (FR), and resolution (Res) output by the graphics card during encoding as input for video quality assessment, without requiring additional video decoding and feature extraction. This method uses multiple metrics as input simultaneously, resulting in higher model accuracy and stronger generalization ability. Since the input information is simpler than images, modeling can be completed using some low-complexity models, enabling real-time image quality prediction without using GPU resources and with minimal CPU resources.

[0040] It is understood that in the specific embodiments of this application, data related to encoding parameters, videos to be evaluated, etc. are involved. When the above embodiments of this application are applied to specific products or technologies, the collection, use and processing of related data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0041] This application will be specifically illustrated through the following embodiments:

[0042] Please see Figure 1 This figure is a schematic diagram of the architecture of an image processing system provided in an exemplary embodiment of this application. The image processing system may specifically include a terminal device 101 and a server 102. The terminal device 101 and the server 102 are connected via a network, such as a local area network (LAN), a wide area network (WAN), or the mobile internet. The user operates on a browser or client application on the terminal device 101 to process various image data. The server 102 can respond to this operation by providing various image data processing services to the user.

[0043] Specifically, during the image and video encoding process of the terminal device 101, the terminal device 101 can collect the encoding parameters corresponding to multiple video frames to be processed in the video to be evaluated in real time. The server 102 can obtain the encoding parameters corresponding to multiple video frames to be processed from the terminal device 101, and call the first quality assessment model to perform quality assessment processing on the encoding parameters of each video frame to be processed, so as to obtain the quality assessment parameters of each video frame to be processed. Over a period of time, the server 102 calculates the average value of the quality assessment parameters of each video frame to be processed, and obtains the overall quality assessment result of each video frame to be processed, which is used as the quality assessment result of the video to be evaluated. The server 102 can perform processing such as screen push and image quality compensation based on the quality assessment result.

[0044] Terminal equipment 101 is also referred to as terminal, user equipment (UE), access terminal, user unit, mobile device, user terminal, wireless communication equipment, user agent, or user device. Terminal equipment can be smart home appliances, handheld devices with wireless communication capabilities (such as smartphones and tablets), computing devices (such as personal computers (PCs), in-vehicle terminals, smart voice interaction devices, wearable devices, or other smart devices, but is not limited to these).

[0045] Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0046] In one embodiment, the architecture of the image processing system proposed in this application may further include a database 103. Database 103 can be used to store data such as encoding parameters, quality assessment parameters, and quality assessment results of the video frames to be processed. It can also be used to store relevant information about the first quality assessment model and the second quality assessment model. This data can be recorded in the database through different database tables. For example, database 103 can be a database located on a server, i.e., a database built into or integrated with the server; database 103 can also be an external database connected to the server, such as a cloud database (i.e., a database deployed in the cloud). Specifically, it can be deployed based on any of the following: private cloud, public cloud, hybrid cloud, edge cloud, etc., thus allowing the cloud database to focus on different functions. For example, a database deployed in a private cloud uses basic cloud hardware that is the user's personal device, focusing more on serving a small number of users. A database deployed in a public cloud, on the other hand, is deployed based on a third-party cloud platform, allowing data sharing. Any user's data can be stored in this database, and any user can use the data in the database.

[0047] It is understood that the system architecture diagrams described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. For example, the image processing method provided in the embodiments of this application can be executed not only by server 102, but also by other servers or server clusters that are different from server 102 and can communicate with terminal device 101 and / or server 102. For example, the method provided in the embodiments of this application can also be executed independently by terminal device 101 with computing power and model processing capabilities. When performing encoding, terminal device 101 obtains encoding parameters, processes them, and obtains the current device's quality assessment results in real time to facilitate automated image quality detection. As those skilled in the art will know, Figure 1 The number of terminal devices and servers shown is merely illustrative. Any number of terminal devices and servers can be configured according to business needs. Furthermore, as system architecture evolves and new business scenarios emerge, the technical solutions provided in this application are also applicable to similar technical problems. In subsequent embodiments, "terminal device" will refer to the aforementioned terminal device 101, and "server" will refer to the aforementioned server 102; further details will not be repeated in subsequent embodiments.

[0048] Please see Figure 2 This figure is a schematic flowchart of an image processing method provided in an exemplary embodiment of this application, applied to a server (referring to...). Figure 1 Taking server 102 as an example, the method may include the following steps:

[0049] S201. Obtain the encoding parameters of each video frame to be processed in the set of video frames to be processed. The set of video frames to be processed is the set of video frames of the video to be evaluated.

[0050] In this embodiment, the video to be evaluated is the video for which video quality assessment is required, and the set of video frames to be processed is image data (including multiple video frames) obtained based on the video to be evaluated. The encoding parameters of each video frame to be processed are the encoding parameters generated when encoding each video frame to be processed during the encoding process of the video to be evaluated. This step is to obtain input data for video quality assessment of the video to be evaluated. In this application, the first quality assessment model performs video quality assessment based on the encoding parameters of each video frame to be processed obtained in this step.

[0051] In one embodiment, the set of video frames to be processed can be all the video frames included in the video to be evaluated, or it can be a subset of the video frames included in the video to be evaluated. When the set of video frames to be processed is a subset of the video frames included in the video to be evaluated, the set of video frames to be processed can be a subset of video frames randomly selected from all the video frames included in the video to be evaluated, or it can be a subset of video frames acquired at intervals from all the video frames (for example, in a video to be evaluated comprising 40 frames, frames 10, the next 20, 30, and 40 are acquired at intervals as the set of video frames to be processed). By determining the set of video frames to be processed from the video to be evaluated using the above method, the number of video frames that need to be analyzed can be reduced, the computational load can be reduced, and thus the efficiency of video quality analysis can be improved.

[0052] In one embodiment, the encoding parameters described in this application may include one or more of the following: absolute error of video frames, quantization parameters, motion vectors, video frame length, frame rate, and resolution.

[0053] In this embodiment, the server directly uses the encoding parameters obtained during the encoding process of the video to be evaluated, without requiring additional processing such as decoding of the image frames of the video to be evaluated, thereby improving the efficiency of video quality assessment. This application uses multiple encoding parameters as input, resulting in a quality assessment model trained with multiple encoding parameters that has higher accuracy and stronger generalization ability. Since the input information is simpler than the image, modeling can be completed using some low-complexity models, enabling real-time image quality prediction without using GPU resources and with minimal CPU resources, further improving the efficiency of video quality assessment.

[0054] In one embodiment, since the values ​​of different components in the encoding parameters of each video frame to be processed differ significantly, scaling or normalization of the different components is necessary for the model to better learn the feature patterns of the encoding parameters. The acquisition of the encoding parameters of each video frame in the set of video frames to be processed can be achieved according to the following steps.

[0055] (a1) Obtain the initial encoding parameters of each video frame to be processed in the set of video frames to be processed, and obtain the scaling factor corresponding to the initial encoding parameters.

[0056] (a2) The initial encoding parameters are processed according to the scaling factor, and the processed encoding parameters of each video frame to be processed are determined as the encoding parameters of each video frame to be processed.

[0057] The initial encoding parameters of the video frames to be processed refer to the unprocessed raw encoding information obtained by the server. Since there can be multiple types of initial encoding parameters, there may be significant numerical differences between different types (e.g., the value of one type of initial encoding parameter fluctuates within the range of [0-1], while another type fluctuates within the range of [1000-5000]), resulting in lower correlation between the feature information of different types of encoding parameters extracted based on the initial encoding parameters. Therefore, the server can first obtain the scaling factor corresponding to each encoding parameter type (e.g., the scaling factor a1 corresponds to the initial encoding parameter of the absolute error sum type, and the scaling factor a2 corresponds to the initial encoding parameter of the quantization parameter type). Then, it processes the initial encoding parameters corresponding to that encoding parameter type according to the scaling factor to obtain the scaled encoding parameters. By scaling the initial encoding parameters using the above method, the numerical differences between different components in the encoding parameters of each video frame to be processed can be reduced, allowing the model to better learn the features of the encoding parameters and improve the model training effect.

[0058] Taking some types of encoding parameters as an example, the calculation formula for scaling the initial encoding parameters is as follows:

[0059]

[0060]

[0061]

[0062]

[0063]

[0064]

[0065] Where SATD, QP, MV, FL, and FR are the initial absolute error, initial quantization parameters, initial motion vector, initial video frame length, and initial frame rate, respectively, among the initial coding parameters of the video frame. w and h are the width and height of the current video frame, and W and H are the maximum width and maximum height in the set of video frames to be processed. These represent the absolute error, quantization parameters, motion vectors, video frame length, frame rate, and resolution, respectively, in the encoded parameters of the scaled video frame. a1, a2, a3, a4, and a5 are the scaling factors for SATD, QP, MV, FL, and FR, respectively.

[0066] S202. Call the first quality assessment model to perform quality assessment processing on the encoding parameters of each video frame to be processed, and obtain the quality assessment parameters of each video frame to be processed.

[0067] In this embodiment, the first quality assessment model can perform quality assessment processing on the encoding parameters of each video frame to be processed, obtaining quality assessment parameters for each video frame. The quality assessment parameters indicate the image quality status of the video frame to be processed. By obtaining the quality assessment parameters of each video frame to be processed, it is convenient to understand the image quality of each video frame, and then analyze the quality assessment results of the video to be evaluated based on the quality assessment parameters of each video frame.

[0068] In one embodiment, the quality assessment parameter can be in the form of an image quality score (e.g., the image quality score ranges from [0, 100], with better image quality resulting in a score closer to 100 and worse image quality resulting in a score closer to 0); the quality assessment parameter can also be in the form of an image quality level (e.g., image quality levels include level 1, level 2, level 3, level 4, and level 5, with better image quality resulting in a level closer to level 5 and worse image quality resulting in a level closer to level 1). This method improves the flexibility of setting the quality assessment parameter, thereby enriching the data format of the quality assessment results.

[0069] S203. Determine the quality assessment result of the video to be evaluated based on the quality assessment parameters of each video frame to be processed; wherein, the first quality assessment model is trained in combination with the second quality assessment model. During the model training process, the first quality assessment model is used to determine the first predicted quality assessment parameters of the first sample video frame based on the encoding parameters of the first sample video frame, and the second quality assessment model is used to determine the second predicted quality assessment parameters of the first sample video frame based on the first sample video frame and its encoding parameters. The first quality assessment model is obtained by adjusting the model parameters using the target loss parameters determined based on the first and second predicted quality assessment parameters; the second quality assessment model is trained using the second sample video frame, its encoding parameters, and quality labels.

[0070] In this embodiment, the set of video frames to be processed is part or all of the video frames obtained from the video to be evaluated. The quality evaluation result of the video to be evaluated can be determined by the quality evaluation parameters of each video frame to be processed.

[0071] In this embodiment, the first quality assessment model is trained in conjunction with the second quality assessment model. During model training, the first quality assessment model is used to determine the first predicted quality assessment parameters of the first sample video frame based on the encoding parameters of the first sample video frame. The second quality assessment model is used to determine the second predicted quality assessment parameters of the first sample video frame based on the first sample video frame and its encoding parameters. The first quality assessment model is obtained by adjusting the model parameters using the target loss parameters determined based on the first and second predicted quality assessment parameters. The second quality assessment model is trained using the second sample video frame, its encoding parameters, and quality labels.

[0072] The specific training process of the first quality assessment model and the second quality assessment model will be described in detail in subsequent implementation steps S401-S405, and will not be repeated in this implementation.

[0073] Please see Figure 3 The diagram shows the processing flowchart of the first quality assessment model. The input to the first quality assessment model is the encoding parameters of the video frame to be processed. By inputting these parameters into the first quality assessment model, the quality assessment result of the video to be evaluated is obtained. Since the first quality assessment model directly uses the encoding parameters obtained during the encoding process of the video to be evaluated, without requiring additional processing such as decoding of the image frames of the video to be evaluated, the efficiency of video quality assessment is improved. This makes it suitable for real-time video quality assessment in scenarios such as cloud gaming, and allows for real-time image quality assessment with very low computational cost.

[0074] In one embodiment, the determination of the quality assessment result of the video to be evaluated based on the quality assessment parameters of each video frame to be processed can be achieved according to the following steps.

[0075] (b1) Calculate the mean of the quality assessment parameters for each video frame to be processed to obtain the average quality assessment parameters.

[0076] (b2) Determine the quality assessment results of the video to be assessed based on the average quality assessment parameters.

[0077] In this embodiment, the quality assessment result can be an average quality assessment parameter, or a judgment result based on the average quality assessment parameter and assessment conditions. For example, the assessment conditions can be a quality score threshold, a quality grade threshold, etc. When the average quality assessment parameter meets the assessment conditions, the judgment result is determined to be qualified; when the average quality assessment parameter does not meet the assessment conditions, the judgment result is determined to be unqualified.

[0078] In one embodiment, when the quality assessment parameter is in the form of an image quality score, and the image quality scores of each video frame to be processed are [70, 80, 60, 50, 90], then the average quality assessment parameter obtained by averaging the image quality scores of each video frame to be processed is 70; if the assessment condition is a quality score threshold (80), then the quality assessment result of the video to be evaluated is determined to be unqualified.

[0079] In one embodiment, when the quality assessment parameter is in the form of image quality levels, and the image quality levels of each video frame to be processed are [level 5, level 3, level 4, level 3, level 5], then the average quality assessment parameter obtained by averaging the image quality levels of each video frame to be processed is level 4; if the assessment condition is a quality level threshold (level 4), then the quality assessment result of the video to be evaluated is determined to be qualified. The various forms of quality assessment parameters enrich the data format of the quality assessment results.

[0080] In one embodiment, the formula for calculating the average quality assessment parameter based on the quality assessment parameters of each video frame to be processed is as follows:

[0081]

[0082] Among them, O s Here, N is the average quality assessment parameter, and N is the number of video frames in the video frames to be processed. Let be the quality evaluation parameters for the i-th video frame to be processed.

[0083] In one embodiment, this application can be applied to real-time scenarios such as cloud gaming for video quality assessment, achieving real-time image quality assessment with very low computational cost. When the quality assessment result of the video to be evaluated indicates that the video quality is poor, it means that the image quality on the terminal device (i.e., the source device that generates the encoding parameters of each video frame to be processed) performing the encoding operation on the video to be evaluated does not meet the service requirements. At this time, the image on the terminal device can be compensated to display a higher quality video on the terminal device, thereby improving the user experience.

[0084] In one embodiment, the method for compensating the image quality of a terminal device can be as follows: First, image quality compensation parameters are determined based on the encoding parameters of each video frame to be processed. These parameters can include multiple types of compensation parameters, each used to compensate for a specific feature dimension during the video encoding process. Then, image quality compensation processing is performed during the encoding process of the terminal device based on these parameters, so that the terminal device encodes the compensated target video. Specifically, the image quality compensation parameters can be applied to the original video pre-encoded by the terminal device to obtain a compensated original video, which the terminal device then encodes to obtain a video with higher quality. Alternatively, the image quality compensation parameters can be applied to the encoded video after the terminal device encodes the original video to obtain a compensated encoded video, which the terminal device then directly displays to obtain a video with higher quality. This embodiment does not limit the scope of the method.

[0085] Based on the above embodiments, the beneficial effects of this application are as follows: This application proposes that the set of video frames to be processed can be all video frames included in the video to be evaluated, or only a portion of the video frames included in the video to be evaluated, thereby reducing the number of video frames that need to be analyzed, reducing the computational load, and thus improving the efficiency of video quality analysis. This application also proposes that the encoding parameters can include one or more of the absolute error sum of video frames, quantization parameters, motion vectors, video frame length, frame rate, and resolution. Using multiple encoding parameters as inputs simultaneously makes the quality assessment model trained with multiple encoding parameters more accurate and has stronger generalization ability. This application further proposes that by scaling or normalizing the initial encoding parameters of different components to obtain the encoding parameters, and then performing quality assessment processing based on the encoding parameters, the numerical differences of different components in the encoding parameters of each video frame to be processed can be reduced, so that the model can better learn the features of the encoding parameters and improve the model training effect. This application also proposes that the quality assessment parameters can be in the form of image quality scores or image quality levels. Through the above methods, the flexibility of setting quality assessment parameters is improved, thereby enriching the data format of the quality assessment results.

[0086] Please see Figure 4 This figure is a schematic flowchart of an image processing method provided in an exemplary embodiment of this application, applied to a server (referring to...). Figure 1 Taking server 102 as an example, the method may include the following steps:

[0087] S401. Obtain the first sample video frame set and the encoding parameters of each first sample video frame in the first sample video frame set. The first sample video frame set is the video frame set of the first sample video.

[0088] In this embodiment, the first sample video is sample data used for training the initial first quality assessment model, and the first sample video frame set is image data (including multiple video frames, which may be all or part of the video frames included in the first sample video) obtained based on the first sample video. The encoding parameters of each first sample video frame are the encoding parameters generated when encoding each first sample video frame during the encoding process of the first sample video.

[0089] S402. Call the initial first quality assessment model to perform quality assessment processing on the encoding parameters of each first sample video frame to obtain the first predicted quality assessment parameters of each first sample video frame.

[0090] In this embodiment, the initial first quality assessment model can perform quality assessment processing based on the coding parameters to obtain the quality assessment parameters corresponding to the coding parameters. For the coding parameters of each first sample video frame, the initial first quality assessment model can process the coding parameters of each first sample video frame separately to obtain the first predicted quality assessment parameters corresponding to each first sample video frame.

[0091] S403. Call the second quality assessment model to perform quality assessment processing on each first sample video frame and the encoding parameters of each first sample video frame to obtain the second quality assessment parameters of each first sample video frame.

[0092] In this embodiment, the second quality assessment model is a pre-trained assessment model. Compared to the first quality assessment model (which is the initial model at this point), the second quality assessment model requires joint quality assessment processing of both the video frame and its coding parameters to obtain quality assessment parameters that integrate the image and coding features of the video frame. The second quality assessment model can employ a complex full-reference or no-reference video quality assessment model based on video images and coding information. Compared to the first quality assessment model, the second quality assessment model has higher prediction accuracy and higher model complexity.

[0093] S404. Determine the target loss parameter based on the first prediction quality assessment parameter and the second prediction quality assessment parameter of each first sample video frame.

[0094] S405. Adjust the model parameters of the initial first quality assessment model according to the target loss parameters to obtain the trained first quality assessment model.

[0095] In steps S404-S405, the second prediction quality assessment parameter is determined by the second quality assessment model based on each first sample video frame and the encoding parameters of each first sample video frame. This serves as supervision information for the first prediction quality assessment parameter determined by the second prediction quality assessment parameter based on the first sample video frames. The server can determine the target loss parameter using the first and second prediction quality assessment parameters, and then adjust the model parameters of the initial first quality assessment model based on the target loss parameter to obtain the trained first quality assessment model.

[0096] The model training process in this application can be divided into two stages. Stage one involves training an initial second quality assessment model using the quality labels of the sample videos. Stage two involves training an initial first quality assessment model using the soft labels (not real quality labels, but image frame labels generated by the second quality assessment model based on the quality labels of each image frame in the sample videos) generated by the trained second quality assessment model. In other words, the learning objective of the second quality assessment model is the real subjective quality score of the video, while the learning objective of the first quality assessment model is the frame-level prediction result generated by the second quality assessment model (i.e., the aforementioned soft labels). This process is also known as "knowledge distillation." The first quality assessment model is the target lightweight quality assessment model in this application.

[0097] Please see Figure 5AThis diagram illustrates the training flowchart of an image processing model. Using sample videos and their encoding parameters as training input data, the second quality assessment model determines second predicted quality assessment parameters based on the sample videos and their encoding parameters. Then, it uses the quality labels of the sample videos and the second predicted quality assessment parameters to determine reference loss parameters. The second quality assessment model is then trained based on these reference loss parameters. Similarly, the first quality assessment model determines first predicted quality assessment parameters based on the encoding parameters of the sample videos. Then, it uses these first predicted quality assessment parameters and the second predicted quality assessment parameters provided by the second quality assessment model to determine target loss parameters. The first quality assessment model is then trained based on these target loss parameters.

[0098] In one embodiment, the first quality assessment model is trained by combining it with a second quality assessment model. The training process of the second quality assessment model can be implemented according to the following steps.

[0099] (c1) Obtain the second sample video frame set, and obtain the encoding parameters and quality labels of each second sample video frame in the second sample video frame set. The second sample video frame set is the video frame set of the second sample video.

[0100] (c2) Call the initial second quality assessment model to perform quality assessment processing on each second sample video frame and the encoding parameters of each second sample video frame to obtain the second prediction quality assessment parameters of each second sample video frame.

[0101] In steps (c1-c2), the second quality assessment model needs to perform quality assessment processing based on the encoding parameters and quality labels of each second sample video frame to obtain the second predicted quality assessment parameters of each second sample video frame. Then, it is trained based on the second predicted quality assessment parameters and quality labels. By combining the image features and encoding features of the second sample video frames, a higher model prediction accuracy is obtained, thereby providing a more accurate frame-level prediction result (soft label, i.e., the second predicted quality assessment parameter) for the first quality assessment model, ensuring the training effect of the first quality assessment model.

[0102] In one embodiment, the quality label can be the Mean Opinion Score (MOS). The second sample video corresponds to the overall quality label of the video, and the quality labels of each second sample video frame are the same as the overall quality label of the video corresponding to the second sample video.

[0103] In one embodiment, the above-mentioned invocation of the initial second quality assessment model to perform quality assessment processing on each second sample video frame and the encoding parameters of each second sample video frame to obtain the second predicted quality assessment parameters of each second sample video frame can be implemented according to the following steps.

[0104] (c21) For any second sample video frame in the set of second sample video frames, perform feature extraction processing on any second sample video frame to obtain image features, and perform feature extraction processing on the encoding parameters of any second sample video frame to obtain encoding features, and perform feature fusion processing on the image features and encoding features to obtain the fused features of any second sample video frame.

[0105] (c22) Call the initial second quality assessment model to perform quality assessment processing on the fusion features of each second sample video frame to obtain the second prediction quality assessment parameters of each second sample video frame.

[0106] In steps (c21-c22), the second quality assessment model performs quality assessment by combining the image features and coding features of the second sample video frame to obtain the second predicted quality assessment parameters. This allows the second predicted quality assessment parameters to incorporate multi-dimensional feature information, ensuring the training effect.

[0107] (c3) Determine the reference loss parameter based on the second prediction quality assessment parameter and quality label of each second sample video frame.

[0108] In one embodiment, the reference loss parameter can be one or more of the following loss functions: 0-1 loss function, absolute value loss function, mean squared error loss function, log loss function, cross-entropy loss function, and exponential loss function. When multiple loss functions are used, the reference loss parameter can be determined jointly based on the calculation results corresponding to the multiple loss functions.

[0109] (c4) Adjust the model parameters of the initial second quality assessment model according to the reference loss parameters to obtain the trained second quality assessment model.

[0110] In one embodiment, the method for training the model using reference loss parameters can be as follows: backpropagation is performed using the reference loss function to update the weights of each node in the initial second quality assessment model. When the reference loss function meets the preset conditions, the final weights of each node in the initial second quality assessment model are determined, thereby obtaining the trained second quality assessment model.

[0111] Please see Figure 5BThe diagram shows the training flowchart of the second quality assessment model. Taking sample videos and their encoding parameters as training input data as an example, the second quality assessment model determines the second predicted quality assessment parameters based on the sample videos and their encoding parameters. Then, it uses the quality labels of the sample videos and the second predicted quality assessment parameters to determine the reference loss parameters. The second quality assessment model is then trained based on the reference loss parameters.

[0112] In this approach, the quality label can be a tag corresponding to the sample video that indicates the overall video quality. The second quality assessment model obtains the frame-level prediction result of the sample video based on this label, thereby providing supervision information for the first quality assessment model and completing its training. This method can reduce the amount of labeled data for sample videos, improve labeling efficiency, and reduce labor costs.

[0113] Please see Figure 5C The figure shows the training flowchart of the first quality assessment model. The first quality assessment model determines the first predicted quality assessment parameter based on the encoding parameters of the sample video. Then, it uses the first predicted quality assessment parameter and the second predicted quality assessment parameter provided by the second quality assessment model to determine the target loss parameter. The first quality assessment model is trained based on the target loss parameter.

[0114] The above embodiments will be described below through specific model processing procedures. Assume there is a set of sample video frames I = {I0, I1, I2…I…} n}, where I i Let be the i-th frame image in the sample video frame set, i∈[0,n]. Let the encoding parameters of each sample video frame in the sample video frame set be E={E0,E1,E2…E n}, where E i Let be the encoding parameters for the i-th sample video frame, where i ∈ [0, n]. Taking an NVIDIA graphics card as an example, we can take... E i The encoding parameters included can be adjusted based on the encoding information obtained in specific business scenarios. The second quality assessment model can use inp = {I, E} as input, integrating the content texture of video frames and information related to encoding parameters to train a more accurate model. The first quality assessment model primarily uses the encoding parameter E as input and can use low-complexity algorithms such as multinomial fitting and fully connected neural networks as the main model. Let the output of the second quality assessment model be... The output of the first quality assessment model is: in, and Let $\frac{i}{i}$ be the predicted quality assessment parameters for the $i$-th sample video frame predicted by the second quality assessment model and the first quality assessment model, respectively, where $i \in [0, N]$. The loss during the training process of the first quality assessment model is $loss = f(oup)$. T .oup s ), where f(.) can use L1 norm or MSE as the loss function.

[0115] S406. Obtain the encoding parameters of each video frame to be processed in the set of video frames to be processed. The set of video frames to be processed is the set of video frames of the video to be evaluated.

[0116] S407. Call the first quality assessment model to perform quality assessment processing on the encoding parameters of each video frame to be processed, and obtain the quality assessment parameters of each video frame to be processed.

[0117] S408. Determine the quality assessment result of the video to be evaluated based on the quality assessment parameters of each video frame to be processed.

[0118] The specific implementation methods of steps S406 to S408 are described in the relevant descriptions of steps S201 to S203 in the aforementioned embodiments, and will not be repeated here.

[0119] In one embodiment, the first quality assessment model is trained in conjunction with a second quality assessment model. During model training, the first quality assessment model determines a first predicted quality assessment parameter for the first sample video frame based on its encoding parameters. The second quality assessment model determines a second predicted quality assessment parameter for the first sample video frame based on both the first sample video frame and its encoding parameters. The first quality assessment model is obtained by adjusting model parameters using a target loss parameter determined based on the first and second predicted quality assessment parameters. The second quality assessment model is trained using the second sample video frame, its encoding parameters, and a quality label. For the specific training processes of the first and second quality assessment models, please refer to the relevant descriptions in steps S401-S405; these will not be repeated in this embodiment.

[0120] Based on the above embodiments, the beneficial effects of this application are as follows: The first quality assessment model proposed in this application only needs to determine the quality assessment parameters of the video frame based on the encoding parameters of the video frame, while the second quality assessment model requires joint quality assessment processing of the video frame and its encoding parameters to obtain quality assessment parameters that integrate the image features and encoding features of the video frame. The second quality assessment model can employ a complex full-reference or no-reference video quality assessment model based on video images and encoding information. Compared to the first quality assessment model, the second quality assessment model has higher prediction accuracy and higher model complexity. By using the second predicted quality assessment parameters provided by the second quality assessment model as supervisory information for the first quality assessment model training, the training effect of the first quality assessment model is improved, thereby improving the accuracy of video quality prediction. This application also proposes that the quality label can be a label corresponding to the sample video that can indicate the overall video quality. The second quality assessment model can obtain the frame-level prediction result of the sample video based on this label, thereby providing supervisory information for the first quality assessment model and completing the model training of the first quality assessment model. This method can reduce the amount of labeled data for sample videos, improve labeling efficiency, and reduce labor costs.

[0121] Please see Figure 6 This figure is a schematic block diagram of an image processing apparatus provided in an embodiment of this application. Specifically, the image processing apparatus may include:

[0122] The acquisition module 601 is used to acquire the encoding parameters of each video frame to be processed in the set of video frames to be processed, wherein the set of video frames to be processed is the set of video frames of the video to be evaluated.

[0123] The processing module 602 is used to call the first quality assessment model to perform quality assessment processing on the encoding parameters of each of the above-mentioned video frames to be processed, so as to obtain the quality assessment parameters of each of the above-mentioned video frames to be processed.

[0124] The evaluation module 603 is used to determine the quality evaluation result of the video to be evaluated based on the quality evaluation parameters of each video frame to be processed.

[0125] The first quality assessment model is trained in conjunction with the second quality assessment model. During model training, the first quality assessment model is used to determine the first predicted quality assessment parameters of the first sample video frame based on the encoding parameters of the first sample video frame. The second quality assessment model is used to determine the second predicted quality assessment parameters of the first sample video frame based on the first sample video frame and its encoding parameters. The first quality assessment model is obtained by adjusting the model parameters using the target loss parameters determined based on the first and second predicted quality assessment parameters. The second quality assessment model is trained using the second sample video frame, its encoding parameters, and quality labels.

[0126] Optionally, the aforementioned acquisition module 601 is also used for:

[0127] Obtain the first sample video frame set and the encoding parameters of each first sample video frame in the first sample video frame set, wherein the first sample video frame set is the video frame set of the first sample video;

[0128] The aforementioned processing module 602 is also used for:

[0129] The initial first quality assessment model is invoked to perform quality assessment processing on the encoding parameters of each of the first sample video frames to obtain the first predicted quality assessment parameters of each of the first sample video frames.

[0130] The second quality assessment model described above is invoked to perform quality assessment processing on each of the first sample video frames and the encoding parameters of each of the first sample video frames, thereby obtaining the second quality assessment parameters of each of the first sample video frames.

[0131] The target loss parameters are determined based on the first and second prediction quality assessment parameters of each of the first sample video frames.

[0132] The model parameters of the initial first quality assessment model are adjusted based on the target loss parameters to obtain the trained first quality assessment model.

[0133] Optionally, the aforementioned acquisition module 601 is also used for:

[0134] Obtain the second sample video frame set, and obtain the encoding parameters and quality labels of each second sample video frame in the second sample video frame set, wherein the second sample video frame set is the video frame set of the second sample video;

[0135] When the aforementioned processing module 602 is used, it is specifically used for:

[0136] The initial second quality assessment model is invoked to perform quality assessment processing on each of the above second sample video frames and the encoding parameters of each of the above second sample video frames, so as to obtain the second prediction quality assessment parameters of each of the above second sample video frames.

[0137] The reference loss parameter is determined based on the second predicted quality assessment parameter and quality label of each of the second sample video frames mentioned above.

[0138] The model parameters of the initial second quality assessment model are adjusted based on the aforementioned reference loss parameters to obtain the trained second quality assessment model.

[0139] Optionally, when the processing module 602 calls the initial second quality assessment model to perform quality assessment processing on each of the second sample video frames and the encoding parameters of each of the second sample video frames to obtain the second prediction quality assessment parameters of each of the second sample video frames, it is specifically used for:

[0140] For any one of the second sample video frames in the set of second sample video frames, feature extraction processing is performed on the second sample video frame to obtain image features, and feature extraction processing is performed on the encoding parameters of the second sample video frame to obtain encoding features. The image features and the encoding features are then fused to obtain the fused features of the second sample video frame.

[0141] The initial second quality assessment model is invoked to perform quality assessment on the fusion features of each of the above second sample video frames, thereby obtaining the second predicted quality assessment parameters for each of the above second sample video frames.

[0142] Optionally, when the acquisition module 601 acquires the encoding parameters of each video frame in the set of video frames to be processed, it is specifically used for:

[0143] Obtain the initial encoding parameters of each video frame to be processed in the set of video frames to be processed, and obtain the scaling factor corresponding to the initial encoding parameters.

[0144] The initial encoding parameters are processed according to the scaling factor, and the processed encoding parameters of each video frame to be processed are determined as the encoding parameters of each video frame to be processed.

[0145] Optionally, the above encoding parameters include one or more of the following: absolute error of video frames, quantization parameters, motion vectors, video frame length, frame rate, and resolution.

[0146] Optionally, when the evaluation module 603 determines the quality evaluation result of the video to be evaluated based on the quality evaluation parameters of each video frame to be processed, it is specifically used for:

[0147] The average quality assessment parameters of each of the above video frames to be processed are calculated to obtain the average quality assessment parameters.

[0148] The quality assessment results of the videos to be evaluated are determined based on the above average quality assessment parameters.

[0149] It should be noted that the functions of each functional module of the image processing device in this application embodiment can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the above method embodiments, which will not be repeated here.

[0150] Please see Figure 7 This figure is a schematic block diagram of a computer device provided in an embodiment of this application. As shown in the figure, the smart terminal in this embodiment may include: a processor 701, a storage device 702, and a communication interface 703. The processor 701, storage device 702, and communication interface 703 can interact with each other.

[0151] The aforementioned storage device 702 may include volatile memory, such as random-access memory (RAM); the storage device 702 may also include non-volatile memory, such as flash memory, solid-state drive (SSD), etc.; the aforementioned storage device 702 may also include a combination of the above types of memory.

[0152] The processor 701 described above may be a central processing unit (CPU). In one embodiment, the processor 701 may also be a graphics processing unit (GPU). Alternatively, the processor 701 may be a combination of a CPU and a GPU. In one embodiment, the storage device 702 is used to store program instructions, which the processor 701 can invoke to perform the following operations:

[0153] Obtain the encoding parameters of each video frame to be processed in the set of video frames to be processed, where the set of video frames to be processed is the set of video frames to be evaluated.

[0154] The first quality assessment model is called to perform quality assessment on the encoding parameters of each of the above-mentioned video frames to be processed, so as to obtain the quality assessment parameters of each of the above-mentioned video frames to be processed.

[0155] The quality assessment results of the video to be evaluated are determined based on the quality assessment parameters of each video frame to be processed.

[0156] The first quality assessment model is trained in conjunction with the second quality assessment model. During model training, the first quality assessment model is used to determine the first predicted quality assessment parameters of the first sample video frame based on the encoding parameters of the first sample video frame. The second quality assessment model is used to determine the second predicted quality assessment parameters of the first sample video frame based on the first sample video frame and its encoding parameters. The first quality assessment model is obtained by adjusting the model parameters using the target loss parameters determined based on the first and second predicted quality assessment parameters. The second quality assessment model is trained using the second sample video frame, its encoding parameters, and quality labels.

[0157] Optionally, the processor 701 described above is also used for:

[0158] Obtain the first sample video frame set and the encoding parameters of each first sample video frame in the first sample video frame set, wherein the first sample video frame set is the video frame set of the first sample video;

[0159] The aforementioned processor 701 is also used for:

[0160] The initial first quality assessment model is invoked to perform quality assessment processing on the encoding parameters of each of the first sample video frames to obtain the first predicted quality assessment parameters of each of the first sample video frames.

[0161] The second quality assessment model described above is invoked to perform quality assessment processing on each of the first sample video frames and the encoding parameters of each of the first sample video frames, thereby obtaining the second quality assessment parameters of each of the first sample video frames.

[0162] The target loss parameters are determined based on the first and second prediction quality assessment parameters of each of the first sample video frames.

[0163] The model parameters of the initial first quality assessment model are adjusted based on the target loss parameters to obtain the trained first quality assessment model.

[0164] Optionally, the processor 701 described above is also used for:

[0165] Obtain the second sample video frame set, and obtain the encoding parameters and quality labels of each second sample video frame in the second sample video frame set, wherein the second sample video frame set is the video frame set of the second sample video;

[0166] When the aforementioned processor 701 is used, it is specifically used for:

[0167] The initial second quality assessment model is invoked to perform quality assessment processing on each of the above second sample video frames and the encoding parameters of each of the above second sample video frames, so as to obtain the second prediction quality assessment parameters of each of the above second sample video frames.

[0168] The reference loss parameter is determined based on the second predicted quality assessment parameter and quality label of each of the second sample video frames mentioned above.

[0169] The model parameters of the initial second quality assessment model are adjusted based on the aforementioned reference loss parameters to obtain the trained second quality assessment model.

[0170] Optionally, when the processor 701 calls the initial second quality assessment model to perform quality assessment processing on each of the second sample video frames and the encoding parameters of each of the second sample video frames to obtain the second prediction quality assessment parameters of each of the second sample video frames, it specifically performs the following:

[0171] For any one of the second sample video frames in the set of second sample video frames, feature extraction processing is performed on the second sample video frame to obtain image features, and feature extraction processing is performed on the encoding parameters of the second sample video frame to obtain encoding features. The image features and the encoding features are then fused to obtain the fused features of the second sample video frame.

[0172] The initial second quality assessment model is invoked to perform quality assessment on the fusion features of each of the above second sample video frames, thereby obtaining the second predicted quality assessment parameters for each of the above second sample video frames.

[0173] Optionally, when the processor 701 obtains the encoding parameters of each video frame in the set of video frames to be processed, it specifically performs the following:

[0174] Obtain the initial encoding parameters of each video frame to be processed in the set of video frames to be processed, and obtain the scaling factor corresponding to the initial encoding parameters.

[0175] The initial encoding parameters are processed according to the scaling factor, and the processed encoding parameters of each video frame to be processed are determined as the encoding parameters of each video frame to be processed.

[0176] Optionally, the above encoding parameters include one or more of the following: absolute error of video frames, quantization parameters, motion vectors, video frame length, frame rate, and resolution.

[0177] Optionally, when the processor 701 determines the quality assessment result of the video to be evaluated based on the quality assessment parameters of each of the video frames to be processed, it specifically performs the following:

[0178] The average quality assessment parameters of each of the above video frames to be processed are calculated to obtain the average quality assessment parameters.

[0179] The quality assessment results of the videos to be evaluated are determined based on the above average quality assessment parameters.

[0180] In specific implementation, the processor 701, storage device 702, and communication interface 703 described in the embodiments of this application can execute the embodiments of this application. Figure 2 or Figure 4 The implementation methods described in the relevant embodiments of the provided image processing method can also be used to execute the embodiments of this application. Figure 6 The implementation methods described in the relevant embodiments of the provided image processing apparatus will not be repeated here.

[0181] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0182] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium storing a computer program executed by the aforementioned image processing apparatus. This computer program includes program instructions, which, when executed by a processor, enable the execution of the aforementioned image processing apparatus. Figure 2 , Figure 4 The methods described in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same methods will also not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application. As an example, program instructions can be deployed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed across multiple locations and interconnected via a communication network. These multiple computer devices distributed across multiple locations and interconnected via a communication network can constitute a blockchain system.

[0183] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned... Figure 2 , Figure 4 The methods described in the corresponding embodiments are therefore not repeated here.

[0184] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0185] The above-disclosed embodiments are only some of the embodiments of this application, and should not be construed as limiting the scope of this application. Those skilled in the art can understand that implementing all or part of the above embodiments and making equivalent changes in accordance with the claims of this application are still within the scope of the invention.

Claims

1. An image processing method, characterized in that, The method includes: Obtain the encoding parameters of each video frame to be processed in the set of video frames to be processed, wherein the set of video frames to be processed is the set of video frames of the video to be evaluated; The first quality assessment model is invoked to perform quality assessment processing on the encoding parameters of each video frame to be processed, so as to obtain the quality assessment parameters of each video frame to be processed. The quality assessment result of the video to be evaluated is determined based on the quality assessment parameters of each video frame to be processed. The first quality assessment model is trained in conjunction with the second quality assessment model. During model training, the first quality assessment model is used to determine the first predicted quality assessment parameter of the first sample video frame based on the encoding parameters of the first sample video frame. The second quality assessment model is used to determine the second predicted quality assessment parameter of the first sample video frame based on the first sample video frame and its encoding parameters. The first quality assessment model is obtained by adjusting the model parameters using the target loss parameter determined based on the first and second predicted quality assessment parameters. The second quality assessment model is trained using the second sample video frame, its encoding parameters, and quality labels.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the first sample video frame set and the encoding parameters of each first sample video frame in the first sample video frame set, wherein the first sample video frame set is the video frame set of the first sample video; The initial first quality assessment model is invoked to perform quality assessment processing on the encoding parameters of each first sample video frame, thereby obtaining the first predicted quality assessment parameters of each first sample video frame. The second quality assessment model is invoked to perform quality assessment processing on each of the first sample video frames and the encoding parameters of each of the first sample video frames, so as to obtain the second quality assessment parameters of each of the first sample video frames. The target loss parameter is determined based on the first prediction quality assessment parameter and the second prediction quality assessment parameter of each first sample video frame. The model parameters of the initial first quality assessment model are adjusted according to the target loss parameters to obtain the trained first quality assessment model.

3. The method according to claim 1, characterized in that, The method further includes: Obtain a second sample video frame set, and obtain the encoding parameters and quality labels of each second sample video frame in the second sample video frame set, wherein the second sample video frame set is the video frame set of the second sample video; The initial second quality assessment model is invoked to perform quality assessment processing on each of the second sample video frames and the encoding parameters of each of the second sample video frames, so as to obtain the second prediction quality assessment parameters of each of the second sample video frames. The reference loss parameter is determined based on the second prediction quality assessment parameter and quality label of each second sample video frame; The model parameters of the initial second quality assessment model are adjusted according to the reference loss parameters to obtain the trained second quality assessment model.

4. The method according to claim 3, characterized in that, The process of calling the initial second quality assessment model to perform quality assessment on each of the second sample video frames and their encoding parameters, to obtain the second predicted quality assessment parameters for each of the second sample video frames, includes: For any one of the second sample video frames in the second sample video frame set, feature extraction processing is performed on the arbitrary second sample video frame to obtain image features, and feature extraction processing is performed on the encoding parameters of the arbitrary second sample video frame to obtain encoding features. Then, feature fusion processing is performed on the image features and the encoding features to obtain the fused features of the arbitrary second sample video frame. The initial second quality assessment model is invoked to perform quality assessment processing on the fusion features of each second sample video frame, thereby obtaining the second predicted quality assessment parameters for each second sample video frame.

5. The method according to any one of claims 1-4, characterized in that, The step of obtaining the encoding parameters of each video frame in the set of video frames to be processed includes: Obtain the initial encoding parameters of each video frame to be processed in the set of video frames to be processed, and obtain the scaling factor corresponding to the initial encoding parameters; The initial encoding parameters are processed according to the scaling factor, and the processed encoding parameters of each video frame to be processed are determined as the encoding parameters of each video frame to be processed.

6. The method according to any one of claims 1-4, characterized in that, The encoding parameters include one or more of the following: absolute error of video frames, quantization parameters, motion vectors, video frame length, frame rate, and resolution.

7. The method according to any one of claims 1-4, characterized in that, The step of determining the quality assessment result of the video to be evaluated based on the quality assessment parameters of each video frame to be processed includes: The average quality assessment parameters of each video frame to be processed are calculated to obtain the average quality assessment parameters. The quality assessment result of the video to be evaluated is determined based on the average quality assessment parameters.

8. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the encoding parameters of each video frame to be processed in the set of video frames to be processed, wherein the set of video frames to be processed is the set of video frames of the video to be evaluated. The processing module is used to call the first quality assessment model to perform quality assessment processing on the encoding parameters of each video frame to be processed, so as to obtain the quality assessment parameters of each video frame to be processed. The evaluation module is used to determine the quality evaluation result of the video to be evaluated based on the quality evaluation parameters of each video frame to be processed. The first quality assessment model is trained in conjunction with the second quality assessment model. During model training, the first quality assessment model is used to determine the first predicted quality assessment parameter of the first sample video frame based on the encoding parameters of the first sample video frame. The second quality assessment model is used to determine the second predicted quality assessment parameter of the first sample video frame based on the first sample video frame and its encoding parameters. The first quality assessment model is obtained by adjusting the model parameters using the target loss parameter determined based on the first and second predicted quality assessment parameters. The second quality assessment model is trained using the second sample video frame, its encoding parameters, and quality labels.

9. A computer device, characterized in that, include: The processor, the communication interface, and the storage device are interconnected, wherein the storage device stores executable program code, and the processor is used to call the executable program code to implement the image processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions, which are executed by a processor to implement the image processing method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which, when executed by a processor, are used to implement the image processing method as described in any one of claims 1 to 7.

Images