Method, system, device and medium for semi-supervised image quality assessment with ranking constraints

Abstract

The embodiment of the application relates to the technical field of digital image processing, and discloses a kind of sequencing constraint's semi-supervised image quality evaluation method, system, equipment and medium, the method is in view of the problems such as subjective annotation data scarcity in image quality evaluation, semi-supervised method pseudo label screening difficulty and traditional strong enhancement tampering quality attribute, constructs the double-branch teacher-student collaborative learning network based on CLIP, and regression task is reshaped as ordered probability distribution prediction.In the unsupervised branch, weakly enhanced images are filtered and checked by ordered concentration, and global pseudo labels are generated by graph-block consistency aggregation, and overall-attribute reverse pairing enhancement strategy is applied to original image blocks to generate difficult samples, and the network is optimized by combining supervised loss and unsupervised loss.The application can realize accurate no-reference image quality prediction under a small amount of labeled data.

Description

Technical Field

[0001] This application relates to the field of digital image processing technology, specifically to a sorting-constrained semi-supervised image quality evaluation method, system, device, and medium. Background Technology

[0002] In the digital age, digital images and videos have become the core carriers of information interaction. However, throughout the entire lifecycle of images—from acquisition and compression to transmission and processing—they are inevitably subject to various complex degradation factors such as noise, blurring, and compression artifacts. This not only severely impairs the user's visual perception experience but also greatly weakens the robustness of downstream computer vision algorithms. Therefore, establishing a referenceless image quality assessment (NR-IQA) system that can accurately and objectively simulate human visual system perception is crucial for ensuring the value of visual content and improving the reliability of intelligent systems.

[0003] Current mainstream deep learning-based image quality assessment models heavily rely on large-scale datasets with high-quality human subjective rating labels. However, obtaining reliable Mean Opinion Scores (MOS) requires a strictly controlled environment and a large number of subjects, making it extremely costly. To overcome the bottleneck of data scarcity, researchers have attempted to introduce semi-supervised learning techniques. However, existing semi-supervised methods face two major challenges when applied to IQA tasks: first, as a regression task, IQA is difficult to effectively filter out false labels using the probability confidence of the model output, unlike classification tasks; second, traditional strong data augmentation (such as blurring and adding noise) directly distorts the true perceived quality of the image, causing label shifts and failures in the consistency learning mechanism. Summary of the Invention

[0004] In view of the above problems, this application provides a sorting-constrained semi-supervised image quality assessment method, system, and storage medium, aiming to solve the problems of lack of subjectively labeled data in real-world scenarios, and the difficulty of pseudo-label screening and easy alteration of quality attributes by traditional semi-supervised methods in IQA tasks. By designing pseudo-label filtering and graph-patch consistency aggregation modules based on ordered centralization, and a global-attribute inverse pairing mechanism based on a cross-modal visual-language large model, based on the synergistic effect of multiple modules, more efficient semi-supervised consistency learning can be achieved without changing the visual quality attributes of the image, providing an accurate quality perception benchmark for intelligent monitoring and optimization of image quality in complex real-world scenarios.

[0005] According to one aspect of the embodiments of this application, a semi-supervised image quality assessment method with ranking constraints is provided, the method comprising: A two-branch teacher-student collaborative learning network based on the contrastive language-image pre-trained model CLIP is constructed to reshape the continuous scalar image quality regression task into a discrete ordered probability distribution prediction task; the CLIP-based two-branch teacher-student collaborative learning network includes a supervised branch and an unsupervised branch trained simultaneously. In the supervised branch, an image with rating labels is input into the student network, and the visual features of the image and the text features of the preset quality level are extracted to obtain the predicted quality probability distribution of the image. The supervised loss is calculated based on the difference between the true quality distribution and the predicted quality probability distribution. In the unsupervised branch, weak enhancement processing is performed on unlabeled images to obtain weakly enhanced images. The weak enhancement processing includes random horizontal flipping and random cropping, without changing the perceptual quality of the image. The weakly enhanced images are input into the teacher network to obtain the initial prediction quality probability distribution at the tile level. The initial prediction quality probability distribution at the tile level is then subjected to ordered centralized filtering verification, block quality distribution aggregation, and image quality distribution caching processing to obtain the global image-level pseudo-label distribution. The original image blocks of the same unlabeled image are input into the student network. At the same time, a global-attribute inverse pairing enhancement strategy is adopted to generate inverse text features with inverted hierarchy in the cross-modal text domain. These features, together with the visual features of the original image blocks, constitute hard samples. The unsupervised loss is calculated based on the difference between the global image-level pseudo-label distribution and the predicted quality probability distribution output by the student network on the difficult samples; the total loss function is constructed based on the supervised loss and the unsupervised loss, and the network parameters of the student network are iteratively optimized through backpropagation; After training convergence, the image to be evaluated is input into the optimized student network, which outputs the corresponding quality probability distribution. The final perceived quality score of the image is calculated based on the quality probability distribution.

[0006] In one alternative approach, in the supervised branch, an image with rating labels is input into a student network, and visual features of the image and text features of a preset quality level are extracted to obtain the predicted quality probability distribution of the image, including: Settings include An ordered quality level space of each quality level, corresponding to the generation Quality level prompt text; The image encoder of CLIP extracts features from the input image patches to obtain visual feature vectors. The CLIP text encoder extracts features from the text prompts for each quality level to obtain a sequence of text feature vectors. Based on the visual feature vectors and the text feature vectors, calculate the cosine similarity score. After normalization using the Softmax function, the prediction quality probability distribution is obtained. .

[0007] In one alternative approach, the formula for calculating the final perceived quality score is: (1) in, Representing the The scalar score corresponding to each quality level To predict the probability distribution of quality of the th The probability value corresponding to each quality level The final perceived quality score.

[0008] In one alternative approach, the method of performing ordered centralized filtering and verification on the initial prediction quality probability distribution at the tile level includes: Probability distribution of initial prediction quality at the tile level for teacher network output If a peak index exists If the probability sequence satisfies the unimodality check condition, then the quality probability distribution is determined to satisfy the unimodality constraint. The unimodality check condition is: (2) Under the premise of satisfying the unimodality constraint, the probability values ​​in the initial prediction quality probability distribution are rearranged in descending order to obtain a descending sequence before calculation. The sum of the maximum probability values To quantify the concentration of the said quality probability distribution, the summation... The calculation formula is: (3) in, This represents the position index of the probability sequence after descending order. To sort all probability values ​​in the initial prediction quality probability distribution in descending order, the th... The probability value corresponding to the bit; The initial prediction quality probability distribution is determined to be a valid candidate pseudo-label distribution if and only if it simultaneously satisfies the following two conditions. : Condition 1: The initial predicted quality probability distribution satisfies the unimodality verification condition shown in equation (2); Condition 2: The sum of the two highest probabilities Or the sum of the three highest probabilities ,in and These are the preset hyperparameters.

[0009] In one alternative approach, the method of obtaining the global image-level pseudo-label distribution through block quality distribution aggregation and image quality distribution caching includes: The state tuple of each unlabeled image is dynamically maintained throughout the training process using an image quality distribution cache. The state tuples record the unique identifier of the image. Current cumulative global quality probability distribution and the number of valid tiles participating in the aggregation ; In each valid forward pass, a weighted moving average strategy is employed through the block quality distribution aggregator, based on the currently acquired distribution of valid candidate pseudo-labels. With the current cumulative global quality probability distribution The cumulative global quality probability distribution is updated using the following formula: (4) in, This represents the cumulative global quality probability distribution for the updated b-th unlabeled image. The cumulative number of valid tiles for the updated b-th unlabeled image; Preset minimum cumulative tile threshold Based on the updated cumulative number of valid tiles With minimum cumulative tile threshold The comparison results reveal the global image-level pseudo-label distribution. The release condition is: (5) in, To indicate that there is no valid output, meaning the current cumulative number of valid patches has not reached the preset threshold, no global image-level pseudo-labels that can be used for training are generated, and the corresponding samples are not included in the unsupervised loss calculation. For all other cases, i.e., not satisfied All scenarios.

[0010] In one alternative approach, the global-attribute reverse pairing enhancement strategy includes: Pre-built using a large language model including Fine-grained quality dimensions Rank attribute hint matrix ; During consistency training, random sampling without replacement is performed. Column attributes constitute the prior submatrix The inverse attribute matrix is ​​calculated using the following formula. : (6) in, for The reverse commutation matrix of order, the reverse commutation matrix The secondary diagonal elements are all 1s, and the rest are all 0s; The inverse attribute matrix The corresponding text features are input into the student network and forcibly matched with the visual features of the original image patch.

[0011] In an alternative approach, the total loss function is expressed as: (7) in, For the total loss, To balance hyperparameters, To monitor losses, Loss due to lack of supervision; The monitoring loss The calculation formula is: (8) in, For sample index, The number of samples in a batch for supervised branching. Denotes KL divergence, Let b be the true quality distribution of the b-th labeled sample. Let be the probability distribution of the predicted quality output of the student network for the b-th labeled sample; Based on the global image-level pseudo-label distribution Output distribution of the student network on the strongly reinforced branch Calculate the unsupervised loss based on the KL divergence between the two: (9) in, The number of batch samples for the unsupervised branch. For indicator functions, when The value is 1 if the condition is met, and 0 otherwise. The total loss also includes Spe-SSIM loss. L Spe-SSIM The calculation formula is: (10) in, C This represents the total number of feature channels. The first output of the encoder Channel characteristics, For the corresponding reconstruction of the first Channel characteristics, The structural similarity function is represented; the Spe-SSIM loss is used as the total loss. Components of.

[0012] According to another aspect of the embodiments of this application, a semi-supervised image quality assessment system with ranking constraints is provided for implementing the method described above, the system comprising: The network construction module is configured to construct a two-branch teacher-student collaborative learning network based on the contrastive language-image pre-trained model CLIP, which reshapes the continuous scalar image quality regression task into a discrete ordered probability distribution prediction task; the CLIP-based two-branch teacher-student collaborative learning network includes a supervised branch and an unsupervised branch trained simultaneously. The supervised processing module is configured to input an image with a rating label into the student network in the supervised branch, extract the visual features of the image and the text features of a preset quality level, obtain the predicted quality probability distribution of the image, and calculate the supervised loss based on the difference between the true quality distribution and the predicted quality probability distribution. An unsupervised processing module is configured to perform weak enhancement processing on unlabeled images in the unsupervised branch to obtain weakly enhanced images. The weak enhancement processing includes random horizontal flipping and random cropping, without changing the perceptual quality of the image. The weakly enhanced image is input into the teacher network to obtain a tile-level initial prediction quality probability distribution. The tile-level initial prediction quality probability distribution is then subjected to ordered centralized filtering verification, block quality distribution aggregation, and image quality distribution caching processing to obtain a global image-level pseudo-label distribution. The original image blocks of the same unlabeled image are input into the student network, and a global-attribute inverse pairing enhancement strategy is adopted to generate inverse text features with inverted hierarchy in the cross-modal text domain. These features, together with the visual features of the original image blocks, constitute hard samples. The iterative optimization module is configured to calculate the unsupervised loss based on the difference between the global image-level pseudo-label distribution and the predicted quality probability distribution output by the student network on the difficult samples; construct a total loss function based on the supervised loss and the unsupervised loss; and backpropagate to iteratively optimize the network parameters of the student network. The quality evaluation module is configured to input the image to be evaluated into the optimized student network after training convergence, output the corresponding quality probability distribution, and calculate the final perceived quality score of the image based on the quality probability distribution.

[0013] According to one aspect of the embodiments of this application, an electronic device is provided, including: a controller; and a memory for storing one or more programs, which, when executed by the controller, perform the method described above.

[0014] According to one aspect of the embodiments of this application, a computer-readable storage medium is also provided, on which computer-readable instructions are stored, which, when executed by a computer's processor, cause the computer to perform the above-described method.

[0015] According to one aspect of the embodiments of this application, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described above.

[0016] This application has at least the following beneficial effects: This application reshapes the regression task of continuous scalars into the prediction of discrete ordered probability distributions, and constructs a two-branch teacher-student collaborative learning network based on the CLIP model, which can effectively maintain the basic quality perception capability of the model under the condition of scarce subjectively labeled data. In the unsupervised branch, an ordered centralized filtering mechanism combining unimodal verification, accumulation of the top k maximum probabilities, and threshold judgment is adopted for weakly enhanced images, which can overcome the defect of traditional semi-supervised methods that are difficult to screen high-quality pseudo-labels due to the lack of probability confidence. Furthermore, through the map-patch consistency aggregation module, a weighted moving average strategy is used to smoothly integrate the effective map-patch level observation distribution into the historical accumulation. The global distribution can eliminate semantic bias between local and global perceived quality, ensuring the reliability of global pseudo-labels. At the same time, a global-attribute inverse pairing enhancement strategy is applied to the original image patches to achieve rank inversion on the attribute cue matrix constructed by the large language model. This can complete feature-level strong perturbation regularization without changing pixel quality, avoiding the tampering of quality labels by traditional pixel-domain strong enhancement. In addition, by combining the joint optimization of KL divergence supervised loss, unsupervised loss and Spe-SSIM frequency domain structural similarity loss, the accuracy and generalization robustness of no-reference image quality prediction can be significantly improved with very little labeled data.

[0017] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of this application are described below. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0019] Figure 1 This is a network framework diagram illustrating a semi-supervised image quality assessment method for implementing ranking constraints, as shown in an exemplary embodiment of this application.

[0020] Figure 2 This is a flowchart illustrating a sorting-constrained semi-supervised image quality assessment method according to an exemplary embodiment of this application.

[0021] Figure 3 This is a flowchart illustrating a CLIP-based quality probability distribution prediction process in an exemplary embodiment of this application.

[0022] Figure 4 This is a structural diagram of a sorting-constrained semi-supervised image quality assessment system illustrated in an exemplary embodiment of this application. Detailed Implementation

[0023] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of systems and methods consistent with some aspects of this application as detailed in the appended claims.

[0024] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor systems and / or microcontroller systems.

[0025] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0026] In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0027] One aspect of this application provides a semi-supervised image quality assessment method with ranking constraints. In this embodiment, the method utilizes semi-supervised deep learning and a multimodal visual-language large model in real-world distortion scenarios. When faced with the dilemma of extremely scarce subjective quality annotation data, it achieves accurate prediction of perceived quality of unreferenced images by constructing ordered probability distribution filtering, tile consistency alignment, and cross-modal semantic reverse pairing mechanisms.

[0028] like Figure 1 The diagram shown is a network framework diagram of a semi-supervised image quality assessment method for implementing ranking constraints, as illustrated in an exemplary embodiment of this application. The network framework, named RevMatch, adopts a teacher-student collaborative learning architecture based on the Contrastive Language-Image Pre-trained Model (CLIP), including parallel supervised and unsupervised branches. The unsupervised branch further includes a weak enhancement branch corresponding to the teacher network and a strong enhancement branch corresponding to the student network. Figure 1 The right side also shows the internal calculation process of the quality probability distribution prediction module used by both the teacher network and the student network.

[0029] The entire network uses a contrastive language-image pre-trained model as its basic feature extraction unit. Both the teacher and student networks include an image encoder (IE) and a text encoder (TE). During training, the text encoder parameters of both the teacher and student networks are frozen to preserve the original semantic priors of the pre-trained model. The image encoder parameters of the student network are updated using backpropagation based on the training loss, while the image encoder parameters of the teacher network are updated using an exponential moving average strategy based on the image encoder parameters of the student network. An ordered centralization-based filtering module, a tile quality distribution aggregator (PQDA), and an image quality distribution cache (IQDC) are used to perform pseudo-label filtering, distribution aggregation, and state caching, but do not participate in gradient backpropagation.

[0030] Supervised branches are located in Figure 1 The lower part is used to establish a basic image quality-perceived mapping relationship using a small number of labeled images. The supervised branch uses labeled images x with subjective quality annotations. bAs input, quality-preserving visual enhancements, including random cropping and random horizontal flipping, are performed on the labeled image to obtain visual input v. Simultaneously, a set h of overall quality cues is constructed, arranged from low to high quality levels, corresponding to five ordered quality levels: "Poor," "Fairly Poor," "Average," "Good," and "Excellent."

[0031] Visual input v and a set of holistic quality cues h are input into the student network. Visual features and text features are extracted using an image encoder and a text encoder, respectively. Figure 1 The quality probability distribution prediction module shown on the right obtains the predicted quality probability distribution. The predicted quality probability distribution With the corresponding true quality probability distribution By comparison, the supervision loss is obtained. This is to constrain the basic quality prediction capabilities of student networks.

[0032] Unsupervised branches are located in Figure 1 The upper part is used to utilize the quality information contained in the unlabeled image. For unlabeled images... u b Two independent quality-preserving visual enhancements were performed to obtain weakly enhanced visual input. v w Visual input of the enhanced branch v s The visual enhancements for both branches include random cropping and random horizontal flipping, without employing blurring, noise injection, or distortion synthesis operations that would alter the perceived quality of the image; the strength difference between the two branches is primarily determined by the construction method of the text prompts.

[0033] The weakly enhanced branch corresponds to the input path of the teacher network. The weakly enhanced visual input... v w Combined with the set h of overall quality cues arranged in normal quality order, this constitutes a weakly enhanced multimodal input: Weakly enhanced multimodal input Enter the teacher network, through Figure 1 The quality probability distribution prediction module shown on the right performs forward inference to obtain the initial predicted quality distribution corresponding to the current tile. .

[0034] Subsequently, the initial predicted quality distribution The input is a filtering module based on ordered centralization. This filtering module sequentially performs unimodality checks and sums on the initial predicted quality distribution. A threshold check based on the sum of the maximum probabilities is used to filter predictions that do not satisfy an ordered distribution structure or have insufficient probability concentration, thus obtaining an effective candidate pseudo-label distribution. .

[0035] Distribution of effective candidate pseudo-labels The input is the Patch Quality Distribution Aggregator (PQDA), which is updated in conjunction with the historical states corresponding to the current unlabeled image identifiers in the Image Quality Distribution Cache (IQDC). These historical states include at least the cumulative image quality probability distribution. Cumulative number of valid tiles The tile quality distribution aggregator uses a weighted moving average method to merge the current distribution of valid candidate pseudo-labels into the cumulative image quality probability distribution of the corresponding image, and updates the cumulative number of valid tiles simultaneously.

[0036] When the cumulative number of valid tiles meets the preset threshold condition: Image-level pseudo-label distribution after image quality distribution buffer output aggregation This serves as a pseudo-label supervision signal for unsupervised training. When the cumulative number of valid patches does not reach a preset threshold, no image-level pseudo-labels are output for consistency training, and the current sample does not participate in the unsupervised loss calculation. Therefore, by aggregating multiple local patch predictions from the same image across different training rounds, the impact of single patch prediction bias on the reliability of pseudo-labels is reduced.

[0037] The enhanced branch corresponds to the input path of the student network. At least one quality attribute is randomly selected from a pre-constructed attribute-level quality cue matrix, and cuees for the selected attributes at the same quality level are integrated to obtain a random attribute-level quality cue set. Subsequently, a reverse transformation is performed on the random attribute-level quality cue set along the quality level dimension to obtain a reverse attribute cue set W. rev The quality attributes include at least one of noise, contrast, color, exposure, detail, artifacts, distortion, naturalness, viewing comfort, and aesthetic impression.

[0038] Visual input v s With the set of reverse attribute hints W rev Combining them creates a strongly enhanced multimodal input: Enhance multimodal input Enter the student network, through Figure 1 The quality probability distribution prediction module shown on the right performs forward inference to obtain the predicted quality probability distribution under strong enhancement conditions. When the image quality distribution buffer has already output the corresponding image-level pseudo-label distribution. At that time, according to and Calculate unsupervised loss based on the distribution differences between them. .

[0039] The global-attribute inverse pairing enhancement method avoids strong visual enhancements that alter image perceptual quality. Instead, it constructs cross-modal hard samples by setting attribute cues with opposite quality level relationships in the text modality. This approach generates strong consistency perturbations while preserving the basic semantics of image quality, suppressing the student network's shortcut dependence on text quality level semantics, and prompting it to extract more stable quality-perceptual features from the image.

[0040] Figure 1 The quality probability distribution prediction module shown on the right is an internal computation module shared by the supervised branch, weak enhancement branch, and strong enhancement branch. For the input image, the visual feature vector is first extracted through the image encoder (IE). For the prompt text corresponding to the five ordered quality levels, text features are extracted using the frozen text encoder (TE) to obtain the text feature vectors corresponding to the five quality levels: Specifically, when the input is a set of holistic quality cues h, the five holistic quality cues are encoded to obtain five text feature vectors; when the input is a set of inverse attribute cues W... rev At that time, the selected attribute prompts under each quality level were encoded, and the text features of multiple attributes corresponding to the same quality level were integrated to obtain five integrated text feature vectors.

[0041] Visual feature vectors Cosine similarity is calculated between the text feature vectors corresponding to the five quality levels, resulting in a similarity score sequence: Among them, the The similarity score indicates the similarity between the input image and the first image. The degree of matching between the quality level prompts. Subsequently, the five similarity scores were normalized using the Softmax function to obtain the quality probability distribution: D= The five probability values ​​correspond to five ordered quality levels, from "poor" to "excellent." Depending on their application, these quality probability distributions represent the predicted quality probability distributions for supervised branches. Initial prediction quality distribution of weakly enhanced branches Or the probability distribution of the prediction quality of strongly enhanced branches .

[0042] During the overall training process, the total training loss of the network is composed of a weighted average of supervised loss and unsupervised loss: in, The weights represent the unsupervised loss. The image encoder parameters of the student network are updated via backpropagation based on the total training loss, and the image encoder parameters of the teacher network are updated synchronously using an exponential moving average strategy until the network training converges. The trained quality assessment network can output an ordered quality probability distribution of the test image based on the overall quality cues between the test image and the normal sequence, and determine the perceptual quality prediction result of the test image based on this probability distribution.

[0043] Specifically, such as Figure 2 As shown, the semi-supervised image quality assessment method with ranking constraints includes the following steps S10-S50.

[0044] S10: Construct a two-branch teacher-student collaborative learning network based on the contrastive language-image pre-trained model CLIP, reshaping the continuous scalar image quality regression task into a discrete ordered probability distribution prediction task; the CLIP-based two-branch teacher-student collaborative learning network includes a supervised branch and an unsupervised branch trained synchronously.

[0045] This dual-branch teacher-student collaborative learning network reshapes the traditional continuous scalar score regression task into discrete ordered probability distribution prediction. During the training phase, the supervised branch uses labeled data to maintain the basic visual-text quality mapping; the unsupervised branch utilizes unlabeled data to enter asymmetric strong and weak augmentation branches. The weak augmentation branch generates pseudo-labels through the teacher model and uses the Image Quality Distribution Cache (IQDC) and Block Quality Distribution Aggregation (PQDA) modules for purification and aggregation; the strong augmentation branch creates hard samples by applying inverse pairing operations to the multimodal text end, guiding the student network's consistency learning.

[0046] S20: In the supervised branch, the image with rating labels is input into the student network, the visual features of the image and the text features of the preset quality level are extracted to obtain the predicted quality probability distribution of the image, and the supervised loss is calculated based on the difference between the true quality distribution and the predicted quality probability distribution.

[0047] In one exemplary implementation, such as Figure 3 As shown, visual and textual features are extracted based on CLIP to predict the quality distribution. The quality level space is defined as a set. The levels correspond to five grades: "bad," "poor," "fair," "good," and "excellent." For an input image patch, CLIP's image encoder extracts visual feature vectors. CLIP's text encoder extracts text feature vector sequences for five quality level cues. Subsequently, the cosine similarity score between image features and text features is calculated. The similarity scores are normalized using the Softmax function to obtain a prediction quality probability distribution with a sum of 1. .

[0048] In one exemplary implementation, the final quality prediction score of the image This can be obtained by calculating the mathematical expectation of the distribution: (1) in, Representing the The scalar score corresponding to each quality level To predict the probability distribution of quality of the th The probability value corresponding to each quality level This is used to determine the final perceived quality score.

[0049] S30: In the unsupervised branch, weak enhancement processing is performed on the unlabeled image to obtain a weakly enhanced image; the weak enhancement processing includes random horizontal flipping and random cropping, without changing the perceptual quality of the image; the weakly enhanced image is input into the teacher network to obtain the initial prediction quality probability distribution at the tile level, and the initial prediction quality probability distribution at the tile level is sequentially subjected to ordered centralized filtering verification, block quality distribution aggregation and image quality distribution caching processing to obtain the global image-level pseudo-label distribution; the original image blocks of the same unlabeled image are input into the student network, and at the same time, the global-attribute inverse pairing enhancement strategy is adopted to generate inverse text features with inverted hierarchy in the cross-modal text domain, which together with the visual features of the original image blocks constitute hard samples.

[0050] In one exemplary implementation, to overcome the problem of lack of confidence in continuous regression tasks, a joint verification mechanism based on the statistical characteristics of human subjective scoring is proposed. This joint verification mechanism can be executed by a pseudo-label filtering module based on ordered centralization.

[0051] Specifically, regarding the predicted distribution of teacher network outputs First, a unimodality check is performed. If a peak index exists... , so that the probability sequence is If a peak value is reached at a certain point and then decreases monotonically on both sides, it is considered to satisfy the unimodality property. (2) While maintaining the unimodality, the concentration of the distribution is further quantified. The probability values ​​are then arranged in descending order to obtain a descending sequence. Before definition The sum of the maximum probability values for: (3) in, This represents the position index of the probability sequence after descending order. To sort all probability values ​​in the initial prediction quality probability distribution in descending order, the th... The probability value corresponding to the bit.

[0052] If and only if the distribution simultaneously and completely satisfies the unimodal structure constraint and the threshold condition. or hour( and (As a preset hyperparameter), this distribution is recognized by the system as a valid candidate pseudo-label with a high degree of consensus. .

[0053] In one exemplary implementation, a graph-patch consistency distribution aggregation module (PQDA and IQDC) is designed to address the semantic discrepancy between local regions and overall objective perceived quality in high-resolution images. This module constructs a long-term memory closed loop through PQDA and IQDC components. IQDC dynamically maintains the state tuples of unlabeled samples throughout the training process. Each element records the unique image identifier ID, the current cumulative global distribution, and the number of valid tiles participating in the aggregation.

[0054] In each valid forward pass, the PQDA module uses a weighted moving average strategy to distribute the latest valid local observations. Smoothly integrate into the historical global distribution of the image: (4) in, This represents the cumulative global quality probability distribution for the updated b-th unlabeled image. This represents the cumulative number of valid tiles for the updated b-th unlabeled image.

[0055] Because a very small number of early image patches may have severe extreme local feature deviations, the system sets a minimum cumulative patch threshold for global stability release control. IQDC will only be officially unlocked and activated when the effective accumulated amount crosses this threshold. Upgraded to reliable global image-level pseudo-labels for output to student networks : (5) in, To indicate that there is no valid output, meaning the current cumulative number of valid patches has not reached the preset threshold, no global image-level pseudo-labels that can be used for training are generated, and the corresponding samples are not included in the unsupervised loss calculation. For all other cases, i.e., not satisfied All scenarios.

[0056] In one exemplary implementation, a specific implementation process for a global-attribute inverse pairing enhancement strategy is provided. This embodiment innovatively breaks through the limitations of traditional visual data augmentation in altering quality labels, transferring the strongly perturbed adversarial region across the visual pixel feature domain to the text multimodal domain. First, a large language model (LLM) is constructed containing... A fine-grained quality dimension (covering sharpness, color reproduction, aesthetic impression, etc.) Rank attribute hint matrix .

[0057] When handling consistency training for unlabeled branches, the system initiates a global-attribute inverse pairing engine for the student model. First, random sampling without replacement is performed. Column attributes constitute the prior submatrix And through mathematical matrix transformation, multiply by a matrix on the left. reverse commutation matrix of order (Its secondary diagonal is all 1, and all other elements are 0), achieving an absolute inversion of the corresponding quality level in the multidimensional fine-grained attribute space: (6) The erroneous text features, after being reversed and inverted, are forcibly matched and input into the student network. By creating this cognitively defying and rigorous multimodal semantic tearing, extremely high-intensity feature-level consistency regularization pressure is successfully achieved without altering the quality of any pixel images.

[0058] S40: Calculate the unsupervised loss based on the difference between the global image-level pseudo-label distribution and the predicted quality probability distribution output by the student network on the difficult samples; construct the total loss function based on the supervised loss and the unsupervised loss, and iteratively optimize the network parameters of the student network through backpropagation.

[0059] The overall training objective is to minimize the total loss. This loss is due to the supervision loss of the labeled data. Unsupervised loss of unlabeled data Together they constitute: (7) in It is a trade-off hyperparameter used to adjust the relative contribution of unsupervised components, and was empirically set to 5.0 in all experiments.

[0060] Monitoring losses Aiming to minimize the true mass distribution Student model prediction The difference between them. We use KL divergence as the optimization metric: (8) in, For sample index, The number of samples in a batch for supervised branching. Denotes KL divergence, For the first The true quality distribution of a labeled sample For student network to the first The probability distribution of the predicted quality of each labeled sample output.

[0061] Student model in Make predictions and output the distribution. Unsupervised loss The calculation formula is as follows: (9) in, The number of batch samples for the unsupervised branch. For indicator functions, when The value is 1 if it is true, and 0 otherwise.

[0062] To quantify the structural similarity in the frequency domain, this embodiment introduces the Spe-SSIM loss. To quantify the structural similarity in the frequency domain, this embodiment transforms the predicted image and the reference image to the frequency domain, and constructs the Spe-SSIM loss based on the structural similarity of their spectral features: (10) in, C This represents the total number of feature channels. The first output of the encoder Channel characteristics, For the corresponding reconstruction of the first Channel characteristics, The structural similarity function is represented; the Spe-SSIM loss is used as the total loss. Components of.

[0063] S50: After training converges, the image to be evaluated is input into the optimized student network, and the corresponding quality probability distribution is output. The final perceived quality score of the image is calculated based on the quality probability distribution.

[0064] In step S50, after the model training converges, for any input image to be evaluated, such as sensor noise images generated during image acquisition, distorted images with block artifacts introduced after lossy compression transmission, blurred images captured under low light conditions, and trailing blurred images generated in motion scenes, it is only necessary to send it into the optimized student network for one forward propagation to output the corresponding quality probability distribution. Based on this distribution, the corresponding final perceived quality score is calculated using equation (1). This final perceived quality score can be used for quality monitoring of image acquisition equipment, parameter tuning of image compression algorithms, comparison of effects before and after image enhancement processing, and dynamic adjustment of image quality in video streaming transmission, etc. It should be noted that the sources and application scenarios of the above-mentioned images to be evaluated are only illustrative examples and do not constitute a limitation on the scope of application of this application.

[0065] The following section will mainly introduce the dataset sources, training environment, optimizer settings, and evaluation metrics during the training phase.

[0066] This embodiment uses the widely accepted real-world distortion dataset KonIQ-10k. In standard semi-supervised experiments, 25% of the data in the training set is used as labeled data, and the remainder as unlabeled data. All input images are first scaled to a resolution of 256×256, and then randomly cropped into 224×224 tiles before being fed into the network.

[0067] The network model in this embodiment is implemented using the PyTorch deep learning framework and runs on an NVIDIA GeForce RTX 3090 GPU. The network model uses the AdamW optimizer with an initial learning rate of 7×10⁻⁶. -6 It follows the cosine annealing strategy for scheduling.

[0068] This embodiment uses Spearman's rank correlation coefficient (SRCC) and Pearson's linear correlation coefficient (PLCC) to measure network performance. Generally, the closer the SRCC and PLCC values ​​are to 1, the better the consistency between the model's predictions and human subjective perception.

[0069] Experimental results of the network proposed in this embodiment show that, on the authoritative KonIQ-10k benchmark dataset (25% labeled), the model in this application achieves an SRCC index of 0.892 and a PLCC index of 0.918, significantly outperforming existing mainstream semi-supervised image quality assessment methods. Furthermore, in the extreme low-resource limit test using only 2% labeled data, this application significantly improves the SRCC from 0.810 to 0.841 compared to the purely supervised baseline model, demonstrating extremely strong data mining capabilities and robustness.

[0070] Another aspect of this application provides a semi-supervised image quality assessment system with ranking constraints, such as... Figure 4 As shown, the system includes: The network construction module 401 is configured to construct a two-branch teacher-student collaborative learning network based on the contrastive language-image pre-trained model CLIP, which reshapes the continuous scalar image quality regression task into a discrete ordered probability distribution prediction task; the CLIP-based two-branch teacher-student collaborative learning network includes a supervised branch and an unsupervised branch trained simultaneously. The supervised processing module 402 is configured to input an image with a rating label into the student network in the supervised branch, extract the visual features of the image and the text features of a preset quality level, obtain the predicted quality probability distribution of the image, and calculate the supervised loss based on the difference between the true quality distribution and the predicted quality probability distribution. The unsupervised processing module 403 is configured to perform weak enhancement processing on the unlabeled image in the unsupervised branch to obtain a weakly enhanced image. The weak enhancement processing includes random horizontal flipping and random cropping, without changing the perceptual quality of the image. The weakly enhanced image is input into the teacher network to obtain a tile-level initial prediction quality probability distribution. The tile-level initial prediction quality probability distribution is then subjected to ordered centralized filtering verification, block quality distribution aggregation, and image quality distribution caching processing to obtain a global image-level pseudo-label distribution. The original image blocks of the same unlabeled image are input into the student network. At the same time, a global-attribute inverse pairing enhancement strategy is adopted to generate inverse text features with inverted hierarchy in the cross-modal text domain. These features, together with the visual features of the original image blocks, constitute hard samples. The iterative optimization module 404 is configured to calculate the unsupervised loss based on the difference between the global image-level pseudo-label distribution and the predicted quality probability distribution output by the student network on the difficult samples; construct a total loss function based on the supervised loss and the unsupervised loss; and backpropagate to iteratively optimize the network parameters of the student network. The quality evaluation module 405 is configured to input the image to be evaluated into the optimized student network after training convergence, output the corresponding quality probability distribution, and calculate the final perceived quality score of the image based on the quality probability distribution.

[0071] It should be noted that the system provided in the above embodiments and the method provided in the foregoing embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here.

[0072] Another aspect of this application provides an electronic device, including: a controller; and a memory for storing one or more programs, which, when executed by the controller, perform the methods described above.

[0073] Another aspect of this application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0074] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various embodiments described above.

[0075] According to one aspect of the embodiments of this application, a computer system is also provided, including a Central Processing Unit (CPU), which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) or a program loaded from storage into random access memory (RAM), such as performing the methods described above. Various programs and data required for system operation are also stored in the RAM. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0076] The following components are connected to the I / O interface: input components including keyboards, mice, etc.; output components including cathode ray tubes (CRTs), liquid crystal displays (LCDs), and speakers; storage components including hard drives; and communication components including network interface cards such as LAN (Local Area Network) cards and modems. The communication components perform communication processing via networks such as the Internet. Drives are also connected to the I / O interface as needed. Removable media, such as disks, optical discs, magneto-optical discs, semiconductor memories, etc., are installed on the drive as needed so that computer programs read from them can be installed into the storage components as required.

[0077] The above description is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

Claims

1. A semi-supervised image quality assessment method with ranking constraints, characterized in that, include: A two-branch teacher-student collaborative learning network based on the contrastive language-image pre-trained model CLIP is constructed to reshape the continuous scalar image quality regression task into a discrete ordered probability distribution prediction task; the CLIP-based two-branch teacher-student collaborative learning network includes a supervised branch and an unsupervised branch trained simultaneously. In the supervised branch, an image with rating labels is input into the student network, and the visual features of the image and the text features of the preset quality level are extracted to obtain the predicted quality probability distribution of the image. The supervised loss is calculated based on the difference between the true quality distribution and the predicted quality probability distribution. In the unsupervised branch, weak enhancement processing is performed on unlabeled images to obtain weakly enhanced images. The weak enhancement processing includes random horizontal flipping and random cropping, without changing the perceptual quality of the image. The weakly enhanced images are input into the teacher network to obtain the initial prediction quality probability distribution at the tile level. The initial prediction quality probability distribution at the tile level is then subjected to ordered centralized filtering verification, block quality distribution aggregation, and image quality distribution caching processing to obtain the global image-level pseudo-label distribution. The original image blocks of the same unlabeled image are input into the student network. At the same time, a global-attribute inverse pairing enhancement strategy is adopted to generate inverse text features with inverted hierarchy in the cross-modal text domain. These features, together with the visual features of the original image blocks, constitute hard samples. The unsupervised loss is calculated based on the difference between the global image-level pseudo-label distribution and the predicted quality probability distribution output by the student network on the difficult samples; the total loss function is constructed based on the supervised loss and the unsupervised loss, and the network parameters of the student network are iteratively optimized through backpropagation; After training convergence, the image to be evaluated is input into the optimized student network, which outputs the corresponding quality probability distribution. The final perceived quality score of the image is calculated based on the quality probability distribution.

2. The semi-supervised image quality assessment method with ranking constraints according to claim 1, characterized in that, In the supervised branch, images with rating labels are input into the student network. Visual features of the images and text features of preset quality levels are extracted to obtain the predicted quality probability distribution of the images, including: Settings include An ordered quality level space of each quality level, corresponding to the generation Quality level prompt text; The image encoder of CLIP extracts features from the input image patches to obtain visual feature vectors. The CLIP text encoder extracts features from the text prompts for each quality level to obtain a sequence of text feature vectors. Based on the visual feature vectors and the text feature vectors, calculate the cosine similarity score. After normalization using the Softmax function, the prediction quality probability distribution is obtained. .

3. The semi-supervised image quality assessment method with ranking constraints according to claim 1, characterized in that, The formula for calculating the final perceived quality score is as follows: (1) in, Representing the The scalar score corresponding to each quality level To predict the probability distribution of quality of the th The probability value corresponding to each quality level The final perceived quality score.

4. The semi-supervised image quality assessment method with ranking constraints according to claim 1, characterized in that, The methods for performing ordered centralized filtering and verification of the initial prediction quality probability distribution at the tile level include: Probability distribution of initial prediction quality at the tile level for teacher network output If a peak index exists If the probability sequence satisfies the unimodality check condition, then the quality probability distribution is determined to satisfy the unimodality constraint. The unimodality check condition is: (2) Under the premise of satisfying the unimodality constraint, the probability values ​​in the initial prediction quality probability distribution are rearranged in descending order to obtain a descending sequence before calculation. The sum of the maximum probability values To quantify the concentration of the said quality probability distribution, the summation... The calculation formula is: (3) in, This represents the position index of the probability sequence after descending order. To sort all probability values ​​in the initial prediction quality probability distribution in descending order, the th... The probability value corresponding to the bit; The initial prediction quality probability distribution is determined to be a valid candidate pseudo-label distribution if and only if it simultaneously satisfies the following two conditions. : Condition 1: The initial predicted quality probability distribution satisfies the unimodality verification condition shown in equation (2); Condition 2: The sum of the two highest probabilities Or the sum of the three highest probabilities ,in and These are the preset hyperparameters.

5. The semi-supervised image quality assessment method with ranking constraints according to claim 3, characterized in that, Methods for obtaining global image-level pseudo-label distributions through block quality distribution aggregation and image quality distribution caching include: The state tuple of each unlabeled image is dynamically maintained throughout the training process using an image quality distribution cache. The state tuples record the unique identifier of the image. Current cumulative global quality probability distribution and the number of valid tiles participating in the aggregation ; In each valid forward pass, a weighted moving average strategy is employed through the block quality distribution aggregator, based on the currently acquired distribution of valid candidate pseudo-labels. With the current cumulative global quality probability distribution The cumulative global quality probability distribution is updated using the following formula: (4) in, This represents the cumulative global quality probability distribution for the updated b-th unlabeled image. The cumulative number of valid tiles for the updated b-th unlabeled image; Preset minimum cumulative tile threshold Based on the updated cumulative number of valid tiles With minimum cumulative tile threshold The comparison results reveal the global image-level pseudo-label distribution. The release condition is: (5) in, To indicate that there is no valid output, meaning the current cumulative number of valid patches has not reached the preset threshold, no global image-level pseudo-labels that can be used for training are generated, and the corresponding samples are not included in the unsupervised loss calculation. For all other cases, i.e., not satisfied All scenarios.

6. The semi-supervised image quality assessment method with ranking constraints according to claim 1, characterized in that, The whole-attribute reverse pairing enhancement strategy includes: Pre-built using a large language model including Fine-grained quality dimensions Rank attribute hint matrix ; During consistency training, random sampling without replacement is performed. Column attributes constitute the prior submatrix The inverse attribute matrix is ​​calculated using the following formula. : (6) in, for The reverse commutation matrix of order, the reverse commutation matrix The secondary diagonal elements are all 1s, and the rest are all 0s; The inverse attribute matrix The corresponding text features are input into the student network and forcibly matched with the visual features of the original image patch.

7. The semi-supervised image quality assessment method with ranking constraints according to claim 5, characterized in that, The total loss function is expressed as: (7) in, For the total loss, To balance hyperparameters, To monitor losses, Loss due to lack of supervision; The monitoring loss The calculation formula is: (8) in, For sample index, The number of samples in a batch for supervised branching. Denotes KL divergence, Let b be the true quality distribution of the b-th labeled sample. Let be the probability distribution of the predicted quality output of the student network for the b-th labeled sample; Based on the global image-level pseudo-label distribution Output distribution of the student network on the strongly reinforced branch Calculate the unsupervised loss based on the KL divergence between the two: (9) in, The number of batch samples for the unsupervised branch. For indicator functions, when The value is 1 if the condition is met, and 0 otherwise. The total loss also includes Spe-SSIM loss. L Spe-SSIM The calculation formula is: (10) in, C This represents the total number of feature channels. The first output of the encoder Channel characteristics, For the corresponding reconstruction of the first Channel characteristics, The structural similarity function is represented; the Spe-SSIM loss is used as the total loss. Components of.

8. A semi-supervised image quality assessment system with ranking constraints, used to implement the method as described in any one of claims 1 to 7, characterized in that, include: The network construction module is configured to construct a two-branch teacher-student collaborative learning network based on the contrastive language-image pre-trained model CLIP, which reshapes the continuous scalar image quality regression task into a discrete ordered probability distribution prediction task; the CLIP-based two-branch teacher-student collaborative learning network includes a supervised branch and an unsupervised branch trained simultaneously. The supervised processing module is configured to input an image with a rating label into the student network in the supervised branch, extract the visual features of the image and the text features of a preset quality level, obtain the predicted quality probability distribution of the image, and calculate the supervised loss based on the difference between the true quality distribution and the predicted quality probability distribution. An unsupervised processing module is configured to perform weak enhancement processing on unlabeled images in the unsupervised branch to obtain weakly enhanced images. The weak enhancement processing includes random horizontal flipping and random cropping, without changing the perceptual quality of the image. The weakly enhanced image is input into the teacher network to obtain a tile-level initial prediction quality probability distribution. The tile-level initial prediction quality probability distribution is then subjected to ordered centralized filtering verification, block quality distribution aggregation, and image quality distribution caching processing to obtain a global image-level pseudo-label distribution. The original image blocks of the same unlabeled image are input into the student network, and a global-attribute inverse pairing enhancement strategy is adopted to generate inverse text features with inverted hierarchy in the cross-modal text domain. These features, together with the visual features of the original image blocks, constitute hard samples. The iterative optimization module is configured to calculate the unsupervised loss based on the difference between the global image-level pseudo-label distribution and the predicted quality probability distribution output by the student network on the difficult samples; construct a total loss function based on the supervised loss and the unsupervised loss; and backpropagate to iteratively optimize the network parameters of the student network. The quality evaluation module is configured to input the image to be evaluated into the optimized student network after training convergence, output the corresponding quality probability distribution, and calculate the final perceived quality score of the image based on the quality probability distribution.

9. An electronic device, characterized in that, include: Controller; A memory for storing one or more programs, which, when executed by a controller, cause the controller to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by the computer's processor, cause the computer to perform the method of any one of claims 1 to 7.

Images