Image quality evaluation method and electronic device

Abstract

The present disclosure provides an image quality evaluation method and an electronic device, and relates to the technical field of image processing. The method comprises the following steps: obtaining a pre-trained teacher network and a to-be-trained student network, wherein the teacher network is a visual neural network based on an attention mechanism, and the student network is a pure trunk network based on a convolutional neural network; inputting a training image into the teacher network and the student network respectively, and obtaining output features of N feature processing stages of the teacher network and output features of M feature processing stages of the student network; taking the N output features of the teacher network as a supervision signal, minimizing semantic reconstruction errors between the output features of the teacher network and corresponding stage output features of the student network, and performing multi-stage knowledge distillation training on the student network, so that the student network internalizes a global attention mode of the teacher network. The present disclosure can maintain high-precision image quality evaluation performance while significantly compressing model size and reducing inference overhead.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more particularly to an image quality assessment method and electronic device. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure as set forth in the claims. The description herein is not intended to be a prior art simply because it is included in this section.

[0003] In related technologies, in order to maintain sufficient evaluation accuracy, existing mobile image quality assessment algorithms generally introduce additional structural designs and branch modules, resulting in significant shortcomings in the inference stage: redundant model parameters and lengthy structures.

[0004] The aforementioned image quality assessment methods can cause a series of problems in mobile deployment environments, including excessive memory consumption, significantly increased inference latency, and low efficiency of hardware parallel acceleration. Summary of the Invention

[0005] The purpose of this disclosure is to provide an image quality assessment method and electronic device that can maintain high-precision image quality assessment performance close to that of a teacher network while significantly compressing model size and reducing inference overhead, thus achieving an effective balance between lightweight and high precision.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] This disclosure provides an image quality assessment method, comprising: acquiring a pre-trained teacher network and a student network to be trained, wherein the teacher network is a visual neural network based on an attention mechanism, used to provide image quality assessment supervision signals, and the student network is a pure backbone network based on a convolutional neural network; the student network does not contain an attention mechanism module and / or does not contain an additional feature extraction branch network during the inference phase; inputting training images into the teacher network and the student network respectively, acquiring the output features of N feature processing stages of the teacher network and the output features of M feature processing stages of the student network, wherein N and M are both integers greater than or equal to 2; using the N output features of the teacher network as supervision signals, by establishing a semantic mapping relationship between the output features of the teacher network and the corresponding stage output features of the student network, transferring the global attention pattern of the teacher network to the convolutional kernel parameters of the student network, thereby realizing multi-stage knowledge distillation training of the student network; wherein the student network trained by distillation is used to perform no-reference image quality assessment.

[0008] In some embodiments, using N output features of the teacher network as supervision signals, a semantic mapping relationship is established between the output features of the teacher network and the corresponding stage output features of the student network. This transfers the global attention pattern of the teacher network to the convolutional kernel parameters of the student network, thereby achieving multi-stage knowledge distillation training of the student network. This includes: determining the stage correspondence between the N feature processing stages of the teacher network and the M feature processing stages of the student network; and, based on the stage correspondence, using the output features of each corresponding stage of the teacher network as supervision signals, performing multi-stage semantic reconstruction distillation training on the student network. The distillation training is achieved by minimizing the reconstruction error between the student network output features and the corresponding teacher network features.

[0009] In some embodiments, based on the stage correspondence, the output features of the teacher network at each corresponding stage are used as supervision signals to perform multi-stage semantic reconstruction distillation training on the student network, including: determining the semantic consistency loss between the output features of the teacher network and the student network at each corresponding stage; and performing implicit attention distillation training for multi-stage semantic reconstruction on the student network based on the semantic consistency loss of each stage correspondence.

[0010] In some embodiments, the N feature processing stages of the teacher network include a first processing stage, and the M feature processing stages of the student network include a second processing stage, with the first processing stage corresponding to the second processing stage; wherein, determining the semantic consistency loss between the output features of the teacher network and the output features of the student network corresponding to each stage includes: obtaining a positive preset hyperparameter; comparing each element value in the output features of the first processing stage with the preset hyperparameter; generating an adjusted teacher feature based on the larger value among the comparison results; determining the consistency loss between the output features of the second processing stage and the adjusted teacher feature as a candidate loss; and determining the semantic consistency loss between the output features of the first processing stage and the second processing stage based on the candidate loss.

[0011] In some embodiments, determining the semantic consistency loss between the output features of the first processing stage and the second processing stage based on the candidate loss includes: generating a binary activation mask based on the comparison relationship between the output features of the second processing stage and the adjusted teacher features; wherein, when the element value of the output feature of the second processing stage is greater than the element value at the corresponding position in the adjusted teacher features, the activation mask takes a first value at the corresponding position; otherwise, it takes a second value; and applying the activation mask to the candidate loss to determine the semantic consistency loss between the output features of the first processing stage and the second processing stage.

[0012] In some embodiments, by establishing a semantic mapping relationship between the output features of the teacher network and the corresponding stage output features of the student network, the global attention pattern of the teacher network is transferred to the convolutional kernel parameters of the student network. This includes: aligning the output features in the semantic mapping relationship corresponding to each stage through a linear projection adapter, so as to use the output features of the corresponding feature extraction stage of the teacher network as a supervision signal to perform implicit attention distillation training for multi-stage semantic reconstruction of the student network.

[0013] In some embodiments, the student network further includes a regression head for outputting a first image quality description text characterizing image quality; the method further includes: obtaining label information corresponding to the training image, the label information including a second image quality description text; performing loss calculation based on the second image quality description text in the label information and the first image quality description text output by the regression head to obtain a text loss; and training the student network based on the text loss.

[0014] In some embodiments, the last feature processing stage of both the teacher network and the student network is used to output image quality assessment values; wherein, using N output features of the teacher network as supervision signals, the student network is trained through multi-stage knowledge distillation by minimizing the semantic reconstruction error between the output features of the teacher network and the corresponding stage output features of the student network, including: obtaining a first quality assessment value output by the last feature processing stage of the teacher network and a second quality assessment value output by the last feature processing stage of the student network; determining a quality assessment loss based on the first quality assessment value and the second quality assessment value; and training the student network using the quality assessment loss.

[0015] In some embodiments, the training images are obtained by the following method: obtaining image quality evaluation datasets from multiple data sources, wherein each dataset contains images and their corresponding subjective quality scores; mapping the subjective quality scores in each image quality evaluation dataset to a unified preset score range using a linear normalization method; mixing the normalized images from the multiple datasets and arranging them randomly to construct a training set; and selecting images from the training set as the training images.

[0016] This disclosure provides an image quality assessment device, including: a network acquisition module, an image input module, and a distillation training module.

[0017] The network acquisition module is used to acquire a pre-trained teacher network and a student network to be trained. The teacher network is a visual neural network based on an attention mechanism, used to provide image quality assessment supervision signals. The student network is a pure backbone network based on a convolutional neural network. The image input module can be used to input training images into the teacher network and the student network respectively, and acquire the output features of N feature processing stages of the teacher network and the output features of M feature processing stages of the student network, where N and M are both integers greater than or equal to 2. The distillation training module can be used to use the N output features of the teacher network as supervision signals, and perform multi-stage knowledge distillation training on the student network by minimizing the semantic reconstruction error between the output features of the teacher network and the corresponding stage output features of the student network, so that the student network internalizes the global attention pattern of the teacher network. The student network trained by distillation is used to perform no-reference image quality assessment.

[0018] This disclosure provides an electronic device comprising: a memory and a processor; the memory for storing computer program instructions; and the processor for calling the computer program instructions stored in the memory to implement the image quality assessment method described above.

[0019] This disclosure provides a computer-readable storage medium storing computer program instructions to implement the image quality assessment method as described in any of the preceding embodiments.

[0020] This disclosure provides a computer program product or computer program that includes computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and the processor executes the computer program instructions to implement the aforementioned image quality assessment method.

[0021] The image quality assessment method, apparatus, electronic device, computer-readable storage medium, and computer program product provided in this disclosure, through heterogeneous knowledge distillation with multi-stage feature alignment, enable a pure convolutional student network to internalize the global perception capability of the teacher network without the need for complex auxiliary structures. This significantly compresses the model size and reduces inference overhead while maintaining high-precision image quality assessment performance close to that of the teacher network, achieving an effective balance between lightweight and high precision.

[0022] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 A schematic diagram of a scenario that can be applied to the image quality assessment method or image quality assessment apparatus of the present disclosure is shown.

[0025] Figure 2 This is a flowchart illustrating an image quality assessment method according to an exemplary embodiment.

[0026] Figure 3 This is a flowchart illustrating a multi-stage knowledge distillation training method according to an exemplary embodiment.

[0027] Figure 4 This is a schematic diagram illustrating a multi-stage semantic reconstruction distillation training method for a student network according to an exemplary embodiment.

[0028] Figure 5 This is a flowchart illustrating a method for determining semantic consistency loss according to an exemplary embodiment.

[0029] Figure 6 This is a flowchart illustrating a method for determining semantic consistency loss according to an exemplary embodiment.

[0030] Figure 7 This is a flowchart illustrating a multi-stage semantic reconstruction distillation training method for a student network according to an exemplary embodiment.

[0031] Figure 8 This is a flowchart illustrating a student network training method according to an exemplary embodiment.

[0032] Figure 9 This is a flowchart illustrating a method for multi-stage knowledge distillation training of a student network according to an exemplary embodiment.

[0033] Figure 10 This is a flowchart illustrating a training image acquisition method according to an exemplary embodiment.

[0034] Figure 11 This is a flowchart illustrating an image quality assessment method according to an exemplary embodiment.

[0035] Figure 12 This is a schematic diagram of the structure of a linear adapter according to an exemplary embodiment.

[0036] Figure 13 This is a block diagram illustrating an image quality assessment apparatus according to an exemplary embodiment.

[0037] Figure 14 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0038] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0039] Those skilled in the art will recognize that embodiments of this disclosure can be a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0040] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0041] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0042] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0043] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0044] In the description of this disclosure, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences; the terms "contains," "includes," and "has" are used to indicate an open-ended meaning of inclusion and refer to the existence of additional elements / components / etc. besides those listed.

[0045] This disclosure embodiment can be implemented by a terminal and / or a server. The terminal can obtain data from a computer device and display that data. The computer device can interact with the terminal, and can be a server hosting the application, or it can belong to the terminal (i.e., the terminal's backend), etc., without limitation.

[0046] The terminal can be a mobile phone, a laptop computer, or a playback device in a vehicle, etc., without limitation. The terminal can be considered a playback device in a vehicle, and it can display the target application. The terminal is only one example of the devices listed; the terminal in this disclosure is not limited to the listed devices. The target application in this disclosure can be any application capable of displaying multimedia information.

[0047] It is understood that the terminal mentioned in the embodiments of this disclosure can be a computer device, including but not limited to a terminal or a server. In other words, the computer device can be a server or a terminal, or a system composed of a server and a terminal. The terminal mentioned above can be an electronic device, including but not limited to mobile phones, tablets, desktop computers, laptops, handheld computers, in-vehicle devices, augmented reality / virtual reality (AR / VR) devices, head-mounted displays, smart TVs, wearable devices, smart speakers, digital cameras, webcams, and other mobile internet devices (MIDs) with network access capabilities, or terminals in scenarios such as trains, ships, and flights.

[0048] The servers mentioned above can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, vehicle-road cooperation, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0049] Optionally, the data involved in the embodiments of this disclosure may be stored in a computer device or may be stored based on cloud storage technology, without limitation.

[0050] To better understand the above-mentioned objectives, features and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0051] In related technologies, existing mobile IQA (Image Quality Assessment) algorithms (such as MobileIQA) rely on additional feature enhancement modules (such as LDA (Local Distortion Awareness) and MAL (Multi-level Attention Learning)) during the inference stage, which leads to redundant model parameters and low deployment efficiency.

[0052] The technical solution proposed in this application can use a heterogeneous knowledge distillation method based on multi-stage (combined with linear projection adaptation of image quality features) (transferring the complex representation capabilities of attention networks to pure convolutional neural networks) and / or a multi-source label normalization strategy to achieve the effect of maintaining high-precision image quality evaluation while completely removing the auxiliary modules in the inference stage to realize ultra-fast inference of the pure backbone network.

[0053] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0054] Figure 1 A schematic diagram of a scenario that can be applied to the image quality assessment method or image quality assessment apparatus of the present disclosure is shown.

[0055] Please refer to Figure 1 The diagram illustrates an implementation environment provided by an exemplary embodiment of this disclosure.

[0056] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0057] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, desktop computers, wearable devices, virtual reality devices, smart home devices, etc.

[0058] Server 105 can be a server that provides various services, such as a backend management server that supports the devices operated by users using terminal devices 101, 102, and 103. The backend management server can analyze and process received requests and other data, and feed the processing results back to the terminal devices.

[0059] A server can be a standalone physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This disclosure does not impose any restrictions on this.

[0060] Server 105 may, for example, acquire a pre-trained teacher network and a student network to be trained, wherein the teacher network is a visual neural network based on an attention mechanism, used to provide image quality assessment supervision signals, and the student network is a pure backbone network based on a convolutional neural network; Server 105 may, for example, input training images into the teacher network and the student network respectively, and acquire the output features of N feature processing stages of the teacher network and the output features of M feature processing stages of the student network, wherein N and M are both integers greater than or equal to 2; Server 105 may, for example, use the N output features of the teacher network as supervision signals, and perform multi-stage knowledge distillation training on the student network by minimizing the semantic reconstruction error between the output features of the teacher network and the corresponding stage output features of the student network, so that the student network internalizes the global attention pattern of the teacher network; wherein the student network trained by distillation is used to perform no-reference image quality assessment.

[0061] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Server 105 can be a single physical server or a combination of multiple servers. Depending on actual needs, it can have any number of terminal devices, networks, and servers.

[0062] Under the above system architecture, this disclosure provides an image quality assessment method that can be executed by any electronic device with computing capabilities.

[0063] Figure 2 This is a flowchart illustrating an image quality assessment method according to an exemplary embodiment. The method provided in this disclosure can be executed by any electronic device with computing power, for example, the method can be implemented by the above-described... Figure 1 The execution can be performed by a server or terminal device in the embodiments, or it can be performed by both a server and a terminal device. In the following embodiments, the server is used as the execution subject for illustration, but this disclosure is not limited to this.

[0064] Reference Figure 2 The image quality assessment method provided in this disclosure may include the following steps.

[0065] Step S202: Obtain the pre-trained teacher network and the student network to be trained. The teacher network is a visual neural network based on the attention mechanism, used to provide image quality assessment supervision signals. The student network is a pure backbone network based on the convolutional neural network. The student network does not contain an attention mechanism module and / or does not contain an additional feature extraction branch network during the inference stage.

[0066] A pre-trained teacher network refers to a visual neural network model based on an attention mechanism that has been trained on image quality assessment tasks and possesses high accuracy. The network's structure is typically a visual Transformer or a variant thereof, and its core is to globally model the input image through a self-attention mechanism, thereby learning deep semantic features that strongly discriminate against image distortion.

[0067] In the distillation framework proposed in this application, the teacher network is no longer used for direct reasoning, but serves as a fixed, high-quality "knowledge source". By extracting the feature maps output at different processing stages, it provides multi-level supervision signals containing global perception patterns to the lightweight student network.

[0068] The student network to be trained refers to a lightweight target model that needs to be trained and optimized. This student network can be a structurally pure convolutional neural network backbone designed specifically for mobile deployment, characterized by the absence of any additional auxiliary branches or modules (such as LDA, MAL, etc.) used to enhance performance. The initial weights of the student network are randomized or pre-initialized for a basic task, resulting in significantly lower network capacity and computational complexity compared to the teacher network. During distillation, the student network learns and internalizes the global attention mechanism and high-quality feature representations inherent in the teacher network through multi-stage feature alignment and semantic reconstruction. After training, the student network can serve as an independent, efficient, and high-precision model, directly applicable to no-reference image quality assessment inference tasks on mobile devices.

[0069] In some embodiments, the student network may be composed of concatenated convolutional neural networks, without additional branching structures or attention modules.

[0070] In some embodiments, the student network can be constructed as a pure backbone network structure, which may not include additional auxiliary branches or modules for improving image quality evaluation performance, but consists only of basic components of a convolutional neural network. This design aims to maximize computational efficiency and deployment friendliness during the inference phase.

[0071] In some embodiments, the student network may further include a regression head, which may be connected to the end of the pure backbone network for performing regression calculations on the features extracted by the backbone network and outputting subjective evaluation text characterizing image quality.

[0072] In some embodiments, a heterogeneous teacher-student architecture with complementary global and local features can be constructed. The teacher network establishes global image dependencies based on an attention mechanism, enabling modeling of overall semantic and structural information; while the student network, a convolutional neural network, naturally focuses on the extraction and aggregation of local features. Through knowledge distillation, the two complement each other at the feature level, allowing the student network to retain efficient local perception capabilities while internalizing the global perception pattern of the teacher network.

[0073] The above technology breaks through the limitations of existing technologies (such as MobileIQA), which must rely on stacked Local Distortion Aware (LDA) and Multi-level Feature Perception (MAL) modules to reduce the bias of image quality evaluation results of lightweight networks.

[0074] This embodiment creatively proposes a minimalist student network structure of "pure backbone + regression head". It solves the problems of high inference latency, large memory usage, and severe operator fragmentation caused by additional functional modules in existing technologies deployed on mobile devices. It achieves zero additional parameter overhead in the inference phase, maximizing the utilization of hardware acceleration.

[0075] Step S204: Input the training images into the teacher network and the student network respectively, and obtain the output features of the teacher network at N feature processing stages and the output features of the student network at M feature processing stages, where N and M are both integers greater than or equal to 2.

[0076] In some embodiments, the feature processing stage can refer to the hierarchical modules in a deep neural network that are divided according to function and structure. Each stage consists of a series of isomorphic layers, which can transform input features from low-level visual information into high-level semantic representations by gradually reducing spatial resolution and increasing channel dimensions.

[0077] In some embodiments, both the teacher and student networks can be divided into multiple feature processing stages. By aligning their feature outputs stage by stage, hierarchical and fine-grained knowledge transfer can be achieved, enabling the student network to fully internalize the teacher network's multi-scale representation capabilities from local to global.

[0078] In some embodiments, the teacher network, as a complete deep model, can be composed of multiple sequentially connected feature processing stages. Each stage is a functional module that performs feature transformation at a specific level of abstraction, responsible for progressively refining the input from low-level visual information into high-level semantic representation. In knowledge distillation, it is by extracting and utilizing the intermediate feature outputs of these different stages of the teacher network as multi-level supervision signals that the systematic transfer of the student network's ability from local perception to global modeling can be achieved.

[0079] In some embodiments, the student network can also be composed of multiple sequentially connected feature processing stages, each stage representing a local feature extraction and fusion unit at a specific depth level of the network. In the distillation framework, each stage of the student network and the corresponding stage of the teacher network establish a mapping relationship through a feature alignment mechanism, so that the student network can learn and internalize the semantic knowledge and attention patterns transmitted by the teacher network from shallow to deep at each level, and finally integrate them into its pure convolutional backbone structure.

[0080] In some embodiments, M may or may not be equal to N. N can be 2, 3, 4, 5, 6…; M can be 2, 3, 4, 5, 6…; this application does not impose any restrictions on this.

[0081] In some embodiments, when the number of stages in the student network and the teacher network is inconsistent, many-to-one or one-to-many mapping can be achieved through feature aggregation or splitting, or key stages can be selected for focused alignment, and a learnable lightweight adapter can be used to automatically adjust the feature dimension and semantic distribution, so that an effective cross-level knowledge transfer path can still be established even in the case of structural asymmetry.

[0082] Step S206: Using the N output features of the teacher network as supervision signals, by establishing a semantic mapping relationship between the output features of the teacher network and the corresponding stage output features of the student network, the global attention pattern of the teacher network is transferred to the convolution kernel parameters of the student network to achieve multi-stage knowledge distillation training of the student network; wherein, the student network trained by distillation is used to perform no-reference image quality assessment.

[0083] In some embodiments, the student network can be trained through multi-stage knowledge distillation by minimizing the semantic reconstruction error between the output features of the teacher network and the corresponding stage output features of the student network, thereby enabling the student network to internalize the global attention pattern of the teacher network; wherein the student network trained by distillation is used to perform no-reference image quality assessment.

[0084] In some embodiments, the feature maps output by each stage of the teacher network can be aligned with the feature maps of the corresponding stages of the student network through feature alignment. Next, a semantic reconstruction loss function with stage-by-stage alignment is constructed to calculate the difference between teacher and student features in the normalized space (e.g., using mean squared error or cosine distance). Simultaneously, this multi-stage feature loss is combined with the final output loss of the image quality regression task to form a multi-objective optimization function. During training, gradient backpropagation is used to update only the parameters of the student network, forcing its convolutional layers to mimic the attention-based feature activation pattern of the teacher network at each feature level. After sufficient training, the student network gradually internalizes the hierarchical representation capabilities of the teacher network, from local texture perception to global structure modeling, thereby enabling it to perform high-precision no-reference image quality assessment inference using only its pure convolutional backbone, even without removing any additional attention or enhancement modules.

[0085] Below, this application will provide a detailed description of implicit attention distillation based on full-stage semantic reconstruction.

[0086] In some embodiments, the convolutional kernel parameters of the student network can be forced to fit the global attention pattern of the teacher network by performing strongly constrained semantic consistency calculations at multiple feature processing stages. It is precisely because of the "strong feature reconstruction" implemented in this step that the "removal of additional modules such as LDA / MAL" becomes possible.

[0087] The above method addresses the inherent lack of global awareness in lightweight CNNs (MobileNet) and the significant drop in accuracy after removing explicit attention modules. It achieves "implicit internalization of attention"—that is, compressing the functionality that originally required external modules to compute into the convolutional weights of the backbone network.

[0088] The technical solution provided in this embodiment constructs a "global-local" complementary heterogeneous distillation architecture and, based on an implicit attention transfer mechanism of multi-stage feature alignment and semantic reconstruction, successfully enables a pure convolutional student network to internalize the global perception capability of the teacher network without relying on any additional auxiliary modules (such as LDA or MAL). This achieves extreme simplification of model parameters and maximization of inference efficiency while maintaining high-precision image quality assessment performance similar to that of complex teacher networks, effectively solving the key problem of balancing lightweight design and high precision in mobile deployment.

[0089] Figure 3 This is a flowchart illustrating a multi-stage knowledge distillation training method according to an exemplary embodiment.

[0090] In some embodiments, training a student network with multi-stage knowledge distillation may include the following steps.

[0091] Step S302: Determine the stage correspondence between the N feature processing stages of the teacher network and the M feature processing stages of the student network.

[0092] In some embodiments, manual or automatic matching can be performed based on the feature map resolution, channel dimension, and semantic depth of each stage of the teacher network and student network to establish a preliminary correspondence between stages with similar resolution and semantic level. If the number of stages is unequal, multiple consecutive stages can be merged or a single stage can be split into multiple sub-stages through cross-stage feature fusion (such as pooling, interpolation, or convolution) to achieve many-to-one or one-to-many mapping. Furthermore, lightweight learnable alignment modules or soft attention weights can be introduced to dynamically optimize the feature similarity between stages during training, thereby adaptively establishing and strengthening the most effective cross-network knowledge transfer path.

[0093] This application does not impose any restrictions on how to determine the stage correspondence between the N feature processing stages of the teacher network and the M feature processing stages of the student network.

[0094] Step S304: Based on the stage correspondence, the output features of each corresponding stage of the teacher network are used as supervision signals to perform multi-stage semantic reconstruction distillation training on the student network. Distillation training is achieved by minimizing the reconstruction error between the output features of the student network and the corresponding teacher network features.

[0095] In some embodiments, the feature maps output by the teacher network at each corresponding stage and the feature maps output by the student network at the corresponding stage can be normalized to make their numerical distributions comparable. Then, a stage-by-stage feature reconstruction loss function is constructed to quantify semantic error by calculating the difference between teacher features and student features in the normalized space (such as mean square error or cosine similarity). Then, the feature reconstruction losses of each stage are weighted and summed, and combined with the main loss of the image quality regression task to form a multi-objective optimization loss. During training, the parameters of the student network are iteratively updated through backpropagation, so that the convolutional kernels of each stage gradually imitate the attention activation patterns of the teacher network at the corresponding stage, and finally, the global perception capability of the teacher network is internalized without any additional auxiliary modules.

[0096] Furthermore, the method provided in this embodiment has good compatibility with teacher network architectures and can adapt to teacher networks based on different attention mechanisms or hybrid architectures for knowledge distillation, thereby supporting the continuous and flexible updating and iteration of the image quality assessment model as the teacher network evolves.

[0097] This embodiment achieves refined knowledge transfer across networks and scales by establishing flexible and optimizable stage correspondences and performing stage-by-stage semantic reconstruction distillation. This method enables student networks to efficiently learn and internalize multi-level perceptual patterns, from local details to global semantics, inherent in various teacher networks (including different attention mechanisms or hybrid architectures) without introducing any additional structural burden. Ultimately, while significantly simplifying the model structure and greatly reducing computational and storage overhead, it ensures that the lightweight student network can continuously follow the evolution and iteration of the teacher network and consistently maintain high-precision image quality evaluation performance comparable to state-of-the-art complex models.

[0098] Figure 4 This is a schematic diagram illustrating a multi-stage semantic reconstruction distillation training method for a student network according to an exemplary embodiment.

[0099] refer to Figure 4 The above-mentioned method for multi-stage semantic reconstruction distillation training of student networks may include the following steps.

[0100] Step S402: Determine the semantic consistency loss between the output features of the teacher network and the student network at each corresponding stage.

[0101] Step S404: Implicit attention distillation training for multi-stage semantic reconstruction of the student network based on the semantic consistency loss of the correspondence at each stage.

[0102] In some embodiments, the semantic consistency loss between the output features of the teacher and student networks at each corresponding stage can be calculated. This loss is quantified by a difference measure after feature normalization (such as normalized mean square error). Subsequently, the semantic consistency losses of each stage are weighted and fused, and jointly constructed with the main loss of image quality regression to form a multi-objective optimization function. The parameters of the student network are iteratively updated through gradient backpropagation, forcing the convolutional kernels of each stage to gradually fit the attention response distribution of the teacher network at the corresponding stage. This enables the global perception capability to be progressively internalized into the pure convolutional backbone network during training, ultimately allowing the lightweight student network to perform high-precision no-reference quality assessment even after removing all explicit enhancement modules.

[0103] The technical solution provided in this embodiment calculates and optimizes the semantic consistency loss between the teacher and student networks in stages, driving the convolutional kernel of the student network to accurately fit the attention response pattern of the teacher network in a multi-level feature space, thereby achieving efficient internalization of global perception capability into a lightweight pure convolutional backbone. This method, by completely removing traditional explicit enhancement modules (such as LDA and MAL), not only significantly reduces model complexity and inference overhead, but also ensures that the lightweight network can still maintain high-precision no-reference image quality evaluation performance comparable to state-of-the-art large models when deployed on mobile devices.

[0104] Figure 5 This is a flowchart illustrating a method for determining semantic consistency loss according to an exemplary embodiment.

[0105] In some embodiments, the N feature processing stages of the teacher network may include a first processing stage, and the M feature processing stages of the student network may include a second processing stage, wherein the first processing stage and the second processing stage are matched accordingly.

[0106] refer to Figure 5 The method for determining the semantic consistency loss of output features between the first processing node and the second output stage may include the following steps.

[0107] Step S502: Obtain a positive preset hyperparameter.

[0108] Preset hyperparameters are a set of fixed values ​​or configuration parameters that are set in advance based on experience, experiments or task requirements before model training begins. They are used to control the training process, model behavior or the shape of the loss function. They usually remain unchanged (or decay according to predetermined rules) during training, rather than model parameters learned from data through backpropagation.

[0109] In some embodiments, the preset hyperparameter m can be a small positive number (e.g., 0.01, 0.02, etc.).

[0110] Step S504: Compare each element value in the output features of the first processing stage with the preset hyperparameters.

[0111] Step S506: Generate the adjusted teacher features based on the larger value among the comparison results.

[0112] Step S508: Determine the consistency loss between the output features of the second processing stage and the adjusted teacher features as a candidate loss.

[0113] Step S510: Determine the semantic consistency loss between the output features of the first processing stage and the second processing stage based on the candidate loss.

[0114] In some embodiments, each element value in the output features of the teacher network in the first processing stage can be compared with a preset hyperparameter, and the larger value can be selected to generate the adjusted teacher features. Subsequently, the consistency loss between the output features of the student network in the second processing stage and the adjusted teacher features can be calculated as a candidate loss. Finally, the semantic consistency loss between the output features of the two stages is determined based on this candidate loss. Thus, in the distillation training, stable alignment of cross-stage features is achieved, and the non-rigid comparison mechanism preserves the learning space for the network to adapt to local features.

[0115] In some embodiments, setting specific hyperparameters can improve the numerical stability of the loss calculation process, effectively avoiding problems such as numerical instability, gradient explosion, or vanishing gradients during normalization when the teacher network feature values ​​are too small or have outliers (such as being close to zero or negative). Furthermore, the introduction of these hyperparameters constructs a non-rigid knowledge transfer mechanism, allowing the student network to retain a certain degree of feature autonomy while learning the global representation of the teacher network, thus balancing the focus on and extraction of local features.

[0116] The above embodiments construct a robust and flexible semantic consistency loss calculation mechanism by introducing preset positive hyperparameters and generating adjusted teacher features based on element-level comparisons. This design not only effectively avoids numerical instability and gradient problems caused by abnormal feature values, ensuring the stable progress of distillation training, but also preserves a certain degree of autonomy for the student network when learning the global representation of the teacher, enabling it to capture and maintain local detailed features. This achieves efficient knowledge transfer while enhancing the feature representation richness and generalization ability of the lightweight model.

[0117] Figure 6 This is a flowchart illustrating a method for determining semantic consistency loss according to an exemplary embodiment.

[0118] refer to Figure 6 The above-mentioned method for determining semantic consistency loss may include the following steps.

[0119] Step S602: Based on the comparison relationship between the output features of the second processing stage and the adjusted teacher features, a binary activation mask is generated; wherein, when the element value of the output feature of the second processing stage is greater than the element value of the corresponding position in the adjusted teacher features, the activation mask takes the first value at the corresponding position; otherwise, the second value is taken.

[0120] Step S604: Apply the activation mask to the candidate loss to determine the semantic consistency loss between the output features of the first processing stage and the second processing stage.

[0121] In some embodiments, a binary activation mask can be generated by comparing the output features of the student network in the second processing stage with the adjusted teacher features. The mask takes a first value (e.g., 1) when the student feature is greater than the corresponding teacher feature, and a second value (e.g., 0) otherwise. Subsequently, the activation mask is applied to the candidate loss (e.g., element-wise loss). The mask is used to filter out the positions where the student features are dominant and the loss is calculated or weighted only in these regions. This allows the student network to selectively strengthen its local feature representation that is already superior to the teacher network during training, while weakening the negative impact of misaligned regions, thus achieving more flexible and autonomous knowledge distillation.

[0122] In some embodiments, the semantic consistency loss can be calculated using the following formula (1).

[0123] (1)

[0124] in, This represents the activation function; when the value inside the parentheses is greater than 0... The corresponding value is 0, when the value inside the parentheses is less than or equal to 0. The corresponding value is 1; This represents the characteristics of the student network in the i-th stage. Let m represent the characteristics of the teacher network in the i-th stage, where m is a hyperparameter and i is an integer greater than 0.

[0125] Figure 7 This is a flowchart illustrating a multi-stage semantic reconstruction distillation training method for a student network according to an exemplary embodiment.

[0126] refer to Figure 7 The multi-stage semantic reconstruction distillation training method for student networks may include the following steps.

[0127] Step S702: Align the output features in the correspondence of each stage using a linear projection adapter, so as to use the output features of the corresponding feature extraction stage of the teacher network as a supervision signal to perform implicit attention distillation training for multi-stage semantic reconstruction of the student network.

[0128] A linear projection adapter is a lightweight, learnable module for aligning the feature spaces of heterogeneous networks. It can consist of a linear transformation layer and its function is to map the high-dimensional, global features output by the teacher network (such as ViT) to a dimension and distribution space that matches the corresponding stage features of the student network (such as CNN), thereby solving the problem of feature semantics and scale mismatch caused by different network architectures. This module is inserted and used only during the training phase and completely removed after distillation, ensuring that the student network maintains the lightweight and efficient nature of the pure backbone structure during inference.

[0129] In some embodiments, a linear projection adapter can be designed to address the difference in image quality feature distribution between the teacher network (such as a ViT structure or a ViT+CNN hybrid network structure) and the student network (CNN structure). This adapter can be specifically designed to establish a lossless mapping channel from the "attention feature space" to the "pure convolutional feature space," and this module can exist only during the training phase. This method solves the problem that the significant differences in feature dimensions and semantic distribution between heterogeneous network architectures prevent effective knowledge distillation directly.

[0130] In some embodiments, each adapter may have the same structure, but the input and output feature dimensions may be different.

[0131] This embodiment introduces a linear projection adapter, which exists only during the training phase, to construct a universal, learnable heterogeneous feature alignment interface. This interface not only accurately maps the high-dimensional global attention features of the teacher network to the convolutional feature space of the student network, effectively overcoming the knowledge transfer barrier caused by architectural differences, but also, due to its structural independence and lightweight parameters, allows the same student network to seamlessly adapt to teacher networks with different architectures without requiring any structural modifications to either the student model or the teacher network. Ultimately, this method ensures that the student network achieves multi-stage semantic representation capabilities comparable to the teacher network while maintaining the extreme lightweight nature and deployment efficiency of its pure backbone structure during inference, achieving a triple unity of model performance, architectural compatibility, and engineering feasibility.

[0132] Figure 8 This is a flowchart illustrating a student network training method according to an exemplary embodiment.

[0133] In some embodiments, the student network may also include a regression head for outputting a first image quality description text characterizing the image quality.

[0134] refer to Figure 8 The above-mentioned online training method for students may include the following steps.

[0135] Step S802: Obtain the label information corresponding to the training image. The label information includes the second image quality description text.

[0136] Step S804: Based on the second image quality description text in the label information and the first image quality description text output by the regression head, the loss is calculated to obtain the text loss.

[0137] Step S806: Train the student network based on text loss.

[0138] In some embodiments, training image labels with second image quality description text can be obtained, and the predicted comment text output by the student network regression head can be compared with the real labels to calculate text loss (such as cross-entropy loss or embedding space-based similarity loss). Subsequently, the text loss is combined with the multi-stage semantic reconstruction loss and used together as the training objective to optimize the student network parameters. This allows the network to not only learn the structured features of the teacher network during the distillation process, but also align with the semantic description of human subjective evaluation, thereby enhancing the interpretability of the model quality assessment results and consistency with human perception.

[0139] Figure 9This is a flowchart illustrating a method for multi-stage knowledge distillation training of a student network according to an exemplary embodiment.

[0140] In some embodiments, the final feature processing stage of both the teacher network and the student network is used to output image quality assessment values.

[0141] refer to Figure 9 The above-mentioned method for multi-stage knowledge distillation training of student networks may include the following steps.

[0142] Step S902: Obtain the first quality assessment value output by the last feature processing stage of the teacher network and the second quality assessment value output by the last feature processing stage of the student network.

[0143] Step S904: Determine the quality assessment loss based on the first quality assessment value and the second quality assessment value.

[0144] Step S906: Train the student network by incorporating the quality assessment loss.

[0145] In some embodiments, the image quality assessment values ​​(i.e., the first quality assessment value and the second quality assessment value) output by the teacher network and the student network in the final feature processing stage can be extracted, and the difference between the two can be calculated as the quality assessment loss (e.g., using mean squared error or absolute value error). Subsequently, this loss is combined with the aforementioned multi-stage feature reconstruction loss and possible text loss to construct a comprehensive optimization objective, which drives the student network to further align its final regression output while imitating the teacher's feature expression, thereby ensuring that the lightweight network obtained by distillation directly approximates the end-to-end evaluation performance of the teacher network in the image quality scoring task.

[0146] Figure 10 This is a flowchart illustrating a training image acquisition method according to an exemplary embodiment.

[0147] refer to Figure 10 The above-mentioned training image acquisition method may include the following steps.

[0148] Step S1002: Obtain image quality evaluation datasets from multiple data sources, where each dataset contains images and their corresponding subjective quality scores.

[0149] Step S1004: For the subjective quality scores in each image quality evaluation dataset, a linear normalization method is used to map them to a unified preset score range.

[0150] Step S1006: Mix the normalized images from multiple datasets and arrange them randomly to construct a training set.

[0151] Step S1008: Select images from the training set as training images.

[0152] This method effectively eliminates the differences in rating scale, distribution, and subjective preferences among data from different sources by aggregating multi-source heterogeneous image quality assessment datasets and performing unified linear normalization on their subjective ratings. This constructs a larger, more balanced, and numerically consistent hybrid training set, thus providing a data foundation for subsequent knowledge distillation training, enhancing the model's generalization ability and robustness, and avoiding the performance degradation in real-world scenarios caused by overfitting to the rating characteristics of a single dataset.

[0153] Below, this application will explain and illustrate the image quality assessment method proposed in this application in conjunction with specific application scenarios.

[0154] Figure 11 This is a flowchart illustrating an image quality assessment method according to an exemplary embodiment.

[0155] refer to Figure 11 The following is a detailed description of the technical solution based on the above flowchart.

[0156] Step S1: Construct a unified multi-source image quality assessment (IQA) data benchmark.

[0157] In some embodiments, publicly available image quality assessment (IQA) datasets from multiple sources can be obtained. These datasets contain original images, distorted images, and corresponding subjective scores (Mean Opinion Score M, MOS). To address the problem of inconsistent scoring scales across different datasets (e.g., different scales like 0-100 versus 0-9) causing model convergence difficulties, a unified normalization mapping mechanism is established. A linear scaling algorithm is used to map all ground truth labels to the standardized interval [0, 1]. A hybrid training queue is constructed, and the normalized multi-source data is randomly shuffled and input to form a training set with a high variance distribution.

[0158] Step S1 provides numerical stability for the subsequent loss and is a necessary prerequisite for the network to converge.

[0159] The above steps eliminate distribution barriers between data sources, providing a numerically stable gradient environment for subsequent distillation training of image quality features. The high-variance mixed data distribution forces the model to learn the essential quality features of image content, rather than overfitting to the scoring habits of a specific dataset. This is the data foundation for lightweight models to maintain generalization ability even after removing auxiliary modules.

[0160] Step S2: Construct a heterogeneous teacher-student architecture that is complementary in terms of "global-local" relationships.

[0161] The above steps can overcome the limitations of existing technologies (such as MobileIQA) that rely on stacked Local Distortion Aware (LDA) and Multi-level Feature Perception (MAL) modules to reduce the bias in image quality evaluation results of lightweight networks.

[0162] This embodiment creatively proposes a minimalist student network structure of "pure backbone + regression head". It solves the problems of high inference latency, large memory usage, and severe operator fragmentation caused by additional functional modules in existing IQA models deployed on mobile devices. It achieves zero additional parameter overhead in the inference phase, maximizing the utilization of hardware acceleration.

[0163] Step S3: Construct a linear projection adaptation mechanism for heterogeneous image quality feature space.

[0164] The above steps address the issue of differences in image quality feature distribution between the teacher network (ViT structure or ViT+CNN hybrid network structure) and the student network (CNN structure). A linear projection adapter is designed to establish a lossless mapping channel from the "attention feature space" to the "pure convolution feature space", and this module exists only during the training phase.

[0165] The above steps solve the problem that effective knowledge distillation cannot be directly performed between heterogeneous network architectures due to the huge differences in feature dimensions and semantic distribution.

[0166] Each adapter has the same structure, but the input and output feature dimensions differ. The structure of a single adapter can be found by referring to... Figure 12 .

[0167] like Figure 12 As shown, an adapter may include one (or more) volume base layers and one (or more) batch normalization layers. This application does not limit the specific structure of the adapter.

[0168] Step S4: Implicit attention distillation based on full-stage semantic reconstruction.

[0169] The above steps, through strongly constrained MSE semantic consistency calculations in five stages, force the student network's convolutional kernel parameters to fit the teacher network's global attention pattern. It is precisely because of this "strong feature reconstruction" in step S2 that the "removal of LDA / MAL modules" becomes possible.

[0170] The above process can solve the problem that lightweight CNNs (MobileNet) inherently lack global perception capabilities and that accuracy drops significantly after removing explicit attention modules. It achieves "implicit internalization of attention"—that is, compressing the functions that originally required external modules to calculate into the convolutional weights of the backbone network. The formula for calculating MSE semantic consistency can be found in formula (1).

[0171] It should be particularly noted that the steps in the various embodiments of the above image quality assessment method can be overlapped, substituted, added, or deleted from each other. Therefore, these reasonable permutations and combinations of image quality assessment methods should also fall within the protection scope of this disclosure, and the protection scope of this disclosure should not be limited to the embodiments.

[0172] It should be noted that the scope of protection of this application should include, but is not limited to, the specific implementation methods described in the embodiments. Any alternative solution that uses a different name but substantially performs the same function and achieves the same technical effect falls within the scope of protection defined by the claims of this application.

[0173] Based on the same inventive concept, this disclosure also provides an image quality assessment device, as shown in the following embodiments. Since the principle by which this device addresses the problem is similar to that of the method embodiments described above, the implementation of this device embodiment can refer to the implementation of the method embodiments described above, and repeated details will not be elaborated further.

[0174] Figure 13 This is a block diagram illustrating an image quality assessment apparatus according to an exemplary embodiment. (Refer to...) Figure 13 The image quality assessment device 1300 provided in this embodiment may include: a network acquisition module 1301, an image input module 1302, and a distillation training module 1303.

[0175] The network acquisition module 1301 can be used to acquire a pre-trained teacher network and a student network to be trained. The teacher network is a visual neural network based on an attention mechanism, used to provide image quality assessment supervision signals, and the student network is a pure backbone network based on a convolutional neural network. The image input module 1302 can be used to input training images into the teacher network and the student network respectively, and acquire the output features of N feature processing stages of the teacher network and the output features of M feature processing stages of the student network, where N and M are both integers greater than or equal to 2. The distillation training module 1303 can be used to use the N output features of the teacher network as supervision signals, and perform multi-stage knowledge distillation training on the student network by minimizing the semantic reconstruction error between the output features of the teacher network and the corresponding stage output features of the student network, so that the student network internalizes the global attention pattern of the teacher network. The student network trained by distillation is used to perform no-reference image quality assessment.

[0176] It should be noted that the network acquisition module 1301, image input module 1302, and distillation training module 1303 correspond to S202 to S206 in the method embodiments. The examples and application scenarios implemented by these modules and their corresponding steps are the same, but they are not limited to the content disclosed in the above method embodiments. It should also be noted that these modules, as part of the apparatus, can be executed in a computer system, such as a set of computer-executable instructions.

[0177] In some embodiments, N output features of the teacher network are used as supervision signals. By establishing a semantic mapping relationship between the output features of the teacher network and the corresponding stage output features of the student network, the global attention pattern of the teacher network is transferred to the convolution kernel parameters of the student network to achieve multi-stage knowledge distillation training of the student network. This includes: determining the stage correspondence between the N feature processing stages of the teacher network and the M feature processing stages of the student network; and, based on the stage correspondence, using the output features of each corresponding stage of the teacher network as supervision signals, performing multi-stage semantic reconstruction distillation training on the student network. The distillation training is achieved by minimizing the reconstruction error between the output features of the student network and the corresponding teacher network features.

[0178] In some embodiments, based on the stage correspondence, the output features of the teacher network at each corresponding stage are used as supervision signals to perform multi-stage semantic reconstruction distillation training on the student network, including: determining the semantic consistency loss between the output features of the teacher network and the student network at each corresponding stage; and performing implicit attention distillation training on the student network for multi-stage semantic reconstruction based on the semantic consistency loss of each stage correspondence.

[0179] In some embodiments, the N feature processing stages of the teacher network include a first processing stage, and the M feature processing stages of the student network include a second processing stage, with the first processing stage corresponding to the second processing stage. Determining the semantic consistency loss between the output features of the teacher network and the output features of the student network corresponding to each stage includes: obtaining a positive preset hyperparameter; comparing each element value in the output features of the first processing stage with the preset hyperparameter; generating an adjusted teacher feature based on the larger value among the comparison results; determining the consistency loss between the output features of the second processing stage and the adjusted teacher feature as a candidate loss; and determining the semantic consistency loss between the output features of the first and second processing stages based on the candidate loss.

[0180] In some embodiments, determining the semantic consistency loss between the output features of the first processing stage and the second processing stage based on the candidate loss includes: generating a binary activation mask based on the comparison relationship between the output features of the second processing stage and the adjusted teacher features; wherein, when the element value of the output feature of the second processing stage is greater than the element value of the corresponding position in the adjusted teacher features, the activation mask takes a first value at the corresponding position; otherwise, it takes a second value; and applying the activation mask to the candidate loss to determine the semantic consistency loss between the output features of the first processing stage and the second processing stage.

[0181] In some embodiments, by establishing a semantic mapping relationship between the output features of the teacher network and the corresponding stage output features of the student network, the global attention pattern of the teacher network is transferred to the convolutional kernel parameters of the student network. This includes: aligning the output features in the semantic mapping relationship corresponding to each stage through a linear projection adapter, so as to use the output features of the corresponding feature extraction stage of the teacher network as a supervision signal to perform implicit attention distillation training for multi-stage semantic reconstruction of the student network.

[0182] In some embodiments, the student network further includes a regression head for outputting a first image quality description text representing image quality; the method further includes: obtaining label information corresponding to the training image, the label information containing a second image quality description text; calculating a loss based on the second image quality description text in the label information and the first image quality description text output by the regression head to obtain a text loss; and training the student network based on the text loss.

[0183] In some embodiments, the last feature processing stage of both the teacher network and the student network is used to output image quality assessment values. Specifically, using N output features of the teacher network as supervision signals, the student network is trained through multi-stage knowledge distillation by minimizing the semantic reconstruction error between the teacher network output features and the corresponding stage output features of the student network. This includes: obtaining a first quality assessment value output by the last feature processing stage of the teacher network and a second quality assessment value output by the last feature processing stage of the student network; determining a quality assessment loss based on the first and second quality assessment values; and training the student network using the quality assessment loss.

[0184] In some embodiments, training images are obtained by: acquiring image quality assessment datasets from multiple data sources, wherein each dataset contains images and their corresponding subjective quality scores; mapping the subjective quality scores in each image quality assessment dataset to a unified preset score range using a linear normalization method; mixing the normalized images from multiple datasets and arranging them randomly to construct a training set; and selecting images from the training set as training images.

[0185] Since the functions of the device 1300 have been described in detail in their respective method embodiments, they will not be repeated here.

[0186] The modules and / or sub-modules and / or units described in the embodiments of this disclosure can be implemented in software or hardware. The described modules and / or sub-modules and / or units can also be located in a processor. The names of these modules and / or sub-modules and / or units do not, in some cases, constitute a limitation on the module and / or sub-module and / or unit itself.

[0187] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a portion of a module or program segment containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer program instructions.

[0188] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0189] Figure 14 A schematic diagram of an electronic device suitable for implementing embodiments of the present disclosure is shown. It should be noted that... Figure 14 The illustrated electronic device 1400 is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0190] like Figure 14As shown, the electronic device 1400 includes a central processing unit (CPU) 1401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1402 or a program loaded from a storage section 1408 into a random access memory (RAM) 1403. The RAM 1403 also stores various programs and data required for the operation of the electronic device 1400. The CPU 1401, ROM 1402, and RAM 1403 are interconnected via a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.

[0191] The following components are connected to I / O interface 1405: an input section 1406 including a keyboard, mouse, etc.; an output section 14013 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1408 including a hard disk, etc.; and a communication section 1409 including a network interface card such as a LAN card, modem, etc. The communication section 1409 performs communication processing via a network such as the Internet. Drive 1410 is also connected to I / O interface 1405 as needed. Removable media 1411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1410 as needed so that computer programs read from it can be installed into storage section 1408 as needed.

[0192] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing computer program instructions for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1409, and / or installed from removable medium 1411. When the computer program is executed by central processing unit (CPU) 1401, it performs the functions defined above in the system of this disclosure.

[0193] It should be noted that the computer-readable storage medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable computer program instructions. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Computer program instructions contained on a computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0194] In another aspect, this disclosure also provides a computer-readable storage medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable storage medium carries one or more programs that, when executed by the device, enable the device to perform the following functions: acquiring a pre-trained teacher network and a student network to be trained, wherein the teacher network is a visual neural network based on an attention mechanism, used to provide image quality assessment supervision signals, and the student network is a pure backbone network based on a convolutional neural network; inputting training images into the teacher network and student network respectively, acquiring the output features of N feature processing stages of the teacher network and the output features of M feature processing stages of the student network, wherein N and M are both integers greater than or equal to 2; using the N output features of the teacher network as supervision signals, performing multi-stage knowledge distillation training on the student network by minimizing the semantic reconstruction error between the output features of the teacher network and the corresponding stage output features of the student network, so that the student network internalizes the global attention pattern of the teacher network; wherein the student network trained by distillation is used to perform no-reference image quality assessment.

[0195] According to one aspect of this disclosure, a computer program product or computer program is provided, comprising computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the methods provided in various optional implementations of the above embodiments.

[0196] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions of the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive) and includes several computer program instructions to cause an electronic device (such as a server or terminal device) to execute the method according to the embodiments of this disclosure.

[0197] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0198] It should be understood that this disclosure is not limited to the detailed structures, drawing arrangements or implementations shown herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. An image quality assessment method, characterized in that, include: Obtain a pre-trained teacher network and a student network to be trained, wherein the teacher network is a visual neural network based on an attention mechanism, used to provide image quality assessment supervision signals, and the student network is a pure backbone network based on a convolutional neural network; the student network does not contain an attention mechanism module and / or does not contain an additional feature extraction branch network during the inference phase. Training images are input into the teacher network and the student network respectively. Output features of N feature processing stages of the teacher network and M feature processing stages of the student network are obtained, where N and M are integers greater than or equal to 2. Using the N output features of the teacher network as supervision signals, a semantic mapping relationship is established between the output features of the teacher network and the corresponding stage output features of the student network. The global attention pattern of the teacher network is transferred to the convolution kernel parameters of the student network to achieve multi-stage knowledge distillation training of the student network. The student network trained by distillation is used to perform no-reference image quality assessment.

2. The method according to claim 1, characterized in that, Using N output features of the teacher network as supervision signals, a semantic mapping relationship is established between the output features of the teacher network and the corresponding stage output features of the student network. This transfers the global attention pattern of the teacher network to the convolutional kernel parameters of the student network, thereby achieving multi-stage knowledge distillation training of the student network. This includes: Determine the stage correspondence between the N feature processing stages of the teacher network and the M feature processing stages of the student network; Based on the stage correspondence, the output features of each corresponding stage of the teacher network are used as supervision signals to perform multi-stage semantic reconstruction distillation training on the student network. The distillation training is achieved by minimizing the reconstruction error between the output features of the student network and the corresponding features of the teacher network.

3. The method according to claim 2, characterized in that, Based on the stage correspondence, the output features of each corresponding stage of the teacher network are used as supervision signals to perform multi-stage semantic reconstruction distillation training on the student network, including: Determine the semantic consistency loss between the output features of the teacher network and the student network at each corresponding stage; Implicit attention distillation training is performed on the student network for multi-stage semantic reconstruction based on the semantic consistency loss of the correspondence at each stage.

4. The method according to claim 3, characterized in that, The teacher network has N feature processing stages, including a first processing stage, and the student network has M feature processing stages, including a second processing stage. The first processing stage corresponds to the second processing stage. Determining the semantic consistency loss between the teacher network output features and the student network output features corresponding to each stage includes: Get a positive preset hyperparameter; Each element value in the output feature of the first processing stage is compared with the preset hyperparameter; The adjusted teacher characteristics are generated based on the larger value among the comparison results; The consistency loss between the output features of the second processing stage and the adjusted teacher features is determined as a candidate loss. The semantic consistency loss between the output features of the first processing stage and the second processing stage is determined based on the candidate loss.

5. The method according to claim 4, characterized in that, Determining the semantic consistency loss between the output features of the first processing stage and the second processing stage based on the candidate loss includes: Based on the comparison between the output features of the second processing stage and the adjusted teacher features, a binary activation mask is generated; wherein, when the element value of the output feature of the second processing stage is greater than the element value of the corresponding position in the adjusted teacher features, the activation mask takes a first value at the corresponding position; otherwise, it takes a second value. The activation mask is applied to the candidate loss to determine the semantic consistency loss between the output features of the first processing stage and the second processing stage.

6. The method according to claim 1, characterized in that, By establishing a semantic mapping relationship between the output features of the teacher network and the corresponding stage output features of the student network, the global attention pattern of the teacher network is transferred to the convolutional kernel parameters of the student network, including: By aligning the output features in the semantic mapping relationship corresponding to each stage through a linear projection adapter, the output features of the corresponding feature extraction stage of the teacher network are used as supervision signals to perform implicit attention distillation training for multi-stage semantic reconstruction of the student network.

7. The method according to claim 1, characterized in that, The student network further includes a regression head for outputting a first image quality description text characterizing image quality; the method further includes: Obtain the label information corresponding to the training image, wherein the label information includes a second image quality description text; The text loss is obtained by calculating the loss based on the second image quality description text in the label information and the first image quality description text output by the regression head. The student network is trained based on the text loss.

8. The method according to claim 1, characterized in that, The final feature processing stage of both the teacher network and the student network is used to output image quality assessment values; wherein, using N output features of the teacher network as supervision signals, the student network is trained through multi-stage knowledge distillation by minimizing the semantic reconstruction error between the output features of the teacher network and the corresponding stage output features of the student network, including: Obtain the first quality assessment value output by the last feature processing stage of the teacher network, and the second quality assessment value output by the last feature processing stage of the student network; Based on the first quality assessment value and the second quality assessment value, determine the quality assessment loss; The student network is trained using the aforementioned quality assessment loss.

9. The method according to claim 1, characterized in that, The training images were obtained using the following method: Obtain image quality assessment datasets from multiple data sources, where each dataset contains images and their corresponding subjective quality scores; For the subjective quality scores in each image quality assessment dataset, a linear normalization method is used to map them to a unified preset score range; The normalized images from the multiple datasets are mixed and randomly arranged to construct a training set; Images are selected from the training set as the training images.

10. An electronic device, characterized in that, include: Memory and processor; The memory is used to store computer program instructions; the processor calls the computer program instructions stored in the memory to implement the image quality assessment method as described in any one of claims 1-9.

Images