An image semantic reconstruction quality evaluation method and device based on attention weighting

Abstract

The application provides an image semantic reconstruction quality evaluation method and device based on attention weighting, wherein the method comprises the following steps: acquiring a video frame comprising a plurality of continuous original images; preprocessing each original image to obtain a preprocessed original image; inputting the preprocessed original image into a pre-trained TimesFormer model to obtain an attention map of the representation category of each original image output by the second-to-last layer of the TimesFormer model; generating a semantic weight map of each original image based on the attention map of the representation category of each original image; determining a quality evaluation value of a reconstructed image of each original image by using the semantic weight map of each original image; and determining a global quality evaluation value of the video frame based on the quality evaluation value of the reconstructed image of each original image. The application improves the accuracy of the quality evaluation value of the reconstructed image.

Description

Technical Field

[0001] This application relates to the field of image quality assessment technology, and in particular to an image semantic reconstruction quality assessment method and apparatus based on attention weighting. Background Technology

[0002] Existing image quality assessment methods include pixel-based difference methods and perceptual similarity methods. Pixel-based difference methods commonly use metrics such as Peak Signal-to-Noise Ratio (PSNR) and Mean Squared Error (MSE), primarily measuring the difference in pixel values ​​between the reference and reconstructed images. This method is simple and intuitive, but insensitive to the loss of semantic information. For example, in visually inconspicuous background areas, even large pixel differences may have little impact on overall perception. Perceptual similarity methods, on the other hand, use the Structural Similarity Index (SSIM) as a metric for perceptual quality. By comparing the brightness, contrast, and structure of the image, they partially address the shortcomings of relying solely on pixel differences. However, SSIM's perceptual capabilities remain limited, struggling to capture differences in semantically significant regions of the image. For instance, in facial images, the loss of key regions such as the eyes and mouth is crucial for quality assessment, but SSIM struggles to quantify this. Summary of the Invention

[0003] In view of this, this application provides an image semantic reconstruction quality assessment method and apparatus based on attention weighting to solve the above-mentioned technical problems.

[0004] Firstly, this application provides an image semantic reconstruction quality assessment method based on attention weighting, including:

[0005] Acquire video frames that include multiple consecutive frames of raw images;

[0006] Each frame of the original image is preprocessed to obtain the preprocessed original image of each frame.

[0007] Each preprocessed original image frame is input into the pre-trained TimesFormer model to obtain the attention map representing the category of each original image frame output by the penultimate layer of the TimesFormer model.

[0008] Based on the attention map of the representation categories of each original image frame, a semantic weight map of each original image frame is generated.

[0009] The quality assessment value of the reconstructed image of each original image is determined by using the semantic weight map of each original image.

[0010] The global quality assessment value of a video frame is determined based on the quality assessment value of the reconstructed image from the original image for each frame.

[0011] In one possible implementation, each frame of the original image is preprocessed to obtain a preprocessed frame of the original image, including:

[0012] The video frame V can be represented as: V = {I1, I2, ..., I...} T}, where I t This represents the original image of frame t, where 1 ≤ t ≤ T; T is the frame number.

[0013] Each frame of the original image is resized to the fixed size required for the TimesFormer model input: 224×224.

[0014] In one possible implementation, a semantic weight map for each frame of the original image is generated based on the attention map representing the categories of the original image, including:

[0015] Obtain the attention map of the original image in frame t. Attention Map It includes N image patches, N = 14 × 14, and each image patch is 16 × 16 in size; attention map Each image patch includes M attention heads representing classification identifiers;

[0016] Calculate the region weight w of the nth image patch t,n :

[0017]

[0018] in, For attention graph The attention of the class identifier in the m-th attention head to the n-th image patch, where the index number of the class identifier is 0; 1≤n≤N;

[0019] Based on the region weight w of the nth image patch t,n Determine the semantic weight value W of its internal pixel (x,y). t (x,y):

[0020] W t (x,y)=w t,n

[0021] The semantic weight values ​​are normalized to obtain normalized semantic weight values.

[0022]

[0023] In one possible implementation, the quality assessment value of the reconstructed image for each original image is determined using the semantic weight map of each original image frame, including:

[0024] Based on the original image I of frame t tand its reconstructed image Determine the reconstructed image First quality assessment value for each pixel Among them, I t (x, y) represents the original image I of frame t. t The pixel value at pixel (x,y); To reconstruct the image The pixel value at pixel (x,y);

[0025] Based on the semantic weight map of the original image in frame t, the reconstructed image is determined. Second Quality Assessment Value (SWQ) t :

[0026]

[0027] Where H is the vertical length of the original image in frame t, and W is the horizontal length of the original image in frame t.

[0028] In one possible implementation, a global quality assessment value for a video frame is determined based on the quality assessment value of the reconstructed image of each original image, including:

[0029] The Global Quality Assessment (GQ) value for a given video frame is:

[0030]

[0031] Secondly, embodiments of this application provide an image semantic reconstruction quality assessment device based on attention weighting, comprising:

[0032] The acquisition unit is used to acquire video frames that include multiple consecutive frames of raw images;

[0033] The preprocessing unit is used to preprocess each frame of the original image to obtain the preprocessed original image of each frame.

[0034] The processing unit is used to input the pre-processed original image of each frame into the pre-trained TimesFormer model to obtain the attention map of the representation category of each original image output by the penultimate layer of the TimesFormer model.

[0035] The generation unit is used to generate a semantic weight map for each frame of the original image based on the attention map of the representation category of each frame of the original image.

[0036] The first determining unit is used to determine the quality assessment value of the reconstructed image of each original image by utilizing the semantic weight map of each original image.

[0037] The second determining unit is used to determine the global quality assessment value of a video frame based on the quality assessment value of the reconstructed image of each original image.

[0038] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of embodiments of this application.

[0039] Fourthly, an embodiment of this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of the embodiment of this application.

[0040] This application improves the accuracy of quality assessment values ​​for reconstructed images. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 A flowchart illustrating the attention-weighted image semantic reconstruction quality assessment method provided in this application embodiment;

[0043] Figure 2 A schematic diagram of attention weight extraction based on TimesFormer provided for an embodiment of this application;

[0044] Figure 3 A functional structure diagram of the attention-weighted image semantic reconstruction quality assessment device provided in the embodiments of this application;

[0045] Figure 4 A functional structure diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0047] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0048] First, a brief introduction to the design concept of the embodiments of this application will be given.

[0049] With the development of deep learning, especially the introduction of the Transformer architecture, video understanding technology has made significant progress. Compared with images, video quality assessment requires comprehensive consideration of features in both temporal and spatial dimensions. Traditional methods based on 3D convolutional networks struggle to capture a wide range of spatiotemporal feature interactions simultaneously due to the limitations of convolutional kernels. TimesFormer is a deep learning model specifically designed for video understanding, its innovation lying in the introduction of a separate spatiotemporal attention mechanism. Spatial attention primarily weights local regions of each frame, capturing the saliency of spatial features. Temporal attention, on the other hand, focuses on analyzing the temporal relationships between frames, capturing dynamically changing semantic relationships. Through this separate attention mechanism, TimesFormer can efficiently model the spatiotemporal distribution of semantically salient regions in a video, providing a strong foundation for semantically weighted image quality assessment.

[0050] The Transformer architecture is unique in its introduction of CLS Tokens. In TimesFormer, CLS Tokens are used to aggregate global spatiotemporal information, and their attention maps reflect the model's importance distribution for each region. Specifically, the final attention map of the CLS Token represents the model's attention level to key regions in a video frame. In dynamic scenes, the CLS Token's attention tends to focus on the moving areas of the target object. In static scenes, the CLS Token's attention focuses on semantically salient regions (such as faces or target objects). These characteristics make the CLS Token's attention map an important basis for constructing semantically weighted quality evaluation metrics.

[0051] In many real-world scenarios, semantically salient regions in images or videos are crucial to the user's perceived quality. For example, in video compression, the compression quality of moving objects or main subject areas significantly impacts the user experience, while the reconstruction effect of highly semantic regions such as faces is far more important than the background. Therefore, introducing a semantically weighted evaluation method not only more closely approximates human subjective perception but also compensates for the shortcomings of traditional methods at the semantic level, improving the accuracy of the quality assessment values ​​for reconstructed images.

[0052] After introducing the application scenarios and design concepts of the embodiments of this application, the technical solutions provided by the embodiments of this application will be described below.

[0053] like Figure 1 As shown, this application provides an image semantic reconstruction quality assessment method based on attention weighting, including the following steps:

[0054] Step 101: Obtain video frames that include multiple consecutive frames of original images;

[0055] Step 102: Preprocess each frame of the original image to obtain the preprocessed original image of each frame;

[0056] Step 103: Input the preprocessed original image of each frame into the pre-trained TimesFormer model to obtain the attention map of the representation category of each original image output by the penultimate layer of the TimesFormer model.

[0057] Step 104: Generate a semantic weight map for each original image frame based on the attention map of the representation category of each original image frame;

[0058] Step 105: Using the semantic weight map of each original image frame, determine the quality assessment value of the reconstructed image of each original image frame;

[0059] Among them, the reconstructed image is the image obtained by reconstructing the original image using methods such as MAE (mask autoencoder), interpolation-based reconstruction, and sparse representation-based reconstruction.

[0060] Step 106: Determine the global quality assessment value of the video frame based on the quality assessment value of the reconstructed image of each original image.

[0061] In some embodiments, each frame of the original image is preprocessed to obtain a preprocessed frame of the original image, including:

[0062] The video frame V can be represented as: V = {I1, I2, ..., I...} T}, where I t This represents the original image of frame t, where 1 ≤ t ≤ T; T is the frame number.

[0063] Each frame of the original image is resized to the fixed size required for the TimesFormer model input: 224×224.

[0064] The TimesFormer model, a video classification model, employs a separate spatiotemporal attention mechanism. Spatial attention weights local regions of each frame to capture the saliency of spatial features. Temporal attention models temporal features across frames, capturing dynamically changing semantic relationships. The penultimate layer attention map of the TimesFormer model (e.g., ...) is shown in the image. Figure 2(As shown) This indicates the importance of the classification identifier to each region, which integrates the spatiotemporal feature distribution.

[0065] In some embodiments, a semantic weight map for each original image is generated based on the attention map of the representation categories of each original image frame, including:

[0066] Obtain the attention map of the original image in frame t. Attention Map A ( t L) It includes N image patches, N = 14 × 14, and each image patch is 16 × 16 in size; attention map Each image patch includes M attention heads representing classification identifiers;

[0067] Calculate the region weight w of the nth image patch t,n :

[0068]

[0069] in, For attention graph The attention of the class identifier in the m-th attention head to the n-th image patch, where the index number of the class identifier is 0; 1≤n≤N;

[0070] Based on the region weight w of the nth image patch t,n Determine the semantic weight value W of its internal pixel (x,y). t (x,y):

[0071] W t (x,y)=w t,n

[0072] The semantic weight values ​​are normalized to obtain normalized semantic weight values.

[0073]

[0074] In some embodiments, the quality assessment value of the reconstructed image of each original image is determined using the semantic weight map of each original image frame, including:

[0075] Based on the original image I of frame t t and its reconstructed image Determine the reconstructed image First quality assessment value for each pixel Among them, I t (x, y) represents the original image I of frame t. t The pixel value at pixel (x,y); To reconstruct the image The pixel value at pixel (x,y);

[0076] Based on the semantic weight map of the original image in frame t, the reconstructed image is determined. Second Quality Assessment Value (SWQ) t :

[0077]

[0078] Where H is the vertical length of the original image in frame t, and W is the horizontal length of the original image in frame t.

[0079] In some embodiments, a global quality assessment value for a video frame is determined based on the quality assessment value of the reconstructed image of each original image, including:

[0080] The Global Quality Assessment (GQ) value for a given video frame is:

[0081]

[0082] Based on the above embodiments, this application provides an image semantic reconstruction quality assessment device based on attention weighting, see below. Figure 3 As shown, the attention-weighted image semantic reconstruction quality assessment device 200 provided in this application embodiment includes at least:

[0083] Acquisition unit 201 is used to acquire video frames including multiple consecutive original images;

[0084] The preprocessing unit 202 is used to preprocess each frame of the original image to obtain each frame of the original image after preprocessing.

[0085] The processing unit 203 is used to input the pre-processed original image of each frame into the pre-trained TimesFormer model to obtain the attention map of the representation category of each original image output by the penultimate layer of the TimesFormer model.

[0086] The generation unit 204 is used to generate a semantic weight map for each frame of the original image based on the attention map of the representation category of each frame of the original image.

[0087] The first determining unit 205 is used to determine the quality assessment value of the reconstructed image of each original image using the semantic weight map of each original image.

[0088] The second determining unit 206 is used to determine the global quality assessment value of a video frame based on the quality assessment value of the reconstructed image of each original image.

[0089] It should be noted that the principle of the image semantic reconstruction quality assessment device 200 based on attention weighting provided in this application embodiment to solve the technical problem is similar to the method provided in this application embodiment. Therefore, the implementation of the image semantic reconstruction quality assessment device 200 based on attention weighting provided in this application embodiment can refer to the implementation of the method provided in this application embodiment, and the repeated parts will not be described again.

[0090] Based on the above embodiments, this application also provides an electronic device, see below. Figure 4 As shown, the electronic device 300 provided in this application embodiment includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, it implements the image semantic reconstruction quality assessment method based on attention weighting provided in this application embodiment.

[0091] The electronic device 300 provided in this application embodiment may further include a bus 303 connecting different components (including processor 301 and memory 302). The bus 303 represents one or more types of bus structures, including memory bus, peripheral bus, local area bus, etc.

[0092] The memory 302 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 3021 and / or cache memory 3022, and may further include read-only memory (ROM) 3023.

[0093] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, including but not limited to: an operating subsystem, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0094] Electronic device 300 can also communicate with one or more external devices 304 (e.g., keyboard, remote control, etc.), and with one or more devices that enable a user to interact with electronic device 300 (e.g., mobile phone, computer, etc.), and / or with any device that enables electronic device 300 to communicate with one or more other electronic devices 300 (e.g., router, modem, etc.). This communication can be performed through input / output (I / O) interface 305. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 306. Figure 4 As shown, network adapter 306 communicates with other modules of electronic device 300 via bus 303. It should be understood that, although... Figure 4 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, and data backup storage subsystems.

[0095] It should be noted that, Figure 4 The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0096] This application also provides a computer-readable storage medium storing computer instructions. When executed by a processor, these computer instructions implement the attention-weighted image semantic reconstruction quality assessment method provided in this application. Specifically, the executable program can be built into or installed in an electronic device 300, so that the electronic device 300 can implement the attention-weighted image semantic reconstruction quality assessment method provided in this application by executing the built-in or installed executable program.

[0097] The image semantic reconstruction quality assessment method based on attention weighting provided in this application embodiment can also be implemented as a program product. The program product includes program code. When the program product can run on the electronic device 300, the program code is used to make the electronic device 300 execute the image semantic reconstruction quality assessment method based on attention weighting provided in this application embodiment.

[0098] The program product provided in this application embodiment can be any combination of one or more readable media, wherein the readable media can be a readable signal medium or a readable storage medium, and the readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0099] The program product provided in this application embodiment can be a CD-ROM and include program code, and can also run on a computing device. However, the program product provided in this application embodiment is not limited thereto. In this application embodiment, the readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0100] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0101] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application, and should all be covered within the scope of the claims of this application.

Claims

1. A method for evaluating the quality of image semantic reconstruction based on attention weighting, characterized in that, include: Acquire video frames that include multiple consecutive frames of raw images; Each frame of the original image is preprocessed to obtain the preprocessed original image of each frame. Each preprocessed original image frame is input into the pre-trained TimesFormer model to obtain the attention map representing the category of each original image frame output by the penultimate layer of the TimesFormer model. Based on the attention map of the representation categories of each original image frame, a semantic weight map of each original image frame is generated. The quality assessment value of the reconstructed image of each original image is determined by using the semantic weight map of each original image. The global quality assessment value of the video frame is determined based on the quality assessment value of the reconstructed image of each original image. Based on the attention map representing the categories of each original image frame, a semantic weight map for each original image frame is generated, including: Get the Attention map of the original frame image Attention map include Image blocks, The size of each image patch is Attention map Each image block includes An attention head representing a classification identifier; Calculate the first Region weights of each image patch : in, For attention graph The attention of the classification identifier in the m-th attention head to the n-th image patch, where the index number of the classification identifier is 0; ; Based on the Region weights of each image patch Determine its internal pixels semantic weight value : The semantic weight values ​​are normalized to obtain normalized semantic weight values. : 。 2. The image semantic reconstruction quality assessment method based on attention weighting according to claim 1, characterized in that, Each frame of the original image is preprocessed to obtain the preprocessed original image, including: video frames Represented as: ,in, Indicates the first Frame of original image, ; For the number of frames; Each frame of the original image is resized to the fixed size required for the TimesFormer model input: .

3. The image semantic reconstruction quality assessment method based on attention weighting according to claim 2, characterized in that, Using the semantic weight map of each original image frame, the quality assessment value of the reconstructed image for each original image frame is determined, including: Based on the Original frame image and its reconstructed image Determine the reconstructed image First quality assessment value for each pixel ,in, For the first Original frame image In pixels Pixel values; To reconstruct the image In pixels Pixel values; Based on the The semantic weight map of the original image is used to determine the reconstructed image. Second quality assessment value : in, For the first The vertical length of the original frame image, For the first The horizontal length of the original frame image.

4. The image semantic reconstruction quality assessment method based on attention weighting according to claim 3, characterized in that, Based on the quality assessment value of the reconstructed image of each original image frame, a global quality assessment value for the video frame is determined, including: Global quality assessment value of fixed video frames for: 。 5. An image semantic reconstruction quality assessment device based on attention weighting, characterized in that, include: The acquisition unit is used to acquire video frames that include multiple consecutive frames of raw images; The preprocessing unit is used to preprocess each frame of the original image to obtain the preprocessed original image of each frame. The processing unit is used to input the pre-processed original image of each frame into the pre-trained TimesFormer model to obtain the attention map of the representation category of each original image output by the penultimate layer of the TimesFormer model. The generation unit is used to generate a semantic weight map for each frame of the original image based on the attention map of the representation category of each frame of the original image. The first determining unit is used to determine the quality assessment value of the reconstructed image of each original image by utilizing the semantic weight map of each original image. The second determining unit is used to determine the global quality assessment value of the video frame based on the quality assessment value of the reconstructed image of each original image. The generating unit is specifically used for: Get the Attention map of the original frame image Attention map include Image blocks, The size of each image patch is Attention map Each image block includes An attention head representing a classification identifier; Calculate the first Region weights of each image patch : in, For attention graph The attention of the classification identifier in the m-th attention head to the n-th image patch, where the index number of the classification identifier is 0; ; Based on the Region weights of each image patch Determine its internal pixels semantic weight value : The semantic weight values ​​are normalized to obtain normalized semantic weight values. : 。 6. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-4.

Images