Image color difference measurement method based on attention and deformable convolution
By introducing attention and deformable convolution methods into image chromatic aberration measurement, combined with the feature extraction capabilities of CNN and VIT, the problem of traditional methods ignoring spatial structure and semantic information is solved, and a more accurate and reliable image chromatic aberration measurement is achieved. Suitable for complex images and extends to other image quality assessments.
Patent Information
- Application Number
- CN202411636965.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-15
- Publication Date
- 2025-01-17
AI Technical Summary
The existing chromatic aberration perception method is difficult to adapt to the complexity and diversity of modern photographic images, and ignores the spatial structure and semantic information in the image, resulting in a deviation from human visual perception. Traditional deep learning algorithms are processing complex global structures and semantics. The information is poorly evaluated when the image is imaged.
Using the image chromatic aberration measurement method based on attention and deformable convolution, local features and global semantic features are extracted through CNN and VIT respectively, deformable convolution is used for attention guidance, and score prediction is performed by combining high-frequency and low-frequency prediction. Get the final score, fusing local details and global semantic information.
The accuracy and reliability of image chromatic aberration metrics are significantly improved, and the results are consistent with human perceived results, suitable for images of different types and complexities, and can be extended to other image quality assessment tasks.
Smart Images
Figure CN119323608A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to an image color difference measurement method based on attention and deformable convolution. Background Art
[0002] With the popularity of smartphones and other digital imaging devices, photography has become an indispensable part of people's daily lives. As an important indicator for evaluating image quality, the color difference perception of photographic images is not only related to user experience, but also an important basis for evaluating the performance of image acquisition equipment. Color difference perception can reveal whether there is a fault in the image acquisition device, as well as the actual effect of image processing algorithms (such as image enhancement, image compression, image decompression, etc.).
[0003] However, existing color difference perception methods still have many problems when dealing with images of large size and complex content. Traditional color difference perception methods, such as those based on color space distance calculation, often have difficulty adapting to the complexity and diversity of modern photographic images. These methods usually only focus on the color difference itself, while ignoring the spatial structure and semantic information in the image, resulting in deviations between the evaluation results and human visual perception.
[0004] In recent years, deep learning-based image quality assessment algorithms have gradually begun to be used. These algorithms use deep learning models such as convolutional neural networks (CNNs) to extract local features from images and then evaluate image quality. However, traditional deep learning-based image quality assessment algorithms mainly focus on local features of images and lack the capture of global semantic information. This means that although these algorithms perform well in local feature extraction, their evaluation results are often unsatisfactory when processing images with complex global structures and semantic information. Summary of the invention
[0005] The purpose of the present invention is to propose an image color difference measurement method based on attention and deformable convolution, and the technical scheme can ensure that the result of the measurement method is consistent with the result of human perception.
[0006] To achieve the above objectives, the embodiments of the present disclosure provide an image color difference measurement method based on attention and deformable convolution, including: using CNN and VIT to extract local features and global semantic features of a reference image and a distorted image respectively; guiding attention through deformable convolution, and using the global semantic features of VIT to guide CNN to focus on salient areas of the image; fusing the attention-guided CNN features and the VIT global semantic features; using high-frequency prediction and low-frequency prediction dual branches to perform score prediction, and using weighted summation to fuse the low-frequency prediction score and the high-frequency prediction score to obtain the final score of the distorted image.
[0007] Beneficial effects of the basic scheme: CNN is used to extract local features of the image, and VIT is used to extract global semantic features of the image, ensuring that both details and overall features can be captured. Attention guidance is performed through deformable convolution, so that CNN can focus on the salient areas in the image and dynamically adjust the sampling position of the convolution kernel, thereby focusing on the image part that is more critical to color difference evaluation, showing good adaptability and flexibility when processing images of different types and complexities. The fusion of attention-guided CNN features and VIT's global semantic features combines the advantages of local details and global semantic information, providing a richer and more comprehensive feature representation for subsequent color difference measurement. High-frequency prediction and low-frequency prediction dual branches are used for score prediction to capture high-frequency detail information and low-frequency structural information in the image, respectively, which helps to evaluate the color difference of the image more carefully. This technical scheme significantly improves the accuracy and reliability of image color difference measurement by combining the feature extraction capabilities of CNN and VIT, introducing an attention mechanism for feature guidance, achieving effective fusion of features, and adopting a dual-branch score prediction strategy. This method is not only applicable to image color difference measurement, but can also be extended to other image quality assessment tasks, such as image clarity, sharpness, contrast, etc.
[0008] As an implementable preferred solution, CNN is used to extract local features and global semantic features of the reference image and the distorted image, including the following: Use the ResNet architecture to extract local features of the image, pass the input image through multiple convolutional layers and pooling layers to extract local features; use batch normalization and ReLU activation function to enhance feature representation; maintain the gradient flow of the deep network through residual connections to prevent the gradient vanishing problem; The reference image and distorted image features extracted by CNN are expressed as follows:
[0009] in, represents the input reference image, represents the reference image features; represents the distorted image of the input, Represents the distorted image features; Represents the CNN feature extraction process.
[0010] As an implementable preferred solution, the image is segmented into image blocks of fixed size; the segmented image blocks are converted into patch embeddings and position encoding information is added; the embeddings are processed using a Transformer encoder, the Transformer encoder includes multi-head self-attention and feed-forward neural networks, and the features of the position markers are removed as global features; The reference image and distorted image features extracted by VIT are expressed as follows:
[0011] in, represents the input reference image, represents the reference image features; represents the distorted image of the input, Represents the distorted image features; Represents the VIT feature extraction process.
[0012] As an implementable preferred solution, attention is guided by deformable convolution, and the global semantic features of VIT are used to guide CNN to focus on the salient areas of the image; including the following: Features of the reference image Perform nearest neighbor bilinear interpolation calculations; generate offsets through additional convolutional layers; adjust the position of the sampling grid according to the offset, and use a standard convolution kernel to perform convolution operations on the adjusted position to obtain and Adjusted feature map and .
[0013] As an implementable preferred solution, the offset is generated through an additional convolutional layer, including the following: The convolution input channel is 768*5, the output channel is 2*3*3, the input of the convolution layer is the feature map, the output is the offset field, and the number of channels is 2N, where N is the size of the area covered by the convolution kernel.
[0014] As an implementable preferred solution, the attention-guided CNN features and VIT global semantic features are fused, including the following: Will obtain and Fusion to obtain features ;Will and Fusion gets features ; The feature , and( - ) to obtain the final feature .
[0015] As an implementable preferred solution, the low-frequency prediction branch is used to perform score prediction, including the following contents: The scores of global features are predicted; the low-frequency prediction branch performs convolution operations on the fused features, performs feature smoothing, and uses the Sigmoid activation function to obtain the global scores of high-level features.
[0016] As an implementable preferred solution, the high-frequency prediction branch is used to perform score prediction, including the following contents: The high-frequency prediction branch performs a convolution operation on the fused features to smooth the features and obtain a high-frequency score prediction map using the convolution operation.
[0017] As an implementable optimal solution, a weighted sum is used to fuse the low-frequency prediction score and the high-frequency prediction score to obtain the final score. The formula is as follows: S
[0018] in, represents the low-frequency prediction score, represents the high-frequency prediction score, and S represents the global score.
[0019] As an implementable preferred solution, it also includes using MES LOSS as a loss function, learning the difference between the predicted score and the grund true to perform back-propagation learning. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 A logical diagram of an image color difference measurement method based on attention and deformable convolution.
[0021] Figure 2 Schematic diagram of the CD-Attention network structure.
[0022] Figure 3 Schematic diagram of VIT feature extraction principle.
[0023] Figure 4 Schematic diagram of the attention guidance mechanism.
[0024] Figure 5 Schematic diagram of the prediction module structure.
[0025] Figure 6 Schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0026] In order to make the technical solution and advantages of the present application clearer, the technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings. It will be understood that the specific embodiments described herein are only partial embodiments of the present invention, which are only used to explain the present application, rather than to limit the present application. It should be noted that the technical features or combinations of technical features described in the following embodiments should not be considered isolated, and they can be combined with each other to achieve better technical effects. The same reference numerals appearing in the drawings of the following embodiments represent the same features or components, which can be applied to different embodiments.
[0027] In addition, unless otherwise defined, the technical terms or scientific terms used in the description of the present invention should have the common meanings understood by those skilled in the art in the art to which the present invention belongs.
[0028] The present invention is further described in detail below in conjunction with the accompanying drawings: Reference numerals: electronic device 500 , processor 501 , communication interface 502 , memory 503 , bus 504 .
[0029] Embodiment 1 Reference Figure 1 ,A method for measuring color difference of an image based on attention and deformable convolution, includes the following steps.
[0030] Step S100, construct a color difference perception network CD-Attention, and use CNN (Convolutional Neural Network) and VIT (Vision Transformer) to extract local features and global semantic features of the reference image and the distorted image respectively. Figure 2 , is the network structure of CD-Attention, including feature extraction module, attention guidance module, feature fusion module and prediction module, specifically including: Step S101, CNN feature extraction of local detail texture features of the image, specifically includes: Step S101-1, input the reference image and the distorted image into CNN.
[0031] Step S101-2, using the ResNet architecture to extract local features of the image, the input image is passed through multiple convolutional layers and pooling layers to extract local features. Batch normalization and ReLU activation function are used to enhance feature representation. Residual connections are used to maintain the gradient flow of the deep network to prevent the gradient vanishing problem.
[0032] The reference image and distorted image features extracted by CNN are expressed as follows:
[0033] in, represents the input reference image, represents the reference image features; represents the distorted image of the input, Represents the distorted image features; Represents the CNN feature extraction process.
[0034] Step S102: VIT extracts global semantic features of the image. The principle of VIT feature extraction is to use the self-attention mechanism in the Transformer model to model the features of the image, which is different from the way CNN extracts local features of the image through convolutional layers and pooling layers. Figure 3 The Block structure of the ViT model is based on the Encoder structure of Transformer, including the Multi-head Attention structure. Figure 2 This is the feature extraction process of VIT, which specifically includes: Step S102 - 1 , dividing the image into image patches of fixed size.
[0035] Step S102-2: Convert the segmented image blocks into Patch Embeddings and add position encoding information.
[0036] Step S102-3, using the Transformer encoder to process the embedding, the Transformer encoder includes multi-head self-attention and feed-forward neural network, and removes the features of the position mark as the global features.
[0037] The reference image and distorted image features extracted by CNN are expressed as follows:
[0038] in, represents the input reference image, represents the reference image features; represents the distorted image of the input, Represents the distorted image features; Represents the CNN feature extraction process.
[0039] Step S200, attention is guided by deformable convolution, and the global semantic features of VIT are used to guide CNN to focus on the salient areas of the image, thereby enhancing the transformation modeling capability of CNN features. Figure 4 , specifically including: Step S201, the features of the reference image Perform nearest neighbor bilinear interpolation to accommodate sampling points with non-integer coordinates. Bilinear interpolation takes into account the values of the four nearest neighbor points around the sampling point and calculates the value of the sampling point by linear interpolation, thereby avoiding the offset that causes the sampling point to fall on non-integer coordinates.
[0040] Step S202, an offset is generated through an additional convolution layer. In this embodiment, the convolution input channel is 768*5, the output channel is 2*3*3, the input of the convolution layer is a feature map, and the output is an offset field. Its shape is the same as the input feature map, but the number of channels is 2N, where N is the size of the area covered by the convolution kernel, for example, a 3×3 convolution kernel, N=9. These 2N channels correspond to the offsets of each sampling point in the x and y directions.
[0041] Step S203, adjusting the position of the sampling grid according to the offset, and performing a convolution operation on the adjusted position using a standard convolution kernel to obtain an adjusted feature map.
[0042] Specifically, if the coordinates of the center point of the convolution kernel are (p, q) and the offset is (Δx, Δy), then the adjusted sampling point coordinates are (p+Δx, q+Δy). After adjusting the positions of the sampling points, the standard convolution kernel is used to perform convolution operations on these adjusted positions to obtain and Adjusted feature map: and .
[0043] The position of the convolution kernel is adjusted by learning the offset on the input feature map, so that the convolution kernel can adapt to the geometric changes of the object, such as scale, posture, viewpoint and component deformation. This technical solution solves the problem that the traditional method of extracting image features through CNN has fixed convolution kernel sampling points, making image feature extraction closer to the human perception process of images.
[0044] Step S300, the attention-guided CNN features and the VIT global semantic features are fused. Specifically, the acquired and Fusion to obtain features ;Will and Fusion gets features ; The feature , and( - ) to obtain the final feature .
[0045] Step S400, using high-frequency prediction and low-frequency prediction branches to perform score prediction, and using weighted summation to obtain the final score, specifically includes: Step S401, using high frequency prediction and low frequency prediction dual branches to predict the final score of the distorted image, referring to Figure 5, specifically including: Step S401-1, predict the scores of global features to obtain a low-frequency prediction score map. The low-frequency prediction branch mainly focuses on the low-frequency components in the image or signal, that is, the slowly changing and relatively smooth parts. The low-frequency prediction branch performs a convolution operation on the fused features, performs feature smoothing, and uses the Sigmoid activation function to obtain the global score of the high-level features.
[0046] Step S401-2, the high-frequency prediction branch performs convolution operation on the fused features, performs feature smoothing, and uses convolution operation to obtain a high-frequency score prediction map. The high-frequency prediction branch mainly focuses on the high-frequency components in the image or signal, that is, the parts that change rapidly and are rich in details.
[0047] Step S402: The low-frequency prediction score and the high-frequency prediction score are fused by weighted average, and the formula is as follows: S
[0048] in, represents the low-frequency prediction score, represents the high-frequency prediction score, and S represents the global score.
[0049] because Each feature in the feature map is a high-level feature obtained through feature fusion. The weighted average method can maximize the use of the information of each feature, and the low-frequency score is used as the weight to learn the impact of global features on the results, allowing the model to focus on the most relevant parts of the input features. The model can learn more accurate feature representations, thereby improving the model's expressiveness and generalization capabilities.
[0050] Step S500, back propagation and optimization of network parameters are performed. MES LOSS is used as the loss function, and the difference between the predicted score and the grund true is learned for back propagation learning.
[0051] It was verified on the large-scale SPCD dataset, and the statistical correlation coefficient with human ratings reached 91.43%, proving that CD-Attention can simulate the process of human color difference measurement, the measurement structure is highly correlated with human perception results, and has good generalization performance.
[0052] The disclosed embodiment also provides an image color difference measurement system based on attention and deformable convolution, which uses the above-mentioned image color difference measurement method based on attention and deformable convolution.
[0053] The embodiment of the present disclosure also provides a storage medium, in which a computer program is stored. When the computer program is executed by a processor, all steps of the above-mentioned image color difference measurement method based on attention and deformable convolution can be implemented.
[0054] A person of ordinary skill in the art can understand that the implementation of all or part of the process in a method for measuring image color difference based on attention and deformable convolution can be completed by instructing related hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include a process of various embodiments of a method for measuring image color difference based on attention and deformable convolution. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0055] The embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the above-mentioned image color difference measurement method based on attention and deformable convolution are implemented. In the embodiment of the present application, the processor is the control center of the computer system, which can be a processor of a physical machine or a processor of a virtual machine.
[0056] Reference Figure 6, the electronic device 500 includes: at least one processor 501, at least one communication interface 502, at least one memory 503 and at least one bus 504. The bus 504 is used to realize the connection and communication between these components, the communication interface 502 is used to communicate signaling or data with other node devices, and the memory 503 stores machine-readable instructions executable by the processor 501. When the electronic device 500 is running, the processor 501 communicates with the memory 503 through the bus 504, and when the machine-readable instructions are called by the processor 501, the steps of the image color difference measurement method based on attention and deformable convolution are executed.
[0057] The above contents are only embodiments of the present invention. The common sense such as the known specific structures and characteristics in the scheme is not described in detail here. The ordinary technicians in the relevant field know all the common technical knowledge in the technical field to which the invention belongs before the application date or the priority date, can obtain all the existing technologies in the field, and have the ability to apply the conventional experimental means before that date. The ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for the ordinary technicians in the relevant field to implement this application. It should be pointed out that for the technicians in this field, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the protection scope of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent. The protection scope required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to explain the content of the claims.
Claims
1. A method for measuring image color difference based on attention and deformable convolution, characterized by: CNN and VIT are used to extract local features and global semantic features of the reference image and distorted image respectively; attention is guided by deformable convolution, and the global semantic features of VIT are used to guide CNN to focus on the salient areas of the image; the attention-guided CNN features and VIT global semantic features are fused; high-frequency prediction and low-frequency prediction dual branches are used for score prediction, and weighted sum is used to fuse the low-frequency prediction score and the high-frequency prediction score to obtain the final score of the distorted image.
2. The image color difference measurement method based on attention and deformable convolution according to claim 1, characterized in that: CNN is used to extract local features and global semantic features of the reference image and the distorted image, including the following: Use the ResNet architecture to extract local features of the image, pass the input image through multiple convolutional layers and pooling layers to extract local features; use batch normalization and ReLU activation function to enhance feature representation; maintain the gradient flow of the deep network through residual connections to prevent the gradient vanishing problem; The reference image and distorted image features extracted by CNN are expressed as follows: in, represents the input reference image, represents the reference image features; represents the distorted image of the input, Represents the distorted image features; Represents the CNN feature extraction process.
3. The image color difference measurement method based on attention and deformable convolution according to claim 1, characterized in that: Segment the image into fixed-size image blocks; convert the segmented image blocks into patch embeddings and add position encoding information; process the embeddings using a Transformer encoder, which contains multi-head self-attention and feed-forward neural networks, and removes the features of position markers as global features; The reference image and distorted image features extracted by VIT are expressed as follows: in, represents the input reference image, represents the reference image features; represents the distorted image of the input, Represents the distorted image features; Represents the VIT feature extraction process.
4. The image color difference measurement method based on attention and deformable convolution according to claim 3, characterized in that: Attention is guided by deformable convolution, and the global semantic features of VIT are used to guide CNN to focus on the salient areas of the image; including the following: Features of the reference image Perform nearest neighbor bilinear interpolation calculations; generate offsets through additional convolutional layers; adjust the position of the sampling grid according to the offset, and use a standard convolution kernel to perform convolution operations on the adjusted position to obtain and Adjusted feature map and .
5. The image color difference measurement method based on attention and deformable convolution according to claim 4, characterized in that: The offsets are generated through an additional convolutional layer, including the following: The convolution input channel is 768*5, the output channel is 2*3*3, the input of the convolution layer is the feature map, the output is the offset field, and the number of channels is 2N, where N is the size of the area covered by the convolution kernel.
6. The image color difference measurement method based on attention and deformable convolution according to claim 4 or 5, characterized in that: The attention-guided CNN features and VIT global semantic features are fused, including the following: Will obtain and Fusion to obtain features ;Will and Fusion gets features ; The feature , and( - ) to obtain the final feature .
7. The image color difference measurement method based on attention and deformable convolution according to claim 1, characterized in that: The low-frequency prediction branch is used for score prediction, including the following: Predict the scores of global features; The low-frequency prediction branch performs convolution operations on the fused features to smooth the features and uses the Sigmoid activation function to obtain the global score in the high-level features.
8. The image color difference measurement method based on attention and deformable convolution according to claim 7, characterized in that: The high-frequency prediction branch is used to perform score prediction, including the following: The high-frequency prediction branch performs a convolution operation on the fused features to smooth the features and obtain a high-frequency score prediction map using the convolution operation.
9. The image color difference measurement method based on attention and deformable convolution according to claim 8, characterized in that: The low-frequency prediction score and the high-frequency prediction score are combined by weighted summation to obtain the final score. The formula is as follows: S in, represents the low-frequency prediction score, represents the high-frequency prediction score, and S represents the global score.
10. The image color difference measurement method based on attention and deformable convolution according to claim 1, characterized in that: It also includes back-propagation learning using MES LOSS as the loss function and learning the difference between the predicted score and the grund true.