Color difference prediction method, system, terminal and medium for image pair

Abstract

The application provides a color difference prediction method and system for an image pair, a terminal and a medium, comprising: converting two images of an original image pair in an RGB color space into two image pairs based on LAB and HSV color spaces, to obtain three image pairs based on LAB, RGB and HSV color spaces; extracting features of the three image pairs by using a double-flow neural network, to obtain low-level to high-level feature maps of each image, and to obtain a color difference map of each image pair; connecting the color difference maps, and calculating the color difference of the image pair in each color space; and performing average pooling on the color differences of the three image pairs based on LAB, RGB and HSV color spaces, to obtain the color difference of the original image pair. The application can effectively predict the color difference of the image pair.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically, to a method, system, terminal, and medium for predicting color differences in image pairs. Background Technology

[0002] Color difference in images has always been a hot research topic. The classic definition of color includes both physical and psychological properties. The physical properties of color are wavelength, brightness, and color purity. The psychological properties of color are hue, saturation, and value. Each psychological property of a color is influenced by its physical properties. Every color perceived by the human eye is a combined effect of these three elements, known as the Munsell color system. However, the colors of an image can change during capture, transmission, and display. For example, the colors of natural scene images obtained by different image capture devices may differ. The colors of images processed by various image processing techniques, such as color enhancement, color correction, and tone transfer, also vary. Furthermore, the colors of the same image represented in different color spaces also differ. Therefore, it is crucial to develop an effective method for calculating the color difference (CD) of an image. Summary of the Invention

[0003] To facilitate the study of color differences between image pairs, the present invention aims to provide a method, system, terminal, and medium for predicting color differences between image pairs, which can effectively predict color differences between image pairs.

[0004] The first objective of this invention is to provide a method for predicting the color difference of image pairs, comprising:

[0005] Two images of an original image pair based on the RGB color space are converted into two image pairs based on the LAB and HSV color spaces, resulting in three image pairs based on the LAB, RGB, and HSV color spaces.

[0006] A two-stream neural network is used to extract features from the three pairs of images to obtain low-level to high-level feature maps for each image and to obtain color difference maps for each pair of images.

[0007] By connecting the color difference map, the color difference of each image pair in each color space is obtained through color difference calculation; average pooling is performed on the color differences of the three image pairs based on LAB, RGB and HSV color spaces to obtain the color difference of the original image pair.

[0008] Optionally, the two images of an original image pair based on the RGB color space are converted into two image pairs based on the LAB and HSV color spaces, wherein the HSV and LAB color spaces are obtained by nonlinear transformation from the RGB color space.

[0009] Optionally, a two-stream neural network is used to extract features from the three pairs of images, wherein: the two sub-networks of the two-stream neural network are VGG16 networks with shared weights, using pooling layers as boundaries; the VGG16 network is divided into 6 blocks, and the 6 blocks of the VGG16 network are divided into three stages: the first stage consists of the first and second blocks of the VGG16 network; the second stage consists of the third block of the VGG16 network; and the third stage consists of the fourth and fifth blocks of the VGG16 network; each image pair is input into the two-stream neural network, and the low-level to high-level feature maps of each image are extracted sequentially through the above three stages; the color difference map of each image pair is calculated based on the three pairs of feature maps corresponding to the three pairs of image pairs.

[0010] Optionally, the color difference maps are concatenated, and the color difference of the image pairs in each color space is obtained by color difference calculation, including: concatenating the color difference maps, obtaining the color difference of the image pairs in each color space through three fully connected layers, that is, obtaining the predicted color difference of the LAB-based image pairs, the predicted color difference of the RGB-based image pairs, and the predicted color difference of the HSV-based image pairs.

[0011] A second objective of this invention is to provide a color difference prediction system for image pairs, comprising:

[0012] Color space transformation module: Converts two images of an original image pair based on the RGB color space into two image pairs based on the LAB and HSV color spaces, resulting in three image pairs based on the LAB, RGB, and HSV color spaces;

[0013] Feature extraction module: A two-stream neural network is used to extract features from the three pairs of images in the color space transformation module to obtain low-level to high-level feature maps for each image and to obtain the color difference map for each image pair;

[0014] Color difference prediction module: Connects to the color difference map obtained by the feature extraction module, and obtains the color difference of each image pair in each color space through color difference calculation; performs average pooling on the color difference of the three image pairs based on LAB, RGB and HSV color spaces to obtain the color difference of the original image pair.

[0015] A third objective of this invention is to provide a color difference prediction terminal for image pairs, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the program to perform the color difference prediction method for the image pairs.

[0016] A fourth objective of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the color difference prediction method for the image pair.

[0017] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0018] The color difference prediction method and system for image pairs provided by this invention first converts the original RGB-based image pairs into LAB-based and HSV-based image pairs through a network's color space conversion module. Then, using a two-stream neural network, feature extraction is performed on three image pairs in three different color spaces to obtain low-level to high-level feature maps for each image, and a color difference map for each image pair is obtained. Finally, the color difference is calculated to obtain the color difference of the image pairs in each color space. Average pooling is then performed on the color differences of the three image pairs in different color spaces to obtain the color difference of the original image pair. This method can effectively predict the color difference of image pairs, and its performance is superior to currently proposed state-of-the-art image pair color difference calculation methods. Attached Figure Description

[0019] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0020] Figure 1 This is a flowchart of a color difference prediction method for image pairs according to an embodiment of the present invention;

[0021] Figure 2 This is a block diagram of a color difference prediction system for image pairs according to an embodiment of the present invention. Detailed Implementation

[0022] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0023] The most successful color space currently used for calculating color difference (CD) between image pairs is the Lab Color Space, also known as the CIELAB color space. LAB is a device-independent color space model and a color space model based on physiological characteristics. Most CD metrics are built upon the LAB color space, such as the well-known CIE76 and CIEDE2000. Besides LAB, there are also HSI, OSA-UCS, and RGB color spaces. Among these, RGB is the most common, and most images are stored and rendered in RGB. The most common CD metric in RGB is the distance-CD formula. However, the RGB color space seems to have less correlation with human color perception differentiation. Another common color space in image processing is the HSV color space, represented by three elements of the Munsell color system. Unlike RGB, HSV's color representation is consistent with human color perception differentiation, and therefore it is frequently used in various image processing techniques, such as image segmentation and image retrieval. Therefore, the study of color difference between image pairs is very important.

[0024] This invention provides a method for predicting the color difference of image pairs, which can effectively predict the color difference of image pairs, and can be well used for studying the color changes of images during image capture, transmission and display.

[0025] Reference Figure 1 As shown, an embodiment of the present invention provides a method for predicting the color difference of an image pair. This embodiment includes the following steps:

[0026] S1, convert the two images of an original image pair based on the RGB color space into two image pairs based on the LAB and HSV color spaces, to obtain three image pairs based on the LAB, RGB and HSV color spaces;

[0027] In this step, a color space conversion module can be used to convert the original RGB-based image pairs into LAB-based and HSV-based image pairs. Optionally, the HSV and LAB color spaces are obtained through non-linear transformations from the RGB color space.

[0028] S2, a two-stream neural network is used to extract features from three pairs of images respectively, to obtain low-level to high-level feature maps of each image, and to obtain the color difference map of each pair of images;

[0029] In this step, the feature extraction module extracts features from three pairs of image pairs in three different color spaces. The feature extraction module can employ a two-stream neural network.

[0030] In one possible implementation, the two sub-networks of the two-stream neural network are VGG16 networks with shared weights. If pooling layers are used as boundaries, the VGG16 network can be divided into six blocks. These six blocks are further divided into three stages. The first stage consists of the first and second blocks of the VGG16 network; the second stage includes the third block; and the third stage consists of the fourth and fifth blocks. This feature extraction module can simultaneously extract low- to high-level feature maps from both images in an image pair and obtain the color difference map for each image pair. In this embodiment, the two-stream neural network with the above structure can simultaneously input two images of a color difference pair, facilitating simultaneous extraction and learning of image features, and enabling weight sharing, thus saving memory. Other network structures cannot simultaneously extract and analyze features from two images in the same image pair, and their network weights are relatively large.

[0031] S3 connects the color difference map and calculates the color difference of each image pair in each color space; average pooling is performed on the color differences of the three image pairs based on LAB, RGB and HSV color spaces to obtain the color difference of the original image pairs.

[0032] In this step, the color difference maps of three image pairs in different color spaces are concatenated. Then, the color difference of the image pairs in each color space is predicted through three fully connected layers, including: the color difference of LAB-based image pairs, the color difference of RGB-based image pairs, and the color difference of HSV-based image pairs. Finally, by performing average pooling on the predicted color differences of the image pairs in these three color spaces, the color difference of the original image pairs can be predicted.

[0033] The above embodiments of the present invention can effectively predict the color difference of image pairs by extracting features of different color spaces and calculating the corresponding color differences.

[0034] Reference Figure 2 As shown, an embodiment of the present invention provides a color difference prediction system for image pairs. Specifically, the system in this embodiment includes three modules:

[0035] Color space transformation module: Converts two images of an original image pair based on the RGB color space into two image pairs based on the LAB and HSV color spaces, resulting in three image pairs based on the LAB, RGB, and HSV color spaces;

[0036] Feature extraction module: A two-stream neural network is used to extract features from the three pairs of images in the color space transformation module to obtain low-level to high-level feature maps for each image and to obtain the color difference map for each image pair;

[0037] Color difference prediction module: Connects to the color difference map obtained by the feature extraction module, and obtains the color difference of each image pair in each color space through color difference calculation; performs average pooling on the color difference of the three image pairs based on LAB, RGB and HSV color spaces to obtain the color difference of the original image pair.

[0038] In this embodiment, the color space conversion module converts the original RGB-based image pairs into LAB-based and HSV-based image pairs. In one possible embodiment, the HSV and LAB color spaces are obtained through a non-linear transformation performed from the RGB color space.

[0039] In this embodiment, the construction process of the feature extraction module is as follows:

[0040] A two-stream neural network is used to extract features from image pairs. The two sub-networks of the two-stream neural network are VGG16 networks with shared weights. If pooling layers are used as boundaries, the VGG16 network can be divided into six blocks. These six blocks are further divided into three stages. The first stage consists of the first and second blocks of the VGG16 network; the second stage includes the third block; and the third stage consists of the fourth and fifth blocks.

[0041] Therefore, for an image pair I1 and I2, the feature map extracted in the first stage is:

[0042]

[0043]

[0044] in, and These are the feature maps I1 and I2 extracted in the first stage, w 1 For shared weights. 1 These are the weights learned by the two-stream neural network. Their values ​​are not specifically required and change during the network learning process.

[0045] Then, the extracted feature map and Input them separately into the second stage:

[0046]

[0047]

[0048] in, and These are the feature maps I1 and I2 extracted in the second stage, w 2 For shared weights, w 2These are the weights learned by the two-stream neural network. Their values ​​are not specifically required and change during the network learning process.

[0049] Finally, the extracted feature maps and Input them into the third stage S respectively:

[0050]

[0051]

[0052] in, and These are the feature maps I1 and I2 extracted in the third stage, w 3 For shared weights, w 3 These are the weights learned by the two-stream neural network. Their values ​​are not specifically required and change during the network learning process.

[0053] Using the three pairs of extracted feature maps ( and and and To calculate the color difference map of the input image pair:

[0054]

[0055] Then CDM 1 CDM 2 and CDM 3 Connect them together and obtain the final chromatic difference feature (CF) of the input image pair through convolutional blocks.

[0056] In this embodiment, the color difference prediction module connects the stitched feature CF with three fully connected layers to predict the color difference of the image pair:

[0057]

[0058] Among them, FC1, FC2, and FC3 are fully connected layers, and ReLU is the correction linearity unit. The predicted color difference for the input image pair.

[0059] Using the formulas described above, we can obtain the predicted color difference for LAB-based image pairs, the predicted color difference for RGB-based image pairs, and the predicted color difference for HSV-based image pairs. Finally, by performing average pooling on these pairs, we can obtain the predicted color difference for the original image pairs.

[0060] Implementation results:

[0061] To verify the effectiveness of the image pair color difference prediction method provided in the above embodiments of the present invention, experimental tests were conducted on the SPCD database. The SPCD database is the world's largest natural image database, containing 10,000 image pairs and their perceived color differences; it was specifically constructed for perceptual color difference evaluation in smartphone photography. In the experiment, the database was randomly segmented. Specifically, 80% of the images in the database were segmented into the training set, while the remaining images were divided into the test set. To avoid bias caused by the randomness of the training and test set segmentation, the above segmentation process was repeated 10 times, and the average results were reported.

[0062] The performance of the method proposed in this embodiment of the invention in calculating color difference of an image is compared with that of some advanced image color difference calculation methods: CIE76, CIEDE2000, CMC, Imai, Ouni, Zhang, Hong, Pedersen, Lee05, Lee14, Huertas, Simone, Orti and CD-Net.

[0063] The following three evaluation criteria were selected in the experiment to measure the performance of the evaluation method: Pearson linear correlation coefficients (PLCC), Spearman rank order correlation coefficients (SRCC), and root mean squared error (RMSE).

[0064] The performance test results are shown in Table 1. It is clear that the proposed method exhibits the best performance in predicting the color difference of image pairs. Specifically, the method proposed in this embodiment (last row) and CD-Net demonstrate good prediction performance on the SPCD database. In contrast, traditional color difference calculation metrics perform poorly in predicting the color difference of image pairs in the SPCD database.

[0065] Table 1

[0066]

[0067] As can be seen from the above data, the color difference prediction method for image pairs in this embodiment of the invention can effectively predict the color difference of image pairs.

[0068] Based on the same concept described above, in another embodiment of the present invention, a color difference prediction terminal for image pairs is also provided, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the program, it is used to execute the color difference prediction method for the image pairs.

[0069] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the aforementioned computer programs, computer instructions, etc., can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0070] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0071] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0072] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0073] In this embodiment of the invention, a computer-readable storage medium is also provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of a color difference prediction method for an image pair in any of the above embodiments are implemented.

[0074] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0075] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0076] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0077] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0078] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0079] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for predicting the color difference of an image pair, characterized in that, include: Two images of an original image pair based on the RGB color space are converted into two image pairs based on the LAB and HSV color spaces, resulting in three image pairs based on the LAB, RGB, and HSV color spaces. A two-stream neural network is used to extract features from the three image pairs to obtain low-level to high-level feature maps for each image and to obtain a color difference map for each image pair. The two sub-networks of the two-stream neural network are VGG16 networks with shared weights, using pooling layers as boundaries. The VGG16 network is divided into six blocks, which are further divided into three stages: the first stage consists of the first and second blocks of the VGG16 network; the second stage consists of the third block of the VGG16 network; and the third stage consists of the fourth and fifth blocks of the VGG16 network. Each image pair is input into the two-stream neural network and sequentially passed through the above three stages to extract low-level to high-level feature maps for each image. The color difference map for each image pair is calculated based on the three feature maps corresponding to the extracted three image pairs. By connecting the color difference map, the color difference of the image pair in each color space is calculated; average pooling is performed on the color difference of the three image pairs based on LAB, RGB and HSV color spaces to obtain the color difference of the original image pair.

2. The color difference prediction method for image pairs according to claim 1, characterized in that, The process involves converting two images from a single original image pair based on the RGB color space into two image pairs based on the LAB and HSV color spaces, where: HSV and LAB color spaces are obtained by non-linear transformation of the RGB color space.

3. The color difference prediction method for image pairs according to claim 1, characterized in that, By concatenating the color difference maps and calculating the color difference of each image pair in each color space, the following steps are taken: By connecting the color difference maps, the color difference of image pairs in each color space is obtained through three fully connected layers, resulting in the predicted color difference of LAB-based image pairs, the predicted color difference of RGB-based image pairs, and the predicted color difference of HSV-based image pairs.

4. A color difference prediction system for image pairs, characterized in that, include: Color space transformation module: Converts two images of an original image pair based on the RGB color space into two image pairs based on the LAB and HSV color spaces, resulting in three image pairs based on the LAB, RGB, and HSV color spaces; Feature Extraction Module: A two-stream neural network is used to extract features from the three image pairs of the color space transformation module, obtaining low-level to high-level feature maps for each image and a color difference map for each image pair. The two sub-networks of the two-stream neural network are VGG16 networks with shared weights, using pooling layers as boundaries. The VGG16 network is divided into six blocks, which are further divided into three stages: the first stage consists of the first and second blocks of the VGG16 network; the second stage consists of the third block of the VGG16 network; and the third stage consists of the fourth and fifth blocks of the VGG16 network. Each image pair is input into the two-stream neural network and sequentially passed through the above three stages to extract low-level to high-level feature maps for each image. The color difference map for each image pair is calculated based on the three feature maps corresponding to the extracted three image pairs. Color difference prediction module: Connects to the color difference map obtained by the feature extraction module, and calculates the color difference of each image pair in the color space; The color difference of the three image pairs based on LAB, RGB and HSV color spaces is averaged and pooled to obtain the color difference of the original image pairs.

5. The color difference prediction system for image pairs according to claim 4, characterized in that, The color space transformation module, wherein: HSV and LAB color spaces are obtained by non-linear transformation of the RGB color space.

6. The color difference prediction system for image pairs according to claim 4, characterized in that, The color difference prediction module includes: By connecting the color difference maps, the color difference of image pairs in each color space is obtained through three fully connected layers, resulting in the predicted color difference of LAB-based image pairs, the predicted color difference of RGB-based image pairs, and the predicted color difference of HSV-based image pairs.

7. A color difference prediction terminal for image pairs, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the program, it is used to perform the method described in any one of claims 1-3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-3.

Images