Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention discloses an image processing method and device, electronic equipment and a computer readable storage medium, and can enable the color of a picture shot by a camera to be closer to the color observed by human eyes, thereby improving the color expression effect of the image. The method comprises the steps of collecting a spectral image of a target scene through a spectral camera, and collecting a first color image of the target scene through a color camera; converting the spectral image according to the human eye spectral response curve to generate a second color image; and performing color adjustment on the first color image through the second color image to obtain a target image corresponding to the first color image.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular, to an image processing method, apparatus, electronic device, and computer-readable storage medium. Background Art

[0002] Currently, there is usually a certain difference between the colors of the images captured by a camera and the colors actually observed by the human eye, thereby reducing the color performance effect of the images. Summary of the Invention

[0003] This application expects to provide an image processing method, apparatus, electronic device, and computer-readable storage medium that can improve the color performance effect of images.

[0004] The technical solution of this application is implemented as follows:

[0005] In a first aspect, this application provides an image processing method, and the method includes:

[0006] Collect a spectral image of a target scene through a spectral camera, and collect a first color image of the target scene through a color camera;

[0007] Convert the spectral image according to the human eye spectral response curve to generate a second color image;

[0008] Adjust the color of the first color image through the second color image to determine the target image corresponding to the first color image.

[0009] In a second aspect, this application provides an image processing apparatus, and the apparatus includes:

[0010] A collection module, configured to collect a spectral image of a target scene through a spectral camera, and collect a first color image of the target scene through a color camera;

[0011] A generation module, configured to convert the spectral image according to the human eye spectral response curve to generate a second color image;

[0012] An adjustment module, configured to adjust the color of the first color image through the second color image to determine the target image corresponding to the first color image.

[0013] Optionally, the adjustment module is further configured to determine the color mapping relationship between the first color image and the second color image by performing pixel matching on the first color image and the second color image; and adjust the color of the first color image according to the color mapping relationship to determine the target image.

[0014] Optionally, the adjustment module is further configured to perform pixel color clustering on the first color image and the second color image respectively, determine at least one first color clustering center corresponding to the first color image, and at least one second color clustering center corresponding to the second color image; based on the at least one first color clustering center and the at least one second color clustering center, perform color matching on the pixels in the first color image and the second color image to determine at least one pair of matching pixels; and determine a color mapping relationship between the first color image and the second color image according to the at least one pair of matching pixels.

[0015] Optionally, the adjustment module is further configured to determine at least one first target pixel corresponding to the at least one first color clustering center according to the color similarity between the pixels in the first color image and the first color clustering center; determine at least one second target pixel corresponding to the at least one second color clustering center according to the color similarity between the pixels in the second color image and the second color clustering center; and perform color matching on the at least one first target pixel and the at least one second target pixel to determine the at least one pair of matching pixels.

[0016] Optionally, for each first target pixel in the at least one first target pixel, the adjustment module is further configured to determine at least one initial matching pixel corresponding to each first target pixel from the at least one second target pixel based on a color similarity matching threshold; and determine a matching pixel corresponding to each first target pixel from the at least one initial matching pixel according to the coordinates of the initial matching pixel and the coordinates of each first target pixel, and use each first target pixel and its corresponding matching pixel as a pair of matching pixels, so as to determine the at least one pair of matching pixels.

[0017] Optionally, before the adjustment module determines a matching pixel corresponding to each second target pixel from the at least one initial matching pixel according to the coordinates of the initial matching pixel and the coordinates of each second target pixel, the adjustment module is further configured to perform coordinate transformation on the original coordinates of each second target pixel based on the first resolution of the color camera and the second resolution of the spectral camera to determine the coordinates of each second target pixel.

[0018] Optionally, the adjustment module is further configured to calculate a color transformation matrix based on the color correspondence relationship between each pair of matching pixels in the at least one pair of matching pixels as the color mapping relationship.

[0019] Optionally, before the adjustment module separately performs pixel color clustering on the first color image and the second color image to determine at least one first color clustering center corresponding to the first color image and at least one second color clustering center corresponding to the second color image, the adjustment module also performs image enhancement on the first color image and the second color image; and determines a second original color value corresponding to the second target pixel in each pair of matching pixels and a first original color value corresponding to the matching pixel corresponding to the second target pixel; the second original color value and the first original color value are color values in the color image before image enhancement; and calculates the color transformation matrix according to the color correspondence relationship between the first original color value and the second original color value corresponding to each pair of matching pixels in the at least one pair of matching pixels.

[0020] Optionally, the first color image and the second color image include at least one color channel, and the adjustment module is further configured to perform clustering on the first color image by calculating the color similarity of the pixels in the first color image on the at least one color channel to determine the at least one first color clustering center; and perform clustering on the second color image by calculating the color similarity of the pixels in the second color image on the at least one color channel to determine the at least one second color clustering center.

[0021] Optionally, the human eye spectral response curve includes at least one response curve corresponding to at least one color channel; the generation module is further configured to determine a spectral curve corresponding to each spectral pixel in the spectral image; for each spectral pixel, perform band weighting and fusion processing on the spectral curve corresponding to the spectral pixel according to each response curve in the at least one response curve to determine the color value corresponding to each spectral pixel in each color channel of the at least one color channel; and thus generate the second color image.

[0022] Optionally, the difference in the field of view angles of the spectral camera and the color camera is less than the field of view angle difference threshold.

[0023] In a third aspect, the present application provides an electronic device, including a memory and a processor; wherein,

[0024] The memory is used to store executable instructions;

[0025] The processor is configured to implement the image processing method provided in the embodiments of the present application when executing the executable instructions stored in the memory.

[0026] In a fourth aspect, the present application provides a computer-readable storage medium storing executable instructions for causing a processor to implement the image processing method provided in the embodiments of the present application when executed.

[0027] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which when executed by a processor implement the image processing method provided by the embodiment of the present application.

[0028] The present application provides an image processing method, apparatus, electronic device, and computer-readable storage medium. A spectral image of a target scene is collected by a spectral camera, and a first color image of the target scene is collected by a color camera. The spectral image is converted to generate a second color image according to the human eye spectral response curve. The first color image is color-adjusted by the second color image to determine a target image corresponding to the first color image. In this way, based on the image collection of the same target scene by the spectral camera and the color camera, the spectral image captured by the spectral camera is converted into a second color image using the human eye spectral response curve, and the first color image is color-adjusted by the second color image, so that the obtained target image is closer to the color effect observed by the human eye, thereby being able to better restore the color of the scene captured by the color camera as seen by the human eye and improving the color performance effect of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is an optional flowchart of the image processing method provided by the embodiment of the present application;

[0030] Figure 2 is an optional schematic diagram of the positional relationship between the color camera and the spectral camera provided by the embodiment of the present application;

[0031] Figure 3 is an optional schematic diagram of the effect of the human eye spectral response curve provided by the embodiment of the present application;

[0032] Figure 4 is an optional schematic diagram of the effect of the spectral curve corresponding to the spectral pixel provided by the embodiment of the present application;

[0033] Figure 5 is an optional flowchart of the image processing method provided by the embodiment of the present application;

[0034] Figure 6 is an optional flowchart of the image processing method provided by the embodiment of the present application;

[0035] Figure 7 is an optional schematic diagram of the clustering of pixel colors provided by the embodiment of the present application;

[0036] Figure 8 is an optional schematic diagram of the effect of image enhancement provided by the embodiment of the present application;

[0037] Figure 9 An optional flowchart of the image processing method applied to the actual scenario provided by the embodiment of the present application;

[0038] Figure 10 A schematic structural diagram of an image processing apparatus provided by the embodiment of the present application;

[0039] Figure 11 A schematic structural diagram of an electronic device provided by the embodiment of the present application. Detailed implementation manners

[0040] In order to make the purpose, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be construed as limiting the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0041] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0042] In the following description, the terms "first / second / third" are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first / second / third" can be interchanged with a specific order or sequence when permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0043] In the current camera system, since the spectral response curves of the color three channels (RGB) are different from those of the human eye, it is difficult to accurately restore the colors seen by the human eye, thereby reducing the color performance effect of the image.

[0044] The embodiment of the present application provides an image processing method, apparatus, electronic device, and computer-readable storage medium, which can utilize the assistance of a spectral camera to make the colors of RGB camera images close to the portrait effect and improve the color performance effect of the images. The image processing method provided by the embodiment of the present application is applied to an electronic device, specifically to an electronic device with an image acquisition function, such as an electronic device equipped with a camera. In some embodiments, the electronic device may include user terminals of various types such as smart phones, tablet computers, laptop computers, smart portable devices, vehicle-mounted devices, etc., and the present application does not make specific limitations.

[0045] The image processing method provided by the embodiment of the present application can be as Figure 1 shown, including S101 - S103, as follows:

[0046] S101. Collect a spectral image of the target scene through a spectral camera, and collect a first color image of the target scene through a color camera.

[0047] In the embodiments of the present application, the color camera includes a device that captures color light information for imaging. Exemplarily, the color camera may include an RGB camera that captures three bands of red, green, and blue in the visible light range, decomposes light into corresponding three color channels through a filter, and generates a color image. The spectral camera includes a device that captures and records the spectral information of an object for imaging. Compared with the color camera, the spectral camera can capture a wider spectral range. For example, the spectral camera can capture spectral data including visible light, near-infrared, mid-infrared, and long-wave infrared, etc., so as to provide richer spectral details.

[0048] In some embodiments, at least one color camera may be configured on the electronic device. Through the image processing method of the embodiments of the present application, the spectral camera can be used to assist any one or more color cameras in at least one color camera for image color restoration.

[0049] In the embodiments of the present application, the color camera and the spectral camera perform image acquisition on the same target scene. The spectral camera acquires a spectral image, and the color camera acquires a first color image, such as an RGB image.

[0050] In some embodiments, the difference in the field of view angles between the spectral camera and the color camera is less than the field of view angle difference threshold. That is to say, on the electronic device, the positions of the spectral camera and the color camera can be configured to be as close as possible, with the same or as close as possible field of view angles. Exemplarily, as Figure 2 shown. This can make the images captured by the spectral camera and the color camera closer, thereby further improving the effect of color restoration using the spectral image.

[0051] S102. Convert the spectral image according to the human eye spectral response curve to generate a second color image.

[0052] In the embodiments of the present application, the spectral image contains spectral information corresponding to multiple wavelengths, and the human eye spectral response curve refers to the curve of the sensitivity change of the human eye to light of different wavelengths. The human eye spectral response curve reflects the maximum visual response ability of the human eye to light of different wavelengths. The cone cells in the human eye retina are divided into three types, which are sensitive to short-wave (S), medium-wave (M), and long-wave (L) light respectively, and their spectral response curves correspond to the blue, green, and red light regions respectively. Lights of different wavelengths have different intensity responses on the RGB three channels. Exemplarily, the human eye spectral response curve can be as Figure 3 shown. Figure 3It includes the spectral response curves of three types of cone cells: blue cones sensitive to blue light, green cones sensitive to green light, and red cones sensitive to red light. Among them, the horizontal axis represents the wavelength of light, and the wavelength range on the horizontal axis corresponds to the visible spectrum from purple to red. The vertical axis represents the percentage of the maximum response intensity of each photosensitive cell to light of a specific wavelength, with a maximum value of 100%.

[0053] In the embodiments of the present application, the spectral camera can capture the light intensity at each pixel point at multiple continuous or discrete wavelengths. Compared with a color camera, the spectral camera no longer only records color values (such as red, green, and blue in an RGB image), but records the light intensity values of the pixel at multiple wavelengths. Therefore, for each pixel in the spectral image, the light intensity values at different wavelengths can be plotted as a spectral curve. This curve shows how the light reflected or emitted by the pixel point changes with the wavelength. That is to say, each pixel in the spectral image can correspond to a spectral curve. Exemplarily, the spectral curve of the spectral camera can be as Figure 4 shown.

[0054] Based on the above description, it can be seen that the human eye spectral response curve includes the maximum response intensity of the human eye to light corresponding to at least one color channel (such as RGB) at each wavelength within the visible light band range, while the spectral curve corresponding to each pixel in the spectral image represents the light intensity values at each wavelength in a certain band. Therefore, for the spectral curve corresponding to each pixel in the spectral image, the response intensity of the human eye corresponding to the light intensity values at each wavelength on each color channel of the human eye spectral response curve can be determined according to each wavelength in the band corresponding to the spectral curve. In this way, for each color channel, the total response intensity corresponding to the color channel can be obtained by statistically calculating the response intensity of the human eye corresponding to the light intensity values at each wavelength on the color channel, and this is used as the color value corresponding to the color channel, and then the color value corresponding to each color channel can be obtained. In this way, it is equivalent to converting the spectral curve corresponding to each pixel into at least one color value corresponding to at least one color channel, so that by performing the above processing on each pixel in the spectral image, the representation of the spectral curve of the pixels in the spectral image can be converted into the representation of the color value corresponding to the color channel, thereby converting the spectral image into a color image to obtain a second color image. Exemplarily, if the first color image is an RGB image, the spectral image can be converted into an RGB image according to the human eye spectral response curve.

[0055] In some embodiments, the human eye spectral response curve includes at least one response curve corresponding to at least one color channel; converting the spectral image according to the human eye spectral response curve to generate a second color image includes:

[0056] Determine the spectral curve corresponding to each spectral pixel in the spectral image; for each spectral pixel, perform band weighting and fusion processing on the spectral curve corresponding to the spectral pixel according to each response curve in at least one response curve, and determine the color value corresponding to each color channel in at least one color channel for each spectral pixel; thereby generating a second color image.

[0057] Here, each response curve in at least one response curve corresponds to one color channel in at least one color channel. Exemplarily, at least one response curve may include Figure 3 the response curve of the blue channel corresponding to the blue cone cell, the response curve of the green channel corresponding to the green cone cell, and the response curve of the red channel corresponding to the red cone cell shown in. The response curve of each color channel represents the correspondence between the wavelength of light and the maximum value of the response intensity of the human eye to light on this color channel, while the spectral curve corresponding to each spectral pixel represents the correspondence between the wavelength of light and the intensity value of light. In this way, for a wavelength, the maximum value of the response intensity of the human eye to the light of this wavelength can be used to weight the intensity value of the light of this wavelength, and the weighted results corresponding to the light of each wavelength can be fused on each color channel to obtain the color value corresponding to each spectral pixel on this color channel, thereby obtaining at least one color value corresponding to each spectral pixel in at least one color channel. Taking the at least one color value (such as RGB value) corresponding to at least one color channel as the color value corresponding to the spectral pixel, performing the above processing on each pixel in the spectral image, obtaining the color value corresponding to each pixel in the spectral image, and thereby converting the spectral image into a second color image.

[0058] Exemplarily, the process of performing band weighting and fusion processing on the spectral curve corresponding to each spectral pixel according to each response curve in at least one response curve to determine the color value corresponding to each color channel in at least one color channel for each spectral pixel can be represented by formulas (1)-(3) as follows:

[0059]

[0060] Wherein, SPD[i] is the spectral curve, represented by an array, and i is the wavelength. RET_R[i], RET_G[i], and RET_B[i] are the spectral response curves of the human eye for R, G, and B respectively, also represented by arrays.

[0061] S103. Perform color adjustment on the first color image through the second color image to determine the target image corresponding to the first color image.

[0062] In the embodiments of the present application, since the spectral image collected by the spectral camera can record more spectral information compared to the color camera, and the second color image is converted from the spectral image using the spectral response curve of the human eye, therefore, the color of the pixels in the second color image is closer to the effect observed by the human eye compared to the first color image. By adjusting the color of the first color image with the second color image, the color of the obtained target image can be made closer to the color in the human eye, thereby improving the color performance effect of the image.

[0063] In some embodiments, since the first color image and the second color image are collected for the same target scene, that is to say, the first color image and the second color image contain pixels corresponding to the same real object in the target scene, that is, there are matching pixel pairs between the first color image and the second color image. In this way, through pixel matching, the matching pixel pairs in the first color image and the second color image can be determined. Furthermore, based on the color values corresponding to the matching pixel pairs in the first color image and the second color image respectively, the color mapping relationship between the first color image and the second color image can be determined. In this way, using this color mapping relationship, the color value of each pixel in the first color image is converted, so as to complete the color adjustment of the first color image and obtain the target image.

[0064] It should be noted that, in some embodiments, dense matching can be performed on the first color image and the second color image. Exemplarily, all pixel points in the first color image and the second color image are matched to obtain the matching pixel pairs, and then the color mapping relationship is calculated based on the matching pixel pairs. Or, sparse matching can also be performed on the first color image and the second color image. Representative pixel points with colors are selected from the first color image and the second color image respectively, and the representative pixel points with colors are matched to obtain the matching pixel pairs, and then the color mapping relationship is calculated based on the matching pixel pairs. The specific selection is made according to the actual situation, and the embodiments of the present application do not make any limitations.

[0065] It can be understood that the embodiments of the present application can be based on the image collection of the same target scene by the spectral camera and the color camera, convert the spectral image captured by the spectral camera into the second color image using the spectral response curve of the human eye, and use the second color image to adjust the color of the first color image, so that the obtained target image is closer to the color effect observed by the human eye, thereby being able to better restore the color of the scene captured by the color camera as seen by the human eye and improving the color performance effect of the image.

[0066] In some embodiments, the process of adjusting the color of the first color image with the second color image in S103 above to determine the target image corresponding to the first color image can be as Figure 5As shown, it includes:

[0067] S201. Determine the color mapping relationship between the first color image and the second color image by performing pixel matching on the first color image and the second color image.

[0068] S202. Adjust the color of the first color image according to the color mapping relationship to determine the target image.

[0069] Here, as described in the foregoing embodiments, the first color image and the second color image can be pixel-matched by dense matching or sparse matching to obtain matching pixel pairs, and then the corresponding relationship of color values can be calculated based on the matching pixel pairs to determine the color mapping relationship between the first color image and the second color image. Below, an example will be given in the way of separately selecting color-representative pixel points from the first color image and the second color image, performing sparse matching on the color-representative pixel points to obtain matching pixel pairs, and then obtaining the color mapping relationship.

[0070] In some embodiments, the process of determining the color mapping relationship between the first color image and the second color image by performing pixel matching on the first color image and the second color image can be as Figure 6 shown, including:

[0071] S301. Perform clustering on the pixel colors of the first color image and the second color image respectively to determine at least one first color clustering center corresponding to the first color image and at least one second color clustering center corresponding to the second color image.

[0072] In some embodiments, the first color image is clustered by calculating the color similarity of the pixels in the first color image on at least one color channel to determine at least one first color clustering center; the second color image is clustered by calculating the color similarity of the pixels in the second color image on at least one color channel to determine at least one second color clustering center.

[0073] Here, for the first color image, according to the color value of each pixel therein, such as the RGB value, clustering calculation is performed on the color values of all pixels to determine at least one first color clustering center corresponding to the first color image. At least one first color clustering center represents at least one representative color included in the first color image.

[0074] For the second color image, according to the color value of each pixel therein, such as the RGB value, clustering calculation is performed on the color values of all pixels to determine at least one second color clustering center corresponding to the second color image. At least one second color clustering center represents at least one representative color included in the second color image.

[0075] Exemplarily, the result effect diagram obtained by clustering the pixel colors of the first color image or the second color image may be as Figure 7 shown. It should be noted that for the case where the color value is an RGB value, clustering needs to be performed in three dimensions corresponding to the three color channels of RGB, and pixels with similar color values are clustered into one category. Figure 7 The clustering effect of the three dimensions is shown in the form of a two-dimensional image, where the horizontal axis and the vertical axis correspond to the number of pixels. As Figure 7 shown, through clustering, similar colors can be clustered into one category, so as to obtain the clustering centers corresponding to various colors, such as at least one first color clustering center or at least one second color clustering center. The clustering algorithm may include the K-Means algorithm or other clustering algorithms, etc., which are not specifically limited here. In this way, a large number of color values corresponding to a large number of pixels in the first color image can be represented by a relatively sparse at least one first color clustering center, and a large number of color values corresponding to a large number of pixels in the second color image can be represented by a relatively sparse at least one second color clustering center.

[0076] S302. Based on at least one first color clustering center and at least one second color clustering center, perform color matching on the pixels in the first color image and the second color image to determine at least one pair of matching pixels.

[0077] Among them, the pixels corresponding to at least one first color clustering center can be matched with the pixels corresponding to at least one second color clustering center, so as to obtain different color values corresponding to the pixels representing the same color in the first color image and the second color image respectively.

[0078] In some embodiments, based on the color similarity between the pixels in the first color image and the first color clustering center, determine at least one first target pixel corresponding to at least one first color clustering center; according to the color similarity between the pixels in the second color image and the second color clustering center, determine at least one second target pixel corresponding to at least one second color clustering center; perform color matching on at least one first target pixel and at least one second target pixel to determine at least one pair of matching pixels.

[0079] That is to say, the color similarity between each pixel in the first color image and each first color clustering center among at least one first color clustering center can be calculated. Thus, for each first color clustering center, a pixel in the first color image with the highest color similarity to it can be determined as a first target pixel corresponding to this first color clustering center, so as to obtain at least one first target pixel corresponding to at least one first color clustering center. Similarly, the color similarity between each pixel in the second color image and each second color clustering center among at least one second color clustering center can be calculated. Thus, for each second color clustering center, a pixel in the second color image with the highest color similarity to it can be determined as a second target pixel corresponding to this second color clustering center, so as to obtain at least one second target pixel corresponding to at least one first color clustering center.

[0080] Exemplarily, the color similarity can be represented by the Euclidean distance of color values. The smaller the Euclidean distance, the higher the color similarity. The Euclidean distance between a pixel and a color clustering center can be obtained through the process of the clustering algorithm. For each first color clustering center among at least one first color clustering center, a pixel in the first color image with the smallest Euclidean distance from this first color clustering center, which is also the pixel in the first color image closest to this first color clustering center, can be determined as the first target pixel corresponding to this first color clustering center. For each second color clustering center among at least one second color clustering center, a pixel in the second color image with the smallest Euclidean distance from this second color clustering center, which is also the pixel in the second color image closest to this second color clustering center, can be determined as the second target pixel corresponding to this second color clustering center.

[0081] In some embodiments, the Euclidean distance between a pixel and a color clustering center can be obtained by calculating the Euclidean distance of the pixel and the color clustering center on each color channel among at least one color channel. Exemplarily, for the case where the first color image and the second color image are RGB images, the distance functions in the three RGB dimensions can be used to calculate the Euclidean distance between pixels, so as to perform clustering in three dimensions to obtain at least one first color clustering center and at least one second color clustering center. Then, according to the Euclidean distance between the pixel and the color clustering center calculated by the distance functions in the three dimensions, the first target pixel closest to each first color clustering center among at least one first color clustering center can be determined to obtain at least one first target pixel; and the second target pixel closest to each second color clustering center among at least one second color clustering center can be determined to obtain at least one second target pixel. In this way, it is equivalent to selecting the pixel closest to the color clustering center as the representative pixel of this color.

[0082] In some embodiments, after determining at least one first target pixel and at least one second target pixel, color similarity matching may be performed on the at least one first target pixel and the at least one second target pixel to determine at least one pair of matching first target pixels and second target pixels as at least one pair of matching pixels, which is equivalent to at least one matching pixel pair.

[0083] In some embodiments, color similarity and the image coordinates between pixels may also be combined to perform pixel matching on the at least one first target pixel and the at least one second target pixel. Exemplarily, the process of performing color matching on the at least one first target pixel and the at least one second target pixel to determine at least one pair of matching pixels includes:

[0084] For each first target pixel among the at least one first target pixel, based on a color similarity matching threshold, at least one initial matching pixel corresponding to each first target pixel is determined from the at least one second target pixel; according to the coordinates of the initial matching pixels and the coordinates of each first target pixel, a matching pixel corresponding to each first target pixel is determined from the at least one initial matching pixel, and each first target pixel and its corresponding matching pixel are used as a pair of matching pixels, thereby determining at least one pair of matching pixels.

[0085] That is to say, a first target pixel may be selected, and based on the color similarity between each second target pixel in the at least one second target pixel and the first target pixel, as well as the color similarity matching threshold, the at least one second target pixel is preliminarily screened, and at least one initial matching pixel with a color similarity greater than or equal to the color similarity matching threshold is determined from the at least one second target pixel. If the number of initial matching pixels is only one, the initial matching pixel is determined as the matching pixel corresponding to the first target pixel, and the initial matching pixel and the first target pixel are used as a pair of matching pixels (a matching pixel pair). If the number of initial matching pixels is greater than one, according to the coordinates of the initial matching pixels and the coordinates of the first target pixel, the initial matching pixel closest to the coordinates of the first target pixel is determined from the at least one initial matching pixel as the matching pixel corresponding to the first target pixel, so that the first target pixel and the initial matching pixel closest to the coordinates of the first target pixel are used as a pair of matching pixels. In this way, at least one pair of matching pixels can be determined.

[0086] In some embodiments, before determining, according to the coordinates of the initial matching pixels and the coordinates of each first target pixel, a matching pixel corresponding to each first target pixel from the at least one initial matching pixel, the method further includes:

[0087] Based on the first resolution of the color camera and the second resolution of the spectral camera, coordinate transformation is performed on the original coordinates of each second target pixel to determine the coordinates of each second target pixel.

[0088] Here, since the resolution of the spectral camera is different from that of the color camera, usually, the resolution of the spectral camera is lower. Therefore, before using the image coordinates to determine the matching pixels corresponding to the second target pixels, coordinate transformation needs to be performed on the second target pixels corresponding to the spectral camera. Exemplarily, based on the first resolution of the color camera and the second resolution of the spectral camera, coordinate transformation is performed on the original coordinates of each second target pixel to determine the coordinates of each second target pixel can be achieved through Formula (4) and Formula (5), as follows:

[0089] X_new = X_old * (Res_X1 / Res_X2) Formula (4)

[0090] Y_new = Y_old * (Res_Y1 / Res_Y2) Formula (5)

[0091] Wherein, X_new and Y_new are the transformed coordinates, X_old and Y_old are the coordinates before transformation, Res_X1 and Res_Y1 are the resolutions of the RGB camera, and Res_X2 and Res_Y2 are the resolutions of the spectral camera.

[0092] S303. Determine the color mapping relationship between the first color image and the second color image according to at least one pair of matching pixels.

[0093] In some embodiments, a color transformation matrix can be calculated based on the corresponding relationship between each pair of matching pixels in at least one pair of matching pixels as the color mapping relationship.

[0094] It should be noted that the color mapping relationship can be represented not only by a color transformation matrix. In some embodiments, data in the form of a table, curve, etc. that can characterize the corresponding relationship between pixel colors can also be determined based on the corresponding relationship between each pair of matching pixels in at least one pair of matching pixels as the color mapping relationship between the first color image and the second color image, which is not specifically limited in the embodiments of the present application.

[0095] It can be understood that by clustering, pixel points with color representativeness are respectively selected from the first color image and the second color image for sparse matching to obtain matching pixel pairs, and then the color mapping relationship is calculated to adjust the first color image, which can reduce the time resources and hardware resources consumed in the pixel matching process and improve the image processing efficiency.

[0096] In some embodiments, before clustering the pixel colors of the first color image and the second color image respectively to determine at least one first color clustering center corresponding to the first color image and at least one second color clustering center corresponding to the second color image, image enhancement may also be performed on the first color image and the second color image. Exemplarily, image enhancement is performed by increasing the contrast between the first color image and the second color image. For example, histogram equalization can be used to process the RGB three channels of the first color image and the second color image. Figure 8 The figure shows the comparison before and after histogram equalization taking a single-channel grayscale image as an example. It can be seen that through image enhancement, the color contrast can be enhanced, which is more conducive to the subsequent processes of pixel color clustering and pixel matching.

[0097] It can be understood that by performing pixel color clustering and pixel matching after image enhancement of the first color image and the second color image, the matching rate of pixels in the first color image and the second color image can be improved, the number of at least one pair of matching pixels can be increased, and thus the accuracy of calculating the color mapping relationship based on at least one pair of matching pixels can be improved.

[0098] In some embodiments, the process of calculating the color transformation matrix as the color mapping relationship based on the corresponding relationship between each pair of matching pixels in at least one pair of matching pixels includes:

[0099] Determine the second original color value corresponding to the second target pixel in each pair of matching pixels, and the first original color value corresponding to the matching pixel corresponding to the second target pixel; the second original color value and the first original color value are color values in the color image before image enhancement; calculate the color transformation matrix according to the corresponding relationship between the first original color value and the second original color value corresponding to each pair of matching pixels in at least one pair of matching pixels.

[0100] Here, for each pair of matching pixels, the color value of the second target pixel in the second color image before image enhancement can be extracted as the second original color value corresponding to the second target pixel, and the color value of the matching pixel corresponding to the second target pixel in the first color image before image enhancement can be extracted as the first original color value. In this way, each pair of matching pixels corresponds to a first original color value and a second original color value. Furthermore, the color transformation matrix can be calculated according to the corresponding relationship between the first original color value and the second original color value corresponding to each pair of matching pixels in at least one pair of matching pixels.

[0101] Exemplarily, the color transformation matrix can be calculated using the least squares method according to the corresponding relationship between the first original color value and the second original color value corresponding to each pair of matching pixels in at least one pair of matching pixels.

[0102] Thus, based on the calculated color transformation matrix, the process of color-adjusting the first color image to determine the target image can be shown as in formula (6) as follows:

[0103]

[0104] Wherein, is the original RGB color value of the pixel in the first color image, [CM] is the color transformation matrix, is the RGB color value obtained after adjusting the original RGB color value by using the color transformation matrix. For each pixel in the first color image, color adjustment is performed using formula (6), and the target image corresponding to the first color image can be obtained.

[0105] It can be understood that by calculating the color mapping relationship through the matching pixel pairs and applying it to the first color image, it is possible to use the pixel color values in the second color image that are closer to the human eye observation effect to adjust the first color image, thereby improving the pixel color performance effect of the adjusted target image.

[0106] In some embodiments, the embodiments of the present application provide an image processing method applied to an actual scenario, as Figure 9 shown, including: for the high-resolution RGB camera image (first color image) collected by a high-resolution RGB camera (color camera), histogram equalization is performed on each of its RGB three channels, and then based on the high-resolution RGB camera image after image enhancement, clustering is performed with the RGB values of each pixel as elements, and for each cluster, the pixel closest to the cluster center (first target pixel) is selected, and its image coordinates and original RGB value (the original color value before histogram equalization, equivalent to the first original color value) are extracted.

[0107] For the low-resolution spectral camera image (spectral image) collected by a low-resolution spectral camera (spectral camera), the spectral curve of each pixel is multiplied by the human eye spectral response curve to generate a low-resolution RGB image (second color image), histogram equalization is performed on each of the RGB three channels in the low-resolution RGB image, and then based on the low-resolution RGB image after image enhancement, clustering is performed with the RGB values of each pixel as elements, and for each cluster, the pixel closest to the cluster center (second target pixel) is selected, and its image coordinates and original RGB value (the original color value before histogram equalization, equivalent to the second original color value) are extracted.

[0108] In this way, alternative pixels of two images (equivalent to at least one first target pixel and at least one second target pixel) can be pixel-matched. Before the matching, since the field of view angles of the spectral camera and the RGB camera are similar, but the resolutions are different, it is necessary to first convert the pixel coordinates of the low-resolution RGB image. And a color distance threshold th1 (equivalent to the color similarity matching threshold) is set. When matching, the color values of the alternative pixels of the two images use the color values after histogram equalization, and the pixel with the closest RGB value in the spectral image (the pixel with the smallest three-dimensional Euclidean distance in RGB) is matched for each alternative pixel in the high-resolution RGB image. If a spectral image pixel is matched by multiple RGB camera pixels, only the match with the closest image coordinates is retained.

[0109] In this way, according to the matched pixel pairs, a series of matching relationships from the RGB values of the RGB camera to the RGB values of the spectral camera image (the original pixel values before histogram equalization) can be obtained. The least squares method is used to calculate a 3×3 color transformation matrix to fit the color mapping relationship between the corresponding low-resolution RGB image and the high-resolution RGB image of the spectral camera. Finally, the 3×3 color transformation matrix is applied to each pixel in the high-resolution RGB image to calculate the final target image.

[0110] It can be understood that with the assistance of a low-resolution spectral camera, the color of the RGB camera image can be made closer to the effect observed by the human eye, thereby improving the performance of the image color.

[0111] An embodiment of the present application provides an image processing device 1, as Figure 10 shown, including:

[0112] An acquisition module 11, configured to acquire a spectral image of a target scene through a spectral camera, and acquire a first color image of the target scene through a color camera;

[0113] A generation module 12, configured to convert the spectral image according to the human eye spectral response curve to generate a second color image;

[0114] An adjustment module 13, configured to perform color adjustment on the first color image through the second color image to determine a target image corresponding to the first color image.

[0115] In some embodiments, the adjustment module 13 is further configured to determine a color mapping relationship between the first color image and the second color image by performing pixel matching on the first color image and the second color image; and perform color adjustment on the first color image according to the color mapping relationship to determine the target image.

[0116] In some embodiments, the adjustment module 13 is further configured to perform pixel color clustering on the first color image and the second color image respectively, determine at least one first color clustering center corresponding to the first color image, and at least one second color clustering center corresponding to the second color image; based on the at least one first color clustering center and the at least one second color clustering center, perform color matching on the pixels in the first color image and the second color image to determine at least one pair of matching pixels; and determine the color mapping relationship between the first color image and the second color image according to the at least one pair of matching pixels.

[0117] In some embodiments, the adjustment module 13 is further configured to determine at least one first target pixel corresponding to the at least one first color clustering center based on the color similarity between the pixels in the first color image and the first color clustering center; determine at least one second target pixel corresponding to the at least one second color clustering center according to the color similarity between the pixels in the second color image and the second color clustering center; and perform color matching on the at least one first target pixel and the at least one second target pixel to determine the at least one pair of matching pixels.

[0118] In some embodiments, for each second target pixel among the at least one second target pixel, the adjustment module 13 is further configured to determine at least one initial matching pixel corresponding to each second target pixel from the at least one first target pixel based on a color similarity matching threshold; determine the matching pixel corresponding to each second target pixel from the at least one initial matching pixel according to the coordinates of the initial matching pixel and the coordinates of each second target pixel, and use each second target pixel and its corresponding matching pixel as a pair of matching pixels, thereby determining the at least one pair of matching pixels.

[0119] In some embodiments, before determining the matching pixel corresponding to each second target pixel from the at least one initial matching pixel according to the coordinates of the initial matching pixel and the coordinates of each second target pixel, the adjustment module 13 is further configured to perform coordinate transformation on the original coordinates of each second target pixel based on the first resolution of the color camera and the second resolution of the spectral camera to determine the coordinates of each second target pixel.

[0120] In some embodiments, the adjustment module 13 is further configured to calculate a color transformation matrix based on the color correspondence relationship between each pair of matching pixels in the at least one pair of matching pixels as the color mapping relationship.

[0121] In some embodiments, before the adjustment module 13 respectively performs pixel color clustering on the first color image and the second color image to determine at least one first color clustering center corresponding to the first color image and at least one second color clustering center corresponding to the second color image, the adjustment module 13 further performs image enhancement on the first color image and the second color image; and determines a second original color value corresponding to a second target pixel in each pair of matching pixels and a first original color value corresponding to the matching pixel corresponding to the second target pixel; the second original color value and the first original color value are color values in the color image before image enhancement; and calculates the color transformation matrix according to the color correspondence relationship between the first original color value and the second original color value corresponding to each pair of matching pixels in the at least one pair of matching pixels.

[0122] In some embodiments, the first color image and the second color image include at least one color channel, and the adjustment module 13 further performs clustering on the first color image by calculating the color similarity of the pixels in the first color image on the at least one color channel to determine the at least one first color clustering center; and performs clustering on the second color image by calculating the color similarity of the pixels in the second color image on the at least one color channel to determine the at least one second color clustering center.

[0123] In some embodiments, the human eye spectral response curve includes at least one response curve corresponding to at least one color channel; the generation module 12 further determines a spectral curve corresponding to each spectral pixel in the spectral image; for each spectral pixel, performs band weighting and fusion processing on the spectral curve corresponding to the spectral pixel according to each response curve in the at least one response curve to determine the color value corresponding to each spectral pixel in each color channel of the at least one color channel; and thus generates the second color image.

[0124] In some embodiments, the difference in the field of view angles of the spectral camera and the color camera is less than the field of view angle difference threshold.

[0125] It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments and has similar beneficial effects to those of the method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0126] The embodiments of the present application further provide an electronic device. Figure 11 It is an optional structural schematic diagram of the electronic device provided by the embodiments of the present application. As Figure 11As shown in the figure, the electronic device 3 includes: a memory 32 and a processor 33. Among them, the memory 32 and the processor 33 are connected through a communication bus 34; the memory 32 is used to store executable instructions; the processor 33 is used to implement the image processing method provided by the embodiments of the present application when executing the executable instructions stored in the memory 32.

[0127] The embodiments of the present application provide a computer-readable storage medium storing executable instructions, where the executable instructions are stored, and when the executable instructions are executed by the above-mentioned processor, the above-mentioned processor will be caused to execute the image processing method provided by the embodiments of the present application.

[0128] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; it may also be various devices including one or any combination of the above-mentioned memories.

[0129] In some embodiments, the executable instructions may be in the form of a program, software, software module, script, or code, and may be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as an independent program or being deployed as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0130] As an example, the executable instructions may or may not correspond to files in the file system, and may be stored as part of a file that stores other programs or data. For example, they may be stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program being discussed, or stored in multiple cooperating files (for example, files that store one or more modules, subroutines, or code portions). As an example, the executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one location, or on multiple computing devices distributed at multiple locations and interconnected through a communication network.

[0131] Those skilled in the art should understand that the embodiments of the present application may be provided as a method, a system, or a computer program product. Therefore, the present application may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory and optical memory, etc.) containing computer-usable program code.

[0132] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to generate a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0133] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0135] As mentioned above, it is only a preferred embodiment of the present application and is not used to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that, Including: Collecting a spectral image of a target scene by a spectral camera, and collecting a first color image of the target scene by a first camera; Converting the spectral image according to the human eye spectral response curve to generate a second color image; Adjusting the color of the first color image through the second color image to determine a target image corresponding to the first color image.

2. The method according to claim 1, wherein The human eye spectral response curve includes at least one response curve corresponding to at least one color channel; the converting the spectral image according to the human eye spectral response curve to generate a second color image includes: Determining a spectral curve corresponding to each spectral pixel in the spectral image; For each spectral pixel, performing band weighting and fusion processing on the spectral curve corresponding to the spectral pixel according to each response curve in the at least one response curve to determine a color value corresponding to each color channel in the at least one color channel for each spectral pixel; thereby generating the second color image.

3. The method according to claim 1 or 2, characterized in that, The adjusting the color of the first color image through the second color image to determine a target image corresponding to the first color image includes: Determining a color mapping relationship between the first color image and the second color image by performing pixel matching on the first color image and the second color image; Adjusting the color of the first color image according to the color mapping relationship to determine the target image.

4. The method according to claim 3, wherein The determining a color mapping relationship between the first color image and the second color image by performing pixel matching on the first color image and the second color image includes: Performing pixel color clustering on the first color image and the second color image respectively to determine at least one first color clustering center corresponding to the first color image and at least one second color clustering center corresponding to the second color image; Based on the at least one first color clustering center and the at least one second color clustering center, performing color matching on the first color image and the second color image to determine at least one pair of matching pixels; Determining a color mapping relationship between the first color image and the second color image according to the at least one pair of matching pixels.

5. The method according to claim 4, characterized in that, The performing color matching on the first color image and the second color image based on the at least one first color clustering center and the at least one second color clustering center to determine at least one pair of matching pixels includes: Determining at least one first target pixel corresponding to the at least one first color clustering center based on the color similarity between the pixels in the first color image and the first color clustering center; Determining at least one second target pixel corresponding to the at least one second color clustering center according to the color similarity between the pixels in the second color image and the second color clustering center; Performing color matching on the at least one first target pixel and the at least one second target pixel to determine the at least one pair of matching pixels.

6. The method according to claim 5, wherein The performing color matching on the at least one first target pixel and the at least one second target pixel to determine the at least one pair of matching pixels includes: For each of the at least one first target pixel, based on a color similarity matching threshold, determine at least one initial matching pixel corresponding to each of the first target pixels from the at least one second target pixel; According to the coordinates of the initial matching pixels and the coordinates of each of the first target pixels, determine the matching pixel corresponding to each of the first target pixels from the at least one initial matching pixel, and use each of the first target pixels and its corresponding matching pixel as a pair of matching pixels, thereby determining the at least one pair of matching pixels.

7. The method according to claim 6, characterized in that, Before the step of determining the matching pixel corresponding to each of the first target pixels from the at least one initial matching pixel according to the coordinates of the initial matching pixels and the coordinates of each of the first target pixels, the method further includes: Based on the first resolution of the color camera and the second resolution of the spectral camera, perform coordinate transformation on the original coordinates of each of the second target pixels to determine the coordinates of each of the second target pixels.

8. The method according to any one of claims 4 to 7, characterized in that The step of determining the color mapping relationship between the first color image and the second color image according to the at least one pair of matching pixels includes: Based on the color correspondence relationship between each pair of matching pixels in the at least one pair of matching pixels, calculate a color transformation matrix as the color mapping relationship.

9. The method according to claim 8, wherein Before the step of respectively clustering the pixel colors of the first color image and the second color image to determine at least one first color clustering center corresponding to the first color image and at least one second color clustering center corresponding to the second color image, the method further includes: Perform image enhancement on the first color image and the second color image; The step of calculating a color transformation matrix as the color mapping relationship based on the color correspondence relationship between each pair of matching pixels in the at least one pair of matching pixels includes: Determine the second original color value corresponding to the second target pixel in each pair of matching pixels, and the first original color value corresponding to the matching pixel corresponding to the second target pixel; the second original color value and the first original color value are color values in the color image before image enhancement; Calculate the color transformation matrix according to the color correspondence relationship between the first original color value and the second original color value corresponding to each pair of matching pixels in the at least one pair of matching pixels.

10. The method according to any one of claims 4-7, characterized in that, The first color image and the second color image include at least one color channel. The step of respectively clustering the pixel colors of the first color image and the second color image to determine at least one first color clustering center corresponding to the first color image and at least one second color clustering center corresponding to the second color image includes: Cluster the first color image by calculating the color similarity of the pixels in the first color image on the at least one color channel to determine the at least one first color clustering center; Cluster the second color image by calculating the color similarity of the pixels in the second color image on the at least one color channel to determine the at least one second color clustering center.

11. The method according to any one of claims 1, 2, 4 - 7, or claim 9, characterized in that, The difference in the field of view angle between the spectral camera and the color camera is less than the field of view angle difference threshold.

12. An image processing apparatus, characterized in that, It includes: An acquisition module, configured to acquire a spectral image of a target scene through a spectral camera, and acquire a first color image of the target scene through a color camera; A generation module, configured to convert the spectral image according to the human eye spectral response curve to generate a second color image; An adjustment module, configured to use the second color image to perform color adjustment on the first color image to determine a target image corresponding to the first color image.

13. An electronic device, characterized in that, It includes a memory and a processor; wherein, The memory is configured to store executable instructions; The processor is configured to implement the method according to any one of claims 1-11 when executing the executable instructions stored in the memory.

14. A computer-readable storage medium, characterized in that, Stored with executable instructions, which are used to cause the processor to implement the method according to any one of claims 1-11 when executed.

15. A computer program product, characterized in that, It includes a computer program or instruction, and when the computer program or instruction is executed by the processor, the method according to any one of claims 1-11 is implemented.

Images