Image processing method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to an image processing method and apparatus, an electronic device, a storage medium and a program product. The method comprises the steps of obtaining a plurality of partition image data and a spectrum type of a spectrum curve corresponding to each partition image data; for the partition image data corresponding to the spectrum curve of which the spectrum type is a narrow-band spectrum, determining a target gain of each color channel corresponding to the partition image data based on the partition image data, the spectrum curve corresponding to the partition image data and the human eye spectrum response curve corresponding to each color channel; adjusting the partitioned image data based on the target gain of each color channel corresponding to the partitioned image data to obtain adjusted image data corresponding to the partitioned image data; and obtaining target image data based on the multiple pieces of adjusted image data. By adopting the method, the image quality can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technologies, and in particular, to an image processing method, apparatus, electronic device, storage medium, and computer program product. Background Art

[0002] With the development of image processing technologies, in order to avoid color deviation of images caused by different light source color temperatures, the white balance processing (AWB, Auto White Balance) method is usually used to correct the colors of images so that the images present natural colors.

[0003] In traditional technologies, the colors of images are corrected based on statistical values of image data, resulting in inaccurate colors in local regions of the images, thereby reducing the quality of the images. Summary of the Invention

[0004] Embodiments of this application provide an image processing method, apparatus, electronic device, storage medium, and program product, which can improve the quality of images.

[0005] In a first aspect, this application provides an image processing method, including:

[0006] Obtaining a plurality of partition image data and the spectral types of spectral curves corresponding to each of the partition image data;

[0007] For the partition image data corresponding to the spectral curve with a narrowband spectrum type, determining the target gains of each color channel corresponding to the partition image data based on the partition image data, the spectral curve corresponding to the partition image data, and the human eye spectral response curves corresponding to each color channel;

[0008] Adjusting the partition image data based on the target gains of each color channel corresponding to the partition image data to obtain adjusted image data corresponding to the partition image data;

[0009] Obtaining target image data based on a plurality of the adjusted image data.

[0010] In a second aspect, this application also provides an imaging module, including:

[0011] An imaging lens;

[0012] A spectral sensor configured to obtain spectral information collected by each sub-region in the spectral sensor;

[0013] An image sensor configured to obtain initial image data;

[0014] A processor, configured to determine a plurality of partitioned image data according to the initial image data, and configured to implement the steps of the method according to any one of the first aspect.

[0015] In a third aspect, the present application further provides an image processing apparatus, including:

[0016] An acquisition module, configured to acquire a plurality of partitioned image data and spectral types of spectral curves corresponding to each of the partitioned image data;

[0017] A determination module, configured to, for the partitioned image data corresponding to a spectral curve with a narrowband spectrum type, determine target gains of each color channel corresponding to the partitioned image data based on the partitioned image data, the spectral curve corresponding to the partitioned image data, and the spectral response curves of the human eye corresponding to each color channel;

[0018] An adjustment module, configured to adjust the partitioned image data based on the target gains of each color channel corresponding to the partitioned image data to obtain adjusted image data corresponding to the partitioned image data;

[0019] A processing module, configured to obtain target image data based on a plurality of the adjusted image data.

[0020] In a fourth aspect, the present application further provides an electronic device, including a memory and a processor, where the memory stores a computer program, and the processor, when executing the computer program, implements the steps of the method according to any one of the first aspect.

[0021] In a fifth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps of the method according to any one of the first aspect.

[0022] In a sixth aspect, the present application further provides a computer program product, including a computer program, and the computer program, when executed by a processor, implements the steps of the method according to any one of the first aspect.

[0023] In the above image processing method, apparatus, electronic device, readable storage medium, and computer program product, the spectral response curve of the image sensor in the electronic device to the narrowband light source is different from the human eye spectral response curve. As a result, in the partitioned image data corresponding to the narrowband spectrum, there is a deviation between the color corresponding to the pixel point and the actual color. By using the partitioned image data, the spectral curve corresponding to the partitioned image data, and the human eye spectral response curves corresponding to each color channel, the target gain of each color channel corresponding to the partitioned image data is determined. Based on the target gain of each color channel corresponding to the partitioned image data, the partitioned image data is adjusted to obtain the adjusted image data corresponding to the partitioned image data, thereby realizing the adjustment of the partitioned image data corresponding to the narrowband spectrum, making the color corresponding to the pixel point in the adjusted image data close to or equal to the actual color. Based on multiple adjusted image data, the target image data is obtained, reducing the deviation between the color corresponding to the pixel point in the target image data and the actual color, thereby improving the quality of the image corresponding to the target image data. Description of the Drawings

[0024] To more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments of the present application or related technologies. Obviously, the following described drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0025] Figure 1 It is a schematic flowchart of an image processing method in an embodiment;

[0026] Figure 2 It is a schematic diagram of a continuous spectrum in an embodiment;

[0027] Figure 3 It is a schematic diagram of a narrowband spectrum in an embodiment;

[0028] Figure 4 It is a schematic diagram of a human eye spectral response curve in an embodiment;

[0029] Figure 5 It is a schematic flowchart of the initial gain determination step in an embodiment;

[0030] Figure 6 It is a schematic diagram of the white balance image corresponding to the white balance image data in an embodiment;

[0031] Figure 7 It is a schematic diagram of an image sensor and a spectral sensor in an embodiment;

[0032] Figure 8 It is a schematic flowchart of an image processing method in an embodiment;

[0033] Figure 9 is a structural block diagram of an image processing device in an embodiment;

[0034] Figure 10 is an internal structure diagram of an electronic device in an embodiment. Detailed implementation manners

[0035] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0036] In an exemplary embodiment, as Figure 1 shown, an image processing method is provided. Taking the application of this method to an electronic device as an example for illustration, the electronic device can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, smart cars, etc., and the portable wearable devices can be smart watches and smart bracelets, etc. It can be understood that this method can also be applied to a system including an electronic device and a server and realized through the interaction between the electronic device and the server. In this embodiment, the method includes steps 102 to 108, where:

[0037] Step 102, obtain a plurality of partition image data and the spectral types of the spectral curves corresponding to each partition image data.

[0038] Among them, the partition image data refers to a data set representing a local image, and the partition image data includes the position information and pixel values of a plurality of pixel points. The position information refers to the information representing the position of a pixel point in the image, and the position information can be represented by two-dimensional coordinates. The pixel value refers to the numerical value representing the brightness of a pixel point. The spectral curve refers to a curve describing the energy distribution of light at different wavelengths. The spectral curve usually has the wavelength (Wavelength) as the horizontal axis and the reflectance of light as the vertical axis. The spectral type refers to the type of spectrum, and the spectral type is one of a narrowband spectrum and a continuous spectrum. The continuous spectrum refers to a spectrum evenly distributed within a wavelength range. For example, the schematic diagram of the continuous spectrum is as Figure 2 shown. As Figure 2 shown, if the reflectance changes evenly corresponding to different wavelengths in the two spectral curves shown, both of the two spectral curves are continuous spectra. The narrowband spectrum refers to a spectrum unevenly distributed within a wavelength range. It can be understood that the reflectance corresponding to some wavelengths is larger, the reflectance corresponding to some wavelengths is smaller, and the difference between the reflectances corresponding to different wavelengths is very large. For example, the schematic diagram of the narrowband spectrum is as Figure 3 shown. If Figure 3In the two spectral curves shown, the reflectance corresponding to some wavelengths is particularly large, and the reflectance corresponding to some wavelengths is close to zero. Then both spectral curves are narrowband spectra.

[0039] Exemplarily, the electronic device acquires initial image data through an image sensor, performs white balance processing on the initial image data to obtain white balance image data, divides the white balance image data to obtain multiple partition image data, acquires the spectral curves respectively corresponding to each partition image data, and for each partition image data, inputs the spectral curve corresponding to the partition image data into the target model to obtain the spectral type of the spectral curve corresponding to the partition image data.

[0040] In an exemplary embodiment, after obtaining the spectral type of the spectral curve corresponding to the partition image data, it further includes: determining whether there is a spectral curve with a spectral type of narrowband spectrum. If there is a spectral curve with a spectral type of narrowband spectrum, then step 104 is executed; if there is no spectral curve with a spectral type of narrowband spectrum, then the white balance image data is determined as the target image data.

[0041] Step 104, for the partition image data corresponding to the spectral curve with a spectral type of narrowband spectrum, based on the partition image data, the spectral curve corresponding to the partition image data, and the human eye spectral response curves corresponding to each color channel, determine the target gains of each color channel corresponding to the partition image data.

[0042] Wherein, a color channel refers to the basic component of a color. It can be understood that the color information is decomposed into different basic components, and each color channel represents a component of the color. For example, the RGB color model includes an R (Red) channel, a G (Green) channel, and a B (Blue) channel. The human eye spectral response curve describes the sensitivity of the human eye to light of different wavelengths. The human eye spectral response curve usually has the wavelength of light as the horizontal axis and the retinal response intensity as the vertical axis. The human eye spectral response curves corresponding to different color channels are different. For example, a schematic diagram of the human eye spectral response curve is as Figure 4As shown, the horizontal axis is the wavelength of light, and the vertical axis is the retinal response. 402 is the spectral response curve of the human eye corresponding to the blue channel, which represents the response intensity of the blue cones in the human eye to light of different wavelengths. The blue cones are the photoreceptor cells in the human eye that are used to perceive light in the blue light band; 404 is the spectral response curve of the human eye corresponding to the green channel, which represents the response intensity of the green cones in the human eye to light of different wavelengths. The green cones are the photoreceptor cells in the human eye that are used to perceive light in the green light band; 406 is the spectral response curve of the human eye corresponding to the red channel, which represents the response intensity of the red cones in the human eye to light of different wavelengths. The red cones are the photoreceptor cells in the human eye that are used to perceive light in the blue light band. The target gain refers to the coefficient used to adjust the pixel value.

[0043] Exemplarily, for the partitioned image data corresponding to the spectral curve of the narrowband spectrum type, the electronic device determines the target gain of each color channel corresponding to the partitioned image data based on the partitioned image data, the spectral curve corresponding to the partitioned image data, and the spectral response curves of the human eye corresponding to each color channel.

[0044] In an exemplary embodiment, for the partitioned image data corresponding to the spectral curve of the narrowband spectrum type, based on the partitioned image data, the spectral curve corresponding to the partitioned image data, and the spectral response curves of the human eye corresponding to each color channel, the initial gain of each color channel corresponding to the partitioned image data is determined; the initial gain of each color channel corresponding to the partitioned image data is smoothed to obtain the target gain of each color channel corresponding to the partitioned image data. That is, local smoothing processing avoids the target gain of each color channel corresponding to the partitioned image data being too large or too small. The methods for smoothing processing include, but are not limited to, one of mean filtering and Gaussian rate filtering, etc.

[0045] In an exemplary embodiment, for the partitioned image data corresponding to the spectral curve of the narrowband spectrum type, based on the partitioned image data, the spectral curve corresponding to the partitioned image data, and the spectral response curves of the human eye corresponding to each color channel, the initial gain of each color channel corresponding to the partitioned image data is determined; for the partitioned image data corresponding to the spectral curve of the continuous spectrum type, the preset gain is determined as the initial gain of each color channel corresponding to the partitioned image data; the initial gain of each color channel corresponding to each partitioned image data is smoothed to obtain the target gain of each color channel corresponding to each partitioned image data. The preset gain refers to the gain set in advance. For example, the preset gain is 1 or 0.95, etc. That is, global averaging processing, thus avoiding a large difference in the target gain of each color channel corresponding to each partitioned image data.

[0046] Step 106: Adjust the partitioned image data based on the target gains of the respective color channels corresponding to the partitioned image data to obtain adjusted image data corresponding to the partitioned image data.

[0047] Exemplarily, for each pixel point in the partitioned image data, the electronic device multiplies the pixel values corresponding to the respective color channels of the pixel point by the target gains of the corresponding color channels to obtain adjusted pixel values corresponding to the respective color channels of the pixel point.

[0048] For example, in the partitioned image data, the pixel value of the red channel of the pixel point is R, the pixel value of the green channel is G, and the pixel value of the blue channel is B, and the target gain of the red channel corresponding to the partitioned image data is Rgain, the target gain of the green channel is Ggain, and the target gain of the blue channel is Bgain. Then, the adjusted pixel value of the red channel of the pixel point is Rnew = R * Rgain, the adjusted pixel value of the green channel is Gnew = G * Ggain, and the adjusted pixel value of the blue channel is Bnew = B * Bgain.

[0049] Step 108: Obtain target image data based on the multiple adjusted image data.

[0050] Exemplarily, the electronic device forms the target image data by combining the multiple adjusted image data.

[0051] In an exemplary embodiment, after step 108, it further includes: Displaying a target image corresponding to the target image data based on the target image data.

[0052] In the above image processing method, the spectral response curve of the image sensor in the electronic device to the narrowband light source is different from the spectral response curve of the human eye, resulting in a deviation between the color corresponding to the pixel point in the partitioned image data corresponding to the narrowband spectrum and the actual color. By using the partitioned image data, the spectral curve corresponding to the partitioned image data, and the spectral response curves of the human eye corresponding to the respective color channels, the target gains of the respective color channels corresponding to the partitioned image data are determined. The partitioned image data is adjusted based on the target gains of the respective color channels corresponding to the partitioned image data to obtain adjusted image data corresponding to the partitioned image data, thereby realizing the adjustment of the partitioned image data corresponding to the narrowband spectrum, making the color corresponding to the pixel point in the adjusted image data close to or equal to the actual color. The target image data is obtained based on the multiple adjusted image data, reducing the deviation between the color corresponding to the pixel point in the target image data and the actual color, thereby improving the quality of the image corresponding to the target image data.

[0053] In an exemplary embodiment, determining the target gain of each color channel corresponding to the partitioned image data based on the partitioned image data, the spectral curve corresponding to the partitioned image data, and the spectral response curve of the human eye corresponding to each color channel includes:

[0054] Based on the spectral curve corresponding to the partitioned image data and the spectral response curve of the human eye corresponding to each color channel, determining the first pixel value corresponding to each color channel; based on the pixel values corresponding to each color channel of each pixel point in the partitioned image data, determining the second pixel value corresponding to each color channel; based on the first pixel value and the second pixel value corresponding to each color channel, determining the initial gain corresponding to each color channel; performing smoothing processing on the initial gain corresponding to each color channel to obtain the target gain corresponding to each color channel.

[0055] Wherein, the initial gain refers to the gain determined according to the first pixel value and the second pixel value corresponding to the channel.

[0056] Exemplarily, for each color channel, the electronic device determines the first pixel value corresponding to the color channel based on the retinal response intensity of each wavelength in the spectral response curve of the human eye corresponding to the color channel and the reflectivity of each wavelength in the spectral curve corresponding to the partitioned image data; averaging the pixel values corresponding to the color channel of each pixel point in the partitioned image data to obtain the second pixel value corresponding to the color channel. Based on the average value of the first pixel values corresponding to each color channel and the average value of the second pixel values corresponding to each color channel, determining the initial gain corresponding to each color channel. For each color channel, the electronic device filters the initial gain corresponding to the color channel to obtain the target gain corresponding to the color channel. The filtering methods include but are not limited to mean filtering and Gaussian filtering, etc.

[0057] In this embodiment, by using the partitioned image data, the spectral curve corresponding to the partitioned image data, and the spectral response curve of the human eye corresponding to each color channel, determining the initial gain corresponding to each color channel, and then performing smoothing processing on the initial gain corresponding to each color channel to obtain the target gain corresponding to each color channel, it avoids the situation that the target gain of each color channel corresponding to the partitioned image data is too large or too small.

[0058] In an exemplary embodiment, determining the first pixel value corresponding to each color channel based on the spectral curve corresponding to the partitioned image data and the spectral response curve of the human eye corresponding to each color channel includes:

[0059] For each color channel, obtaining the retinal response intensity corresponding to each wavelength from the spectral response curve of the human eye corresponding to the color channel; obtaining the reflectivity corresponding to each wavelength from the spectral curve; based on the reflectivity and the retinal response intensity corresponding to each wavelength, determining the first pixel value corresponding to the color channel.

[0060] Among them, the wavelength refers to the distance between two adjacent wave crests (or wave troughs) within one period of the waveform during the propagation of a wave. The unit of wavelength can be nanometers (nm). Each wavelength can be a wavelength within a preset wavelength range, and the preset wavelength range can be set according to actual needs. For example, the preset wavelength range can be from 380 nanometers to 750 nanometers, or from 350 nanometers to 780 nanometers. The retinal response intensity refers to the degree of response of the human eye's retina to light of different wavelengths. For different color channels, the degree of response of the human eye's retina to light of the same wavelength is different, that is, the retinal response intensity is determined by the spectral response curve of the human eye corresponding to the color channel and the wavelength. The reflectivity refers to the proportion of light reflected by the surface of an object.

[0061] Exemplarily, for each color channel, obtain the retinal response intensity corresponding to each wavelength from the spectral response curve of the human eye corresponding to the color channel, and obtain the reflectivity corresponding to each wavelength from the spectral curve. For each wavelength, multiply the retinal response intensity corresponding to that wavelength by the reflectivity to obtain the product corresponding to that wavelength, and add up the products corresponding to each wavelength to obtain the first pixel value corresponding to that color channel.

[0062] In an exemplary embodiment, the preset wavelength range is from 380 nanometers to 750 nanometers, and the first pixel value corresponding to the color channel can be calculated by the following formula:

[0063] Formula (1)

[0064] Formula (2)

[0065] Formula (3)

[0066] Among them, R1 is the first pixel value corresponding to the red channel; i is the wavelength, and i takes an integer; SPD[i] is the reflectivity corresponding to wavelength i; RET_R[i] is the retinal response intensity corresponding to wavelength i in the spectral response curve of the human eye corresponding to the red channel; G1 is the first pixel value corresponding to the green channel; RET_G[i] is the retinal response intensity corresponding to wavelength i in the spectral response curve of the human eye corresponding to the green channel; B1 is the first pixel value corresponding to the blue channel; RET_B[i] is the retinal response intensity corresponding to wavelength i in the spectral response curve of the human eye corresponding to the blue channel.

[0067] In this embodiment, the first pixel value corresponding to the color channel is determined by the reflectivity and retinal response intensity corresponding to each wavelength, that is, by fusing the spectral response curve of the image sensor in the electronic device and the spectral response curve of the human eye of the human eye, the first pixel value corresponding to the color channel is determined, providing accurate basic data for subsequent determination of the target gain.

[0068] In an exemplary embodiment, asFigure 5 As shown, determining the initial gain corresponding to each color channel based on the first pixel value and the second pixel value corresponding to each color channel includes:

[0069] Step 502, averaging the first pixel values corresponding to each color channel to obtain a first average pixel value.

[0070] Exemplarily, the electronic device averages the first pixel values corresponding to each color channel to obtain a first average pixel value.

[0071] In one embodiment, the first average pixel value can be calculated by the following formula:

[0072] Formula (4)

[0073] Where AVE1 is the first average pixel value; R1 is the first pixel value corresponding to the red color channel; G1 is the first pixel value corresponding to the green color channel; B1 is the first pixel value corresponding to the blue color channel.

[0074] Step 504, averaging the second pixel values corresponding to each color channel to obtain a second average pixel value.

[0075] Exemplarily, the electronic device averages the second pixel values corresponding to each color channel to obtain a second average pixel value.

[0076] In one embodiment, the first average pixel value can be calculated by the following formula:

[0077] Formula (5)

[0078] Where AVE2 is the first average pixel value; R2 is the first pixel value corresponding to the red color channel; G2 is the first pixel value corresponding to the green color channel; B2 is the first pixel value corresponding to the blue color channel.

[0079] Step 506, for each color channel, determine the initial gain of the color channel based on the first pixel value of the color channel, the second pixel value of the color channel, the first average pixel value, and the second average pixel value.

[0080] Exemplarily, for each color channel, the electronic device determines the initial gain of the color channel based on a first ratio between the first pixel value of the color channel and the first average pixel value, and a second ratio between the second pixel value of the color channel and the second average pixel value.

[0081] In this embodiment, by using the first pixel value of the color channel, the second pixel value of the color channel, the first average pixel value, and the second average pixel value to determine the initial gain of the color channel, accurate basic data is provided for subsequent determination of the target gain.

[0082] In an exemplary embodiment, determining an initial gain of a color channel based on a first pixel value of the color channel, a second pixel value of the color channel, a first average pixel value, and a second average pixel value includes:

[0083] Determining a first ratio of the first pixel value of the color channel to the first average pixel value; determining a second ratio of the second pixel value of the color channel to the second average pixel value; determining a target ratio of the color channel based on the first ratio and the second ratio; and determining the initial gain of the color channel based on the target ratio of the color channel.

[0084] Wherein, the first ratio refers to the ratio between the first pixel value of the color channel and the first average pixel value. The second ratio refers to the ratio between the second pixel value of the color channel and the second average pixel value. The target ratio refers to the ratio between the first ratio and the second ratio.

[0085] Exemplarily, the electronic device divides the first pixel value of the color channel by the first average pixel value to obtain the first ratio, divides the second pixel value of the color channel by the second average pixel value to obtain the second ratio, divides the first ratio by the second ratio to obtain the target ratio of the color channel, and determines the initial gain of the color channel based on the target ratio of the color channel.

[0086] In an exemplary embodiment, determining the initial gain of the color channel based on the target ratio of the color channel includes: multiplying the difference between the target ratio and one by a color modification coefficient to obtain a reference gain; and determining the sum of the reference gain and one as the initial gain of the color channel. Wherein, the color modification coefficient is a coefficient characterizing the color modification intensity.

[0087] In an exemplary embodiment, the initial gain of the color channel can be calculated by the following formula:

[0088] Formula (6)

[0089] Formula (7)

[0090] Formula (8)

[0091] Wherein, Rgain is the initial gain of the red channel; Ggain is the initial gain of the green channel; Bgain is the initial gain of the blue channel; AVE1 is the first average pixel value; R1 is the first pixel value corresponding to the red channel; G1 is the first pixel value corresponding to the green channel; B1 is the first pixel value corresponding to the blue channel; AVE2 is the first average pixel value; R2 is the first pixel value corresponding to the red channel; G2 is the first pixel value corresponding to the green channel; B2 is the first pixel value corresponding to the blue channel.

[0092] In this embodiment, the target ratio of the color channel is determined by the first ratio and the second ratio, and the initial gain of the color channel is determined based on the target ratio of the color channel, providing accurate basic data for subsequent determination of the target gain.

[0093] In an exemplary embodiment, obtaining a plurality of partition image data includes:

[0094] Obtaining the initial image data collected by the image sensor; performing white balance processing on the initial image data to obtain white balance image data; dividing the white balance image data based on the number of sub-regions in the spectral sensor to obtain a plurality of partition image data.

[0095] Among them, the image sensor refers to a component in an electronic device that is used to collect optical signals and convert the optical signals into digital image data. The initial image data refers to digital image data. White balance processing refers to the process of adjusting the color temperature of an image so that white objects in the image remain natural white under different lighting conditions. The spectral sensor refers to a sensor that can capture and analyze light of different wavelengths (including visible light, infrared light, ultraviolet light, etc.). It can obtain the spectral information of a specific object or scene at different wavelengths, thereby generating spectral data, which are used to describe the light reflection or emission characteristics of the object or environment at different wavelengths. The spectral sensor can be divided into multiple sub-regions. A sub-region refers to a local region in the spectral sensor. There are multiple spectral pixel units in the sub-region, and different spectral pixel units can collect optical signals of different bands. It can be understood that there are multiple spectral pixel units in the sub-region, and the multiple spectral pixel units are responsible for collecting optical signals of different bands, and the optical signals of different bands are used to generate the spectral data of the sub-region.

[0096] Exemplarily, the electronic device collects the initial image data through the image sensor, performs white balance processing on the initial image data to obtain white balance image data. When the field of view angle of the spectral sensor is the same as that of the image sensor, the number of sub-regions in the spectral sensor is obtained, and the white balance image data is divided based on the number of sub-regions in the spectral sensor to obtain a plurality of partition image data. For example, a schematic diagram of the white balance image corresponding to the white balance image data is as Figure 6 shown. If the number of sub-regions in the spectral sensor is 16, the white balance image is evenly divided into 16 partition images, and the set composed of the position information and pixel values of all pixel points in the partition image is the partition image data.

[0097] In this embodiment, the initial image data collected by the image sensor is subjected to white balance processing to obtain white balance image data. Based on the number of sub-regions in the spectral sensor, the white balance image data is divided to obtain a plurality of partitioned image data, and the partitioned image data corresponds one-to-one to the spectral information collected by the sub-regions, laying a foundation for subsequently determining the spectral type of the spectral curve corresponding to the partitioned image data.

[0098] In an exemplary embodiment, obtaining the spectral type of the spectral curve corresponding to each partitioned image data includes:

[0099] Obtaining the spectral information collected by each sub-region in the spectral sensor; the field of view angle of the spectral sensor is the same as that of the image sensor; determining the spectral curve corresponding to the spectral information; inputting the spectral curve into the target model to obtain the spectral type of the spectral curve; based on the position of the image block corresponding to the partitioned image data in the white balance image corresponding to the white balance image data, and the position of the sub-region in the spectral sensor, determining the spectral type of the spectral curve corresponding to the partitioned image data.

[0100] Wherein, the spectral information refers to a set of information such as the light signal intensity or reflectivity of different wavelengths collected by the sub-region. The field of view angle refers to the angular range of the covered spatial region, and the field of view angle describes the size or range of the scene that the image sensor or spectral sensor can capture. The target model refers to a neural network model that has been trained to determine the spectral type of the spectral curve. The input of the target model is the spectral curve, and the output of the target model is the spectral type. The image block refers to a local image composed of the partitioned image data. The white balance image refers to an image composed of the white balance image data, and the image block is a local image of the white balance image.

[0101] Exemplarily, when the field of view angle of the spectral sensor is the same as that of the image sensor, the electronic device obtains the spectral information collected by each sub-region in the spectral sensor; for each sub-region, converts the spectral information collected by the sub-region into a spectral curve, inputs the spectral curve into the target model to obtain the spectral type of the spectral curve corresponding to the above sub-region; the position of the image block corresponding to the partitioned image data in the white balance image corresponding to the white balance image data is the same as the position of the sub-region in the spectral sensor, then determines the spectral type of the spectral curve corresponding to the above partitioned image data as the spectral type of the spectral curve corresponding to the above partitioned image data.

[0102] In this embodiment, the field of view angle of the spectral sensor is the same as that of the image sensor, indicating that the scene range of the spectral information captured by the spectral sensor is the same as the scene range captured by the image sensor. The position of the sub-region in the spectral sensor is the same as the position of the image block corresponding to the partitioned image data in the white balance image corresponding to the white balance image data. Then, the spectral information captured by the above sub-region represents the spectral information corresponding to the above partitioned image data, and whether the light source of the partitioned image data is a narrow-band light source can be determined through the spectral curve corresponding to the spectral information captured by the above sub-region.

[0103] In an exemplary embodiment, an imaging module includes:

[0104] An imaging lens;

[0105] A spectral sensor for obtaining spectral information collected by each sub-region in the spectral sensor;

[0106] An image sensor for obtaining initial image data;

[0107] A processor configured to determine a plurality of partitioned image data according to the initial image data, and configured to implement the method of any of the above embodiments.

[0108] Among them, the imaging module refers to a key component in an electronic device for capturing optical signals and generating image data or video data. The imaging module is composed of multiple sub-components, including but not limited to an imaging lens, a spectral sensor, an image sensor, a processor, etc. The imaging lens refers to a lens that focuses optical signals onto the image sensor.

[0109] Exemplarily, the imaging module is composed of an imaging lens, a spectral sensor, an image sensor, and a processor. Each sub-region in the spectral sensor receives the optical signal passing through the imaging lens, and each sub-region respectively converts the optical signal into corresponding spectral information; the image sensor receives the optical signal passing through the imaging lens and converts the optical signal into initial image data; the processor determines a plurality of partitioned image data according to the initial image data and executes the steps of any of the above embodiments.

[0110] In this embodiment, the spectral sensor is used to obtain the spectral information collected by each sub-region in the spectral sensor, and the image sensor is used to obtain the initial image data, providing accurate basic data for the subsequent processor to determine a plurality of partitioned image data and execute the steps of any of the above embodiments.

[0111] In an exemplary embodiment, the linear distance between the spectral sensor and the image sensor is less than a distance threshold, and the difference value between the field of view angle of the spectral sensor and the field of view angle of the image sensor is less than an angle threshold.

[0112] Among them, the distance threshold refers to the maximum straight-line distance between the spectral sensor and the image sensor. The angle threshold refers to the maximum difference value between the field of view angle of the spectral sensor and the field of view angle of the image sensor.

[0113] Exemplarily, in the imaging module of the electronic device, the image sensor is located above the spectral sensor, the straight-line distance between the spectral sensor and the image sensor is less than the distance threshold, and the difference value between the field of view angle of the spectral sensor and the field of view angle of the image sensor is less than the angle threshold. For example, the schematic diagrams of the image sensor and the spectral sensor are as Figure 7 shown. In the imaging module, the image sensor is located above the spectral sensor, the positions of the spectral sensor and the image sensor are close, and the field of view angles of the spectral sensor and the image sensor are equal.

[0114] In an exemplary embodiment, in the imaging module of the electronic device, the image sensor is located below the spectral sensor, the straight-line distance between the spectral sensor and the image sensor is less than the distance threshold, and the difference value between the field of view angle of the spectral sensor and the field of view angle of the image sensor is less than the angle threshold.

[0115] In this embodiment, the straight-line distance between the spectral sensor and the image sensor is less than the distance threshold, and the difference value between the field of view angle of the spectral sensor and the field of view angle of the image sensor is less than the angle threshold, indicating that the positions of the spectral sensor and the image sensor are close and the field of view angles are basically the same, which lays a foundation for implementing the image processing method of the above embodiment.

[0116] In an exemplary embodiment, in the imaging module of the electronic device, the image sensor is located above the spectral sensor, the positions of the spectral sensor and the image sensor are close and the field of view angles are basically the same. The schematic flow diagram of the image processing method is as Figure 8 shown, and includes:

[0117] The electronic device acquires the spectral information collected by each sub-region in the spectral sensor. For each sub-region, the spectral information collected by the sub-region is converted into a spectral curve, and the spectral curve is input into the target model to obtain the spectral type of the spectral curve corresponding to the above sub-region.

[0118] The electronic device acquires initial image data through an image sensor, performs white balance processing on the initial image data to obtain white balance image data; obtains the number of sub-regions in the spectral sensor, and divides the white balance image data based on the data of the sub-regions in the spectral sensor to obtain a plurality of partitioned image data. If the position of the image block corresponding to the partitioned image data in the white balance image corresponding to the white balance image data is the same as the position of the sub-region in the spectral sensor, then determine the spectral type of the spectral curve corresponding to the above-mentioned partitioned image data as the spectral type of the spectral curve corresponding to the sub-region.

[0119] For each partitioned image data, if the spectral type of the spectral curve corresponding to the partitioned image data is a continuous spectrum, then determine the preset gain as the initial gain of each color channel corresponding to the partitioned image data.

[0120] If the spectral type of the spectral curve corresponding to the partitioned image data is a narrowband spectrum, then for each color channel, obtain the retinal response intensity corresponding to each wavelength from the human eye spectral response curve corresponding to the color channel, and obtain the reflectivity corresponding to each wavelength from the spectral curve. For each wavelength, multiply the retinal response intensity corresponding to the wavelength by the reflectivity to obtain the product corresponding to the wavelength, and sum the products corresponding to each wavelength to obtain the first pixel value corresponding to the color channel. That is, use formula (1) to determine the first pixel value corresponding to the red channel, use formula (2) to determine the first pixel value corresponding to the green channel, and use formula (3) to determine the first pixel value corresponding to the blue channel; average the pixel values corresponding to each pixel point in the partitioned image data in the color channel to obtain the second pixel value corresponding to the color channel.

[0121] Average the first pixel values corresponding to each color channel using formula (4) to obtain the first average pixel value; average the second pixel values corresponding to each color channel using formula (5) to obtain the second average pixel value. Substitute the first pixel value corresponding to the red channel, the first average pixel value, the second pixel value corresponding to the red channel, and the second average pixel value into formula (6) to obtain the initial gain of the red channel; substitute the first pixel value corresponding to the green channel, the first average pixel value, the second pixel value corresponding to the green channel, and the second average pixel value into formula (7) to obtain the initial gain of the green channel; substitute the first pixel value corresponding to the blue channel, the first average pixel value, the second pixel value corresponding to the blue channel, and the second average pixel value into formula (8) to obtain the initial gain of the blue channel.

[0122] Perform smoothing processing on the initial gains of each color channel corresponding to each partitioned image data to obtain the target gains of each color channel corresponding to each partitioned image data. The methods for performing smoothing processing include but are not limited to one of mean filtering and Gaussian filtering, etc.

[0123] For each pixel point in the partitioned image data, the electronic device multiplies the pixel values corresponding to the respective color channels of the pixel point by the target gain of the corresponding color channel to obtain the adjusted pixel values corresponding to the respective color channels of the pixel point.

[0124] The electronic device combines multiple adjusted image data to form target image data, and based on the target image data, displays the target image corresponding to the target image data.

[0125] In the above image processing method, the spectral response curve of the image sensor in the electronic device to the narrowband light source is different from the spectral response curve of the human eye, resulting in a deviation between the color corresponding to the pixel point in the partitioned image data corresponding to the narrowband spectrum and the actual color. By using the partitioned image data, the spectral curve corresponding to the partitioned image data, and the spectral response curves of the human eye corresponding to the respective color channels, the target gain of each color channel corresponding to the partitioned image data is determined. The partitioned image data is adjusted based on the target gain of each color channel corresponding to the partitioned image data to obtain the adjusted image data corresponding to the partitioned image data, thereby realizing the adjustment of the partitioned image data corresponding to the narrowband spectrum, making the color corresponding to the pixel point in the adjusted image data close to or equal to the actual color, and obtaining the target image data based on multiple adjusted image data, reducing the deviation between the color corresponding to the pixel point in the target image data and the actual color, thereby improving the quality of the image corresponding to the target image data.

[0126] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are displayed sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0127] Based on the same inventive concept, an embodiment of the present application also provides an image processing apparatus for implementing the above-mentioned image processing method. The solution provided by this apparatus to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the following image processing apparatus can refer to the limitations on the image processing method in the above text, and will not be repeated here.

[0128] In an exemplary embodiment, as Figure 9As shown, an image processing apparatus is provided, including: an acquisition module 902, a determination module 904, an adjustment module 906, and a processing module 908, where:

[0129] The acquisition module 902 is configured to acquire a plurality of partition image data and the spectral types of the spectral curves corresponding to each partition image data.

[0130] The determination module 904 is configured to, for the partition image data corresponding to the spectral curve with a narrowband spectrum type, determine the target gain of each color channel corresponding to the partition image data based on the partition image data, the spectral curve corresponding to the partition image data, and the spectral response curves of the human eye corresponding to each color channel.

[0131] The adjustment module 906 is configured to adjust the partition image data based on the target gain of each color channel corresponding to the partition image data to obtain the adjusted image data corresponding to the partition image data.

[0132] The processing module 908 is configured to obtain the target image data based on a plurality of adjusted image data.

[0133] In an exemplary embodiment, the determination module 904 is further configured to: determine the first pixel value corresponding to each color channel based on the spectral curve corresponding to the partition image data and the spectral response curves of the human eye corresponding to each color channel; determine the second pixel value corresponding to each color channel based on the pixel values corresponding to each color channel of each pixel point in the partition image data; determine the initial gain corresponding to each color channel based on the first pixel value and the second pixel value corresponding to each color channel; perform smoothing processing on the initial gain corresponding to each color channel to obtain the target gain corresponding to each color channel.

[0134] In an exemplary embodiment, the determination module 904 is further configured to: for each color channel, obtain the retinal response intensity corresponding to each wavelength from the spectral response curve of the human eye corresponding to the color channel; obtain the reflectivity corresponding to each wavelength from the spectral curve; determine the first pixel value corresponding to the color channel based on the reflectivity and the retinal response intensity corresponding to each wavelength.

[0135] In an exemplary embodiment, the determination module 904 is further configured to: average the first pixel values corresponding to each color channel to obtain the first average pixel value; average the second pixel values corresponding to each color channel to obtain the second average pixel value; for each color channel, determine the initial gain of the color channel based on the first pixel value of the color channel, the second pixel value of the color channel, the first average pixel value, and the second average pixel value.

[0136] In an exemplary embodiment, the determining module 904 is further configured to: determine a first ratio of a first pixel value of a color channel to a first average pixel value; determine a second ratio of a second pixel value of the color channel to a second average pixel value; determine a target ratio of the color channel based on the first ratio and the second ratio; and determine an initial gain of the color channel based on the target ratio of the color channel.

[0137] In an exemplary embodiment, the obtaining module 902 is further configured to: obtain initial image data collected by an image sensor; perform white balance processing on the initial image data to obtain white balance image data; and divide the white balance image data based on the number of sub-regions in the spectral sensor to obtain a plurality of partitioned image data.

[0138] In an exemplary embodiment, the obtaining module 902 is further configured to: obtain spectral information collected by each sub-region in the spectral sensor; the field of view angle of the spectral sensor is the same as that of the image sensor; determine a spectral curve corresponding to the spectral information; input the spectral curve into a target model to obtain a spectral type of the spectral curve; and determine the spectral type of the spectral curve corresponding to the partitioned image data based on the position of the image block corresponding to the partitioned image data in the white balance image corresponding to the white balance image data and the position of the sub-region in the spectral sensor.

[0139] Each module in the above image processing device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in a processor in an electronic device in a hardware form or be independent of the processor, or can be stored in a memory in the electronic device in a software form, so that the processor can call and execute the operations corresponding to the above respective modules.

[0140] In an exemplary embodiment, an imaging module is provided, including: an imaging lens; a spectral sensor configured to obtain spectral information collected by each sub-region in the spectral sensor; an image sensor configured to obtain initial image data; and a processor configured to determine a plurality of partitioned image data based on the initial image data and configured to implement the method according to any one of the above embodiments.

[0141] In an exemplary embodiment, an imaging module is provided, where the straight-line distance between the spectral sensor and the image sensor is less than a distance threshold, and the difference value between the field of view angle of the spectral sensor and the field of view angle of the image sensor is less than an angle threshold.

[0142] In an exemplary embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as Figure 10As shown in the figure. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and external devices. The communication interface of the electronic device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the electronic device, or an external keyboard, touchpad, or mouse, etc.

[0143] Those skilled in the art can understand that Figure 10 the structure shown in the figure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0144] In one embodiment, an electronic device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0145] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0146] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0147] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0148] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, artificial intelligence (AI) processors, etc., and are not limited thereto.

[0149] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this application.

[0150] The above-described embodiments merely represent several implementation manners of this application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of this application. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of this application shall be subject to the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a plurality of subarea image data and a spectrum type of a spectrum curve corresponding to each of the subarea image data; For the subarea image data corresponding to the spectral curve whose spectral type is a narrowband spectrum, based on the subarea image data, the spectral curve corresponding to the subarea image data, and the human eye spectral response curve corresponding to each color channel, determine the target gain of each color channel corresponding to the subarea image data; Based on the target gain of each color channel corresponding to the subarea image data, the subarea image data is adjusted to obtain adjusted image data corresponding to the subarea image data; Based on the plurality of adjusted image data, target image data is obtained.

2. The method according to claim 1, characterized in that The determining, based on the subarea image data, the spectral curve corresponding to the subarea image data and the human eye spectral response curve corresponding to each color channel, the target gain of each color channel corresponding to the subarea image data comprises: Determine a first pixel value corresponding to each color channel based on a spectral curve corresponding to the partitioned image data and a human eye spectral response curve corresponding to each color channel; Determine, based on the pixel value corresponding to each color channel of each pixel point in the partitioned image data, a second pixel value corresponding to each color channel; Determining initial gains corresponding to the color channels based on first pixel values ​​and second pixel values ​​corresponding to the color channels; The initial gains corresponding to the color channels are smoothed to obtain target gains corresponding to the color channels.

3. The method according to claim 2, characterized in that The determining, based on the spectral curve corresponding to the partitioned image data and the human eye spectral response curve corresponding to each color channel, the first pixel value corresponding to each color channel comprises: For each color channel, obtaining the retinal response intensity corresponding to each wavelength from the human eye spectral response curve corresponding to the color channel; Obtaining the reflectivity corresponding to each wavelength from the spectral curve; Based on the reflectivity corresponding to each wavelength and the retinal response intensity, a first pixel value corresponding to the color channel is determined.

4. The method according to claim 2, characterized in that: The determining the initial gain corresponding to each color channel based on the first pixel value and the second pixel value corresponding to each color channel includes: Averaging the first pixel values ​​corresponding to the color channels to obtain a first average pixel value; Averaging the second pixel values ​​corresponding to the color channels to obtain a second average pixel value; For each color channel, an initial gain of the color channel is determined based on a first pixel value of the color channel, a second pixel value of the color channel, the first average pixel value, and the second average pixel value.

5. The method according to claim 4, characterized in that The determining the initial gain of the color channel based on the first pixel value of the color channel, the second pixel value of the color channel, the first average pixel value, and the second average pixel value comprises: determining a first ratio of a first pixel value of the color channel to the first average pixel value; determining a second ratio of a second pixel value of the color channel to the second average pixel value; Determining a target ratio of the color channel based on the first ratio and the second ratio; Based on the target ratio of the color channel, an initial gain of the color channel is determined.

6. The method according to claim 1, characterized in that The step of acquiring a plurality of subarea image data comprises: Acquire initial image data collected by the image sensor; Performing white balance processing on the initial image data to obtain white balance image data; The white balance image data is divided based on the number of sub-regions in the spectral sensor to obtain a plurality of sub-region image data.

7. The method according to claim 6, characterized in that The obtaining of the spectrum type of the spectrum curve corresponding to each of the subarea image data comprises: Acquiring spectral information collected by each sub-area in the spectral sensor; the field of view of the spectral sensor is the same as the field of view of the image sensor; Determining a spectral curve corresponding to the spectral information; Inputting the spectral curve into a target model to obtain a spectral type of the spectral curve; The spectrum type of the spectrum curve corresponding to the subregion image data is determined based on the position of the image block corresponding to the subregion image data in the white balance image corresponding to the white balance image data and the position of the subregion in the spectrum sensor.

8. An imaging module, characterized in that: include: Imaging lens; A spectral sensor, the spectral sensor is used to obtain spectral information collected by each sub-region in the spectral sensor; An image sensor, wherein the image sensor is used to acquire initial image data; A processor is configured to determine a plurality of subarea image data according to the initial image data, and is configured to implement the method according to any one of claims 1 to 7.

9. The imaging module according to claim 8, characterized in that: A straight-line distance between the spectral sensor and the image sensor is smaller than a distance threshold, and a difference between a field of view angle of the spectral sensor and a field of view angle of the image sensor is smaller than an angle threshold.

10. An image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a plurality of subarea image data and a spectrum type of a spectrum curve corresponding to each of the subarea image data; a determination module, for determining, for the subarea image data corresponding to the spectral curve whose spectral type is a narrowband spectrum, a target gain of each color channel corresponding to the subarea image data based on the subarea image data, the spectral curve corresponding to the subarea image data, and the human eye spectral response curve corresponding to each color channel; an adjustment module, configured to adjust the subarea image data based on the target gain of each color channel corresponding to the subarea image data, so as to obtain adjusted image data corresponding to the subarea image data; The processing module is used to obtain target image data based on the multiple adjusted image data.

11. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images