Image processing method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to an image processing method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: performing first preprocessing on first original domain image data acquired by a multispectral sensor to obtain first image data; performing second preprocessing on second original domain image data acquired by the color sensor to obtain second image data; aligning the first image data with the second image data to obtain third image data corresponding to the first image data; and performing color restoration processing on the second image data according to the third image data to obtain a target image. By adopting the method, the image color rendition precision can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] Currently, widely used electronic devices such as cameras and mobile phones are equipped with color image sensors, such as RGB image sensors. RGB image sensors are usually composed of three separate photoreceptors, each of which corresponds to the three color channels of red (Red, R), green (Green, G), and blue (Blue, B). When light enters the sensor, it is divided into different wavelengths, namely red light, green light, and blue light, and these three colors of light are received by the corresponding receptors in the sensor.

[0003] However, three-channel images can only record the chromaticity information of the target object under specific lighting, and it is difficult to obtain device-independent color representation. Under some special lighting and light conditions, objects with different reflective characteristics may appear the same color, resulting in the phenomenon of metamerism. Three-channel images are difficult to distinguish the spectral differences of these objects, resulting in limited image color reproduction performance and inability to accurately restore image colors. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, device, electronic device, and computer-readable storage medium, which can improve the accuracy of image color restoration.

[0005] In a first aspect, the present application provides an image processing method, which is applied to an electronic device, wherein the electronic device includes a multispectral sensor and a color sensor; the method includes:

[0006] Performing a first preprocessing on the first original domain image data acquired by the multispectral sensor to obtain first image data;

[0007] Performing a second preprocessing on the second original domain image data acquired by the color sensor to obtain second image data;

[0008] Aligning the first image data with the second image data to obtain third image data corresponding to the first image data;

[0009] The second image data is subjected to color restoration processing according to the third image data to obtain a target image.

[0010] In a second aspect, the present application further provides an image processing device, comprising:

[0011] A first image data processing module, used for performing a first preprocessing on the first original domain image data acquired by the multispectral sensor to obtain first image data;

[0012] A second image data processing module, used for performing a second preprocessing on the second original domain image data acquired by the color sensor to obtain second image data;

[0013] An image data alignment module, used for aligning the first image data with the second image data to obtain third image data corresponding to the first image data;

[0014] The color restoration processing module is used to perform color restoration processing on the second image data according to the third image data to obtain a target image.

[0015] In a third aspect, the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the image processing method provided in the first aspect when executing the computer program.

[0016] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the image processing method provided in the first aspect are implemented.

[0017] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the image processing method provided in the first aspect.

[0018] The above-mentioned image processing method, device, electronic device, computer-readable storage medium and computer program product obtain first image data by performing a first preprocessing on the first original domain image data acquired by the multispectral sensor, obtain second image data by performing a second preprocessing on the second original domain image data acquired by the color sensor, align the first image data and the second image data to obtain third image data corresponding to the first image data, perform color restoration processing on the second image data according to the third image data to obtain a target image, and can realize color restoration processing of the original domain image data acquired by the color sensor with the assistance of original domain image data of more spectra, that is, color restoration processing of the three-channel image data of the original domain is performed through the multispectral information of the original domain, which can improve the accuracy of color restoration, thereby improving the color accuracy of the target image. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 is a schematic flow chart of an image processing method in some embodiments;

[0021] Figure 2 A schematic diagram of a field of view angle corresponding to a multispectral sensor and a field of view angle corresponding to a color sensor in some embodiments;

[0022] Figure 3 is a schematic flow chart of an image processing method in some other embodiments;

[0023] Figure 4 is a schematic flow chart of an image processing method in some other embodiments;

[0024] Figure 5 is a schematic flow chart of an image processing method in some other embodiments;

[0025] Figure 6 is a structural block diagram of an image processing device in some embodiments;

[0026] Figure 7 1 is a diagram of the internal structure of an electronic device in some embodiments. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0028] At present, most of the CMOS (Complementary Metal Oxide Semiconductor) sensors commonly used in electronic devices such as mobile phones are RGB three-channel. However, three-channel images can only record the chromaticity information of the target under specific lighting, which makes it difficult to obtain device-independent color representation. Under certain lighting and observation conditions, objects with different reflective characteristics may appear to have the same color, that is, the phenomenon of metamerism, but three-channel images are difficult to distinguish the spectral differences of these objects. In addition, due to the wide spectral band design of the RGB channel, conventional three-channel images lack the ability to provide detailed information in specific bands.

[0029] Most cameras used in daily life use RGB three-channel imaging, which cannot fully capture the real spectral data. In many practical applications, the demand for spectral information goes beyond the scope of traditional RGB imaging. Some applications require a specific combination of visible light bands to improve imaging effects in specific environments; others need to capture information outside the visible light band, such as ultraviolet light for material detection, or infrared light for night monitoring. With the continuous upgrading of demands in scientific research and industrial production, imaging scenes have become more complex, and many tasks require more or narrower spectral channels. Therefore, in order to make up for the shortcomings of three-channel images, multispectral imaging technology came into being. It reveals the richer physical information of the target hidden under the chromaticity by expanding the resolution of the spectral dimension.

[0030] Multispectral imaging is a technology that generates images by capturing light information from multiple spectral bands, usually covering a wide wavelength range from ultraviolet to near-infrared. In traditional technology, a global color sensor device that can capture multi-channel spectral information is added to the RGB three-channel sensor. The camera's white balance correction and color restoration are then performed based on the information from the RGB sensor and the global color sensor. However, most of the multi-color temperature sensors used in cameras are single-point multi-channel devices, and the number of channels is generally between 5-15. Because it is a single-point device, it cannot identify the color temperature of different areas in space, and outputs a comprehensive, average color temperature information. This device and color processing solution cannot achieve accurate color restoration of the camera.

[0031] In response to the above-mentioned problem of low color restoration accuracy, an embodiment of the present application provides an image processing method, which uses the original domain image data obtained by a multispectral sensor to assist in color restoration of the original domain image data obtained by a color sensor, thereby improving the precision and accuracy of color restoration, thereby improving imaging quality.

[0032] In an exemplary embodiment, Figure 1 As shown, an image processing method is provided, which is applied to an electronic device, the electronic device includes a multi-spectral sensor and a color sensor, and the method includes the following steps 102 to 108. Among them:

[0033] Step 102: Perform a first preprocessing on the first original domain image data acquired by the multispectral sensor to obtain first image data.

[0034] Among them, the multispectral sensor is a technical device that uses light in different wavelength ranges for remote sensing detection and analysis. The multispectral sensor can divide the incident full-band or wide-band light signal into several narrow-band light beams, and then image these light beams on the corresponding detectors. The detector is usually a series of pixel arrays, each pixel corresponding to a specific spectral band. When the light signal is irradiated on the surface of an object, the object will reflect or emit light of a specific band, which is captured by the sensor and converted into electrical signals, and then the characteristic information of the target material in different spectral bands is extracted through data processing and image analysis. These spectral channels usually cover multiple bands such as visible light, near infrared, and mid-infrared. Multispectral sensors include single-point multispectral sensors and multi-point multispectral sensors. Multi-point multispectral can also be called multi-zone multispectral. Multi-point multispectral sensors can capture multispectral image data at multiple spatial locations at the same time, while single-point multispectral sensors capture multispectral image data at a fixed point. Multispectral sensors can also include partitioned multispectral sensors, which can capture information of target objects in different spectral bands and divide the information into multiple regions according to spatial positions for analysis. Different regions have different spectral characteristics.

[0035] The first original domain image data refers to the original domain data collected by the multispectral sensor, that is, raw domain data. In other words, the first original domain image data is the image data collected by the multispectral sensor without being processed by any image processing flow.

[0036] Exemplarily, the electronic device includes an image signal processor (ISP), and the image signal processor may include a two-stage or three-stage processing stage for image data. Regardless of whether the two-stage or three-stage processing flow is included, the first original domain image data can be processed by the first stage processing flow to obtain the first stage image data, and then the first stage image data is spectrally reconstructed to obtain the first image data. In other words, the first preprocessing may include the processing of the first stage of the image data processing stage by the image signal processor and the processing of spectral reconstruction.

[0037] Step 104: Perform a second preprocessing on the second original domain image data acquired by the color sensor to obtain second image data.

[0038] Among them, the color sensor may include an RGB sensor, an HSV / HSI sensor, a CMYK sensor, or a monochrome color sensor. Among them, the RGB sensor usually determines the color of the object according to the ratio of the three colors of light by detecting the intensity of red light, green light, and blue light reflected or transmitted by the object. The HSV / HSI sensor describes the color based on hue, saturation, and brightness. CMYK stands for cyan, magenta, yellow, and key. The monochrome color sensor has the highest frequency response to light of a specific wavelength. The color sensor usually constructs three color filters of red, green, and blue on the photodiode of each pixel, while multispectral imaging constructs more color filters on the photodiode of the image sensor, which can transmit light of different wavelengths. Generally, the number of color channels of the color sensor is less than the number of color channels of the multispectral sensor. The second original domain image data refers to the original domain data collected by the color sensor, that is, the raw domain data. In other words, the second original domain image data is the image data collected by the color sensor without any image processing process. It is easy to understand that the second original domain image data and the first original domain image data are image data collected for the same scene.

[0039] Exemplarily, the electronic device includes an image signal processor, and the image signal processor may include two or three stages of processing the image data. If the image signal processor may include two stages of processing the image data, the second preprocessing may include the processing of the first stage. If the image signal processor may include three stages of processing the image data, the second preprocessing may include the processing of the first stage and the second stage.

[0040] Step 106: align the first image data with the second image data to obtain third image data corresponding to the first image data.

[0041] The first image data and the second image data are aligned, which means that the first image data is aligned with the second image data based on the second image data. The third image data is used to represent the first image data aligned with the second image data. The first image data and the second image data can be aligned by a computer vision processor.

[0042] Exemplarily, the first image data and the second image data may be aligned by motion estimation or geometric transformation. Motion estimation refers to estimating the motion information between the first image data and the second image data, and then performing motion compensation on the first image data so that the first image data is aligned with the second image data. Geometric transformation refers to translating, rotating, scaling, twisting, deforming, etc. the first image data so that the first image data is aligned with the second image data.

[0043] Step 108, performing color restoration processing on the second image data according to the third image data to obtain a target image.

[0044] The third image data may represent the spectral information of the photographed object, and the color information lost in the second image data may be restored according to the third image data to obtain a target image.

[0045] Exemplarily, the third image data may be spectrally estimated to obtain the spectral power of the light source in the shooting scene, and the color restoration parameters may be determined based on the spectral power, thereby performing color restoration on the second image data based on the color restoration parameters to obtain the target image. Spectral estimation may be performed based on the third image data to obtain the local spectral power of each region in the shooting scene, for example, highlight detection may be performed on a multi-spectral image to obtain the highlight region, thereby obtaining the local spectral power corresponding to the highlight region. Then, based on the local spectral power of each region in the shooting scene, the local color restoration parameters of each region are determined, and the local color restoration parameters include a local color correction matrix. Based on the local color correction matrix, the color of the corresponding region in the second image data may be restored.

[0046] In the above-mentioned image processing method, a first preprocessing is performed on the first original domain image data acquired by the multispectral sensor to obtain the first image data, a second preprocessing is performed on the second original domain image data acquired by the color sensor to obtain the second image data, the first image data and the second image data are aligned to obtain the third image data corresponding to the first image data, and the second image data is color restored according to the third image data to obtain the target image. This method can realize color restoration processing of the original domain image data acquired by the color sensor with the assistance of original domain image data of more spectra, that is, color restoration processing of the three-channel image data of the original domain is performed through the multispectral information of the original domain, which can improve the accuracy of color restoration, thereby improving the color accuracy of the target image.

[0047] In some embodiments, the image signal processor of the electronic device processes the image data in stages including a first stage, a second stage, and a third stage; and the step 102 of performing a first preprocessing on the first original domain image data acquired by the multispectral sensor to obtain the first image data includes:

[0048] The first original domain image data is processed in the first stage to obtain intermediate image data; and the intermediate image data is spectrally reconstructed to obtain first image data.

[0049] The electronic device includes an image signal processor, and the processing stages of the image signal processor include the first stage, the second stage, and the third stage. Among them, the processing of the first stage may include sensor nonlinear correction, PD pixel correction (Phase Detection Pixel Correction), deduction of OB (Optical Black), lens distortion correction, raw domain noise reduction and other processing, the processing of the second stage may include defective pixel correction (DPC), black level correction (BLC), lens shading correction, Auto White Balance (AWB), demosaicing and other processing, and the processing of the third stage may include color space conversion, brightness noise reduction, color noise reduction, edge enhancement, saturation adjustment, color adjustment, brightness control, contrast enhancement and other processing. It should be noted that the first stage and the second stage are mainly processes for processing raw domain (raw domain) image data, the first stage may refer to the preprocessing of raw domain data, the second stage may refer to the processing of preprocessed raw domain data, and the third stage is the process of processing YUV domain image data. Among them, the YUV domain refers to a pixel encoding format that represents the brightness information (Y) and chrominance information (U, V) of an image separately. Among them, Y represents brightness (Luminance or Luma), which is the grayscale value of the image; U and V represent chrominance (Chrominance or Chroma), which are the two components that constitute color. Exemplarily, the first stage can be a pre row domain process, the second stage can be a raw domain process, and the third stage can be a yuv domain process.

[0050] Exemplarily, the first stage of processing can be performed on the first original domain image data, that is, sensor nonlinear correction, PD pixel correction and other processing are performed in sequence to obtain intermediate image data, and then the intermediate image data is spectrally reconstructed to obtain the first image data. Among them, spectral reconstruction is a process of splitting the original spectral data into several basic components and then reconstructing them according to these basic components to obtain a clearer and more accurate waveform. The essence of spectral reconstruction is to recover the original spectral information from the measured spectral data. In actual application scenarios, the intermediate image data can be estimated by light source to obtain a light source estimation result, and the intermediate image data can be spectrally reconstructed according to the light source estimation result to obtain a spectrally reconstructed image, and each pixel in the spectrally reconstructed image is sequentially chromaticity calculated to obtain chromaticity image data, and the chromaticity image data is converted to an xyz color space, and then the chromaticity image data is converted from the xyz color space to the RGB color space. For example, the chromaticity image data can be converted to the RGB color space through the p3 color gamut or the conversion matrix to obtain the first image data. It should be noted that in the xyz color space, each color is described by three values ​​x, y, and z, which are called tristimulus values. x is mainly associated with red, y is mainly associated with green (and is proportional to the brightness), and z is related to blue. Through the combination of these values, xyz can cover all colors visible to the human eye. The y component in the xyz color space is specially designed as a brightness value, while x and z represent other color information respectively. This brightness separation design enables the xyz space to more accurately represent the human eye's separate perception of light intensity and color. In other words, the first preprocessing includes the first stage of processing and the spectral reconstruction process.

[0051] Exemplarily, performing spectral reconstruction on the intermediate image data to obtain the first image data may include: identifying scene information of the shooting scene based on the second image data by an artificial intelligence algorithm, and then performing spectral reconstruction on the intermediate image data in combination with the scene information of the shooting scene to obtain the first image data. Since the spectral reconstruction is performed in combination with the scene information, the accuracy of the spectral reconstruction can be further improved.

[0052] In this embodiment, in a three-stage image data processing flow, the first original domain image data is processed in the first stage to obtain intermediate image data, and then the intermediate image data is spectrally reconstructed to obtain the first image data. That is, the first original domain image data is first processed in the first stage and then spectrally reconstructed. This can improve the accuracy of spectral reconstruction, thereby improving the accuracy of spectral information in the first image data.

[0053] In some embodiments, the image signal processor of the electronic device processes the image data in stages including a first stage, a second stage, and a third stage; performing a second preprocessing on the second original domain image data acquired by the color sensor to obtain the second image data includes:

[0054] The second original domain image data is processed in the first stage and the second stage in sequence to obtain second image data.

[0055] It is easy to understand that the image signal processor in this embodiment processes image data in the first stage, the second stage and the third stage. Please refer to the explanation and description of the corresponding processing stages in the aforementioned embodiments, and will not be repeated here.

[0056] In this embodiment, the second original domain image data is processed in the first stage and the second stage in sequence to obtain the second image data, that is, the second preprocessing may include the first stage and the second stage processing. The first stage processing is performed on the second original domain image data to obtain the first stage processed data, and then the second stage processing is performed on the first stage processed data to obtain the second image data.

[0057] In this embodiment, by sequentially performing the first stage and the second stage processing on the second original domain image data, more accurate second image data can be obtained, and color restoration is performed based on the second image data, thereby improving the accuracy of color restoration.

[0058] In some embodiments, the image signal processor of the electronic device processes the image data including raw domain processing and YUV domain processing; and the step 102 of performing first preprocessing on the first raw domain image data acquired by the multispectral sensor to obtain the first image data includes:

[0059] The first original domain image data is processed in the original domain to obtain image data to be reconstructed; and the image data to be reconstructed is spectrally reconstructed to obtain first image data.

[0060] In this embodiment, the electronic device includes an image signal processor, and the image signal processor includes raw domain processing and YUV domain processing for image data, that is, the image signal processor has a two-stage processing flow for image data. Among them, raw domain processing refers to the process of processing raw domain image data, and YUV domain refers to the process of processing YUV domain image data. Raw domain processing may include the first stage and the second stage processing, namely sensor nonlinear correction, PD pixel correction, OB deduction, lens distortion correction, raw domain noise reduction, bad pixel correction, black level correction, lens shadow correction, automatic white balance, de-mosaicing and other processing. The YUV domain may include the third stage processing, namely color space conversion, brightness noise reduction, color noise reduction, edge enhancement, saturation adjustment, color adjustment, brightness control, contrast enhancement and other processing.

[0061] In this embodiment, the first original domain image data is processed in the original domain to obtain image data to be reconstructed, and then the image data to be reconstructed is spectrally reconstructed to obtain the first image data. Since the spectral reconstruction is performed after the first original domain image data is processed in the original domain, the accuracy of the spectral reconstruction can be further improved, thereby improving the accuracy of the first image data.

[0062] In some embodiments, performing spectral reconstruction on the image data to be reconstructed to obtain first image data includes:

[0063] The scene information of the shooting scene is identified based on the second image data through an artificial intelligence algorithm; the spectral reconstruction is performed on the image data to be reconstructed in combination with the scene information to obtain the first image data.

[0064] Among them, the artificial intelligence algorithm is implemented based on deep learning and computer vision technology. The artificial intelligence algorithm can identify corresponding scene information based on the captured image data. Exemplarily, the electronic device identifies the scene information of the captured scene based on the second image data through the artificial intelligence algorithm, and can determine different spectral reconstruction methods to perform spectral reconstruction on the image data to be reconstructed according to the scene type corresponding to the scene information to obtain the first image data, thereby improving the accuracy of spectral reconstruction.

[0065] In actual application scenarios, a corresponding artificial intelligence algorithm processing flow can be added to the image signal processor's image data processing flow to realize the function of AI-Camera (smart camera), and the scene information corresponding to the second image data is identified according to the AI-Camera. According to the identified scene information, the image data to be reconstructed is spectrally reconstructed to obtain the first image data. Among them, the scene information is used to characterize the corresponding scene, for example, the scene information includes portrait scenes, blue scenes, green plant scenes, mixed light scenes, etc. The artificial intelligence algorithm can also be an artificial intelligence model.

[0066] In this embodiment, the scene information corresponding to the second image data is identified by an artificial intelligence algorithm, and then the spectral reconstruction of the image data to be reconstructed is performed in combination with the scene information, that is, the spectral reconstruction is performed in combination with the scene information, which can improve the accuracy of the spectral reconstruction.

[0067] In some embodiments, the image signal processor of the electronic device processes the image data in a process including raw domain processing and YUV domain processing; and the step 104 of performing a second preprocessing on the second raw domain image data acquired by the color sensor to obtain the second image data includes:

[0068] The second original domain image data is processed in the original domain to obtain second image data.

[0069] Among them, for the specific process of raw domain processing and YUV domain processing, please refer to the introduction of the corresponding content in the above embodiment, which will not be repeated here.

[0070] In this embodiment, in the two-stage processing, the first stage of original domain processing is performed on the second original domain image data to obtain the second image data, which can remove low-quality or invalid pixel information, improve the quality of the three-channel image data, and further improve the accuracy of color restoration based on the three-channel image.

[0071] In some embodiments, the step 106 of aligning the first image data with the second image data to obtain third image data corresponding to the first image data includes:

[0072] The motion information between pixels at corresponding positions in the first image data and the second image data is calculated by a computer vision processor; the motion compensation is performed on the first image data according to the motion information by the computer vision processor to obtain third image data corresponding to the first image data.

[0073] The electronic device includes a computer vision processor. It is easy to understand that the computer vision processor can be independent of the image signal processor or can be set in the image signal processor. The computer vision processor is used to calculate the motion information between pixels at corresponding positions in the first image data and the second image data. For example, the motion speed and direction between pixels at corresponding positions in the first image data and the second image data can be obtained by optical flow estimation, that is, the motion information includes the motion speed and direction. After obtaining the motion information between pixels at all positions, the optical flow field between the first image data and the second image data can be obtained. Then, motion compensation is performed on the first image data according to the motion information between pixels at corresponding positions to obtain third image data corresponding to the first image data.

[0074] In one example, the motion information between the pixels at the positions in the first image data and the second image data can be calculated by a computer vision processor, and the first image data can be motion compensated according to the motion information to obtain the motion compensated image data, and the motion compensated image data can be geometrically transformed to align with the second image data to obtain the third image data corresponding to the first image data. Geometric transformation refers to translation, rotation, scaling, distortion and deformation of the image, and the geometric transformation can include affine transformation and transmission transformation.

[0075] In this embodiment, motion information between pixels at corresponding positions in the first image data and the second image data is calculated by a computer vision processor, and motion compensation is performed on the first image data based on the motion information to obtain third image data aligned with the second image data corresponding to the first image data. That is, by performing motion information calculation and motion compensation by a computer vision processor, the calculation speed can be improved and the alignment efficiency can be improved. In addition, by calculating the motion information between pixels at corresponding positions in the first image data and the second image data and performing motion compensation for the corresponding pixels, the alignment precision can be improved and the alignment accuracy can be improved.

[0076] In some embodiments, the image signal processor of the electronic device processes the image data in stages including a first stage, a second stage, and a third stage; and step 108 of performing color restoration processing on the second image data according to the third image data to obtain a target image includes:

[0077] The third image data is smoothed to obtain smoothed third image data; the second image data is processed in a third stage according to the smoothed third image data to obtain a target image.

[0078] Among them, the image signal processor's processing stages of image data include the first stage, the second stage and the third stage. Please refer to the introduction and description of the corresponding contents in the above embodiments, and will not be repeated here.

[0079] In this embodiment, based on the three-stage processing stage of the image data by the image signal processor, the color restoration processing includes smoothing processing and the third stage of processing. The third image data can be smoothed by a filtering algorithm, and the filtering algorithm can be at least one of mean filtering, Gaussian filtering, median filtering, bilateral filtering or non-local mean filtering. The specific filtering algorithm can be selected according to the actual application scenario.

[0080] Exemplarily, the third image data may be smoothed to obtain smoothed third image data, chromaticity edge detection may be performed on the smoothed third image data, and then the second image data may be processed in a third stage based on the detected chromaticity edge to obtain a target image.

[0081] In this embodiment, by smoothing the third image data to obtain smoothed third image data, the mutation or noise between pixels in the third image data can be reduced, and then the second image data is processed in the third stage based on the smoothed third image data, which can improve the color restoration accuracy of the second image data.

[0082] In the three-stage processing of image data by the image signal processor, the second image data is processed in the third stage according to the smoothed third image data, while in the two-stage processing, the second image data is processed in the YUV domain according to the smoothed third image data.

[0083] In some embodiments, the image signal processor of the electronic device processes the image data in a process including raw domain processing and YUV domain processing; and the step 108 of performing color restoration processing on the second image data according to the third image data to obtain the target image includes:

[0084] The third image data is smoothed to obtain smoothed third image data; and the second image data is processed in the YUV domain according to the smoothed third image data to obtain a target image.

[0085] The processing process of the image signal processor on the original domain processing and the YUV domain processing of the image data can be found in the introduction of the corresponding contents in the above embodiments, which will not be repeated here.

[0086] In this embodiment, by smoothing the third image data, the mutation or noise between pixels in the third image data can be reduced, and then the second image data is processed in the YUV domain based on the smoothed third image data, which can further improve the color restoration accuracy of the second image data.

[0087] In some embodiments, performing smoothing on the third image data to obtain smoothed third image data includes:

[0088] The scene information of the shooting scene is identified based on the second image data through an artificial intelligence algorithm; the third image data is smoothed in combination with the scene information to obtain smoothed third image data.

[0089] Among them, the processing flow of identifying the scene information corresponding to the second image data based on the artificial intelligence algorithm can be found in the description of the corresponding content in the above embodiment, and will not be repeated here.

[0090] After the scene information corresponding to the second image data is identified by the artificial intelligence algorithm, a corresponding smoothing method can be determined according to the scene information, and the third image data can be smoothed to obtain smoothed third image data. For example, for a portrait scene with more noise, a mean filter or a Gaussian filter can be used for smoothing, and for a green plant scene that needs to maintain edge information, a median filter or a bilateral filter can be used for smoothing.

[0091] In this embodiment, the scene information corresponding to the second image data is identified by an artificial intelligence algorithm, and the third image data is smoothed in combination with the scene information. This allows for targeted smoothing based on the scene information, thereby improving the accuracy of the smoothing process.

[0092] It is easy to understand that in the process of smoothing the third image data to obtain the smoothed third image data, smoothing can be performed according to pixels in the third image data, or the third image data can be divided into blocks and smoothed according to image blocks.

[0093] In an exemplary embodiment, performing smoothing on the third image data to obtain smoothed third image data includes:

[0094] Color estimation is performed on pixels in the third image data to obtain pixel color estimation values; and smoothed third image data is obtained according to the pixel color estimation values ​​of each pixel in the third image data.

[0095] Among them, color estimation refers to the process of recalculating pixels. For example, the color estimation of the target pixel can be achieved by filtering the pixels in the neighborhood of the target pixel. The target pixel can be any pixel in the third image data. Exemplarily, if the color estimation of the pixel is performed by mean filtering, the pixel value of the target pixel can be replaced by the average value of the pixels in the target pixel and its neighborhood; if the color estimation of the pixel is performed by median filtering, the pixel value of the target pixel can be replaced by the median of the pixel values ​​of all pixels in its domain; if the selective edge preserving smoothing method is used, the mean and variance of the pixels in the neighborhood of the target pixel are calculated using windows of different shapes, and the pixel mean corresponding to the window with the smallest variance is used as the pixel value of the target pixel.

[0096] Exemplarily, the pixel color estimation values ​​of the pixels in the third image data may be arranged and combined according to the original pixel positions to obtain the smoothed third image data.

[0097] In this embodiment, by performing pixel-by-pixel color estimation on the third image data to obtain a pixel color estimation value of each pixel, and obtaining a smoothed third image data based on the pixel color estimation value of each pixel, the precision and accuracy of smoothing can be improved, that is, image data with better smoothness can be obtained.

[0098] In an exemplary embodiment, performing smoothing on the third image data to obtain smoothed third image data includes:

[0099] The third image data is processed into blocks to obtain a plurality of image blocks; a feature is extracted for each image block to obtain an image block feature; the similarity between the image blocks is calculated based on the image block feature; each image block is matched based on the similarity between the image blocks to obtain a matching block, and the smoothed third image data is obtained based on the matching block.

[0100] The image block feature refers to a feature that characterizes image information, such as color, texture, and gradient. The third image data is processed into blocks to obtain a plurality of image blocks, wherein the image blocks may overlap or may not overlap, and the size and number of the image blocks may be set according to the actual application scenario. For example, the size of each image block is 4*4, 8*8 pixels, or 16*16 pixels, etc., but is not limited thereto. Generally, the larger the image block, the more obvious the smoothing effect.

[0101] Exemplarily, feature extraction is performed on each image block to obtain image block features, and the similarity between image block features of different image blocks is calculated. For example, the similarity between image block features of any two image blocks is calculated, and the similarity between the two image block features can be characterized by the mean square error (MSE) or structural similarity (SSIM) of the two image block features. If the mean square error is smaller, it means that the similarity between the two image block features is greater, that is, the more similar they are, or, the closer the value of the structural similarity result is to 1, it means that the similarity between the two image block features is greater.

[0102] For any target image block among the multiple image blocks, the similarity between the target image block and each other image block among the multiple image blocks except the target image block can be obtained, and the other image blocks with the highest similarity to the target image block can be determined as the matching blocks of the target image block, or a target number of other image blocks with similarities to the target image block from high to low can be determined as the matching blocks of the target image block. According to the matching block of the target image block, the smoothed third image data can be obtained. Exemplarily, the pixels of the target image block can be updated according to the pixels of the matching block of the target image block to obtain the updated target image block. By analogy, each updated image block can be obtained, and each updated image block is combined according to the original position to obtain the smoothed third image data. For example, the pixel value of the target image block can be replaced by the average value or weighted average value of the pixel values ​​of the matching block pixels of the target image block, wherein the weighted weight corresponding to the weighted average value refers to the weight corresponding to the matching block, and the weights corresponding to different matching blocks can be determined according to the similarity. The higher the similarity, the greater the weight of the corresponding matching block. In an actual application scenario, after obtaining each updated image block, each updated image block may be subjected to inter-block smoothing processing to obtain smoothed third image data, which can reduce the abruptness of the block boundary.

[0103] In this embodiment, the third image data is processed into blocks to obtain multiple image blocks, image block features of each image block are extracted, similarities between the image blocks are calculated based on the image block features, each image block is matched based on the similarities between the image blocks to obtain matching blocks, and smoothed third image data is obtained based on the matching blocks, so that image smoothing is achieved by image block feature matching, which can improve image smoothing efficiency.

[0104] In some embodiments, the field of view corresponding to the multispectral sensor is greater than the field of view corresponding to the color sensor.

[0105] Among them, the first original domain image data obtained by the multispectral sensor is subjected to a first preprocessing to obtain the first image data, and the second original domain image data obtained by the color sensor is subjected to a second preprocessing to obtain the second image data, and then the first image data is aligned with the second image data, which is equivalent to aligning the image data obtained by the multispectral sensor with the image data obtained by the color sensor. Therefore, the cropped image data can be aligned with the image data obtained by the color sensor only when the field of view corresponding to the multispectral sensor is sufficiently cropped, that is, the field of view corresponding to the multispectral sensor needs to be larger than the field of view corresponding to the color sensor.

[0106] In actual application scenarios, electronic devices are often equipped with multiple cameras of different focal lengths, such as a telephoto camera, an ultra-telephoto camera, a wide-angle camera, an ultra-wide-angle camera, etc. Each camera corresponds to a color sensor. In order to adapt to color sensors of different focal lengths, the field of view corresponding to the multispectral sensor needs to be larger than the maximum field of view of the cameras corresponding to the multiple color sensors. For example, if the electronic device includes an ultra-wide-angle camera, a wide-angle camera, and a telephoto camera, the field of view corresponding to the configured multispectral sensor is larger than the field of view corresponding to the ultra-wide-angle camera.

[0107] In one example, when calibrating a camera of an electronic device, the target area of ​​the multispectral sensor and each color sensor can be determined. In the process of aligning the first image data with the second image data, the first image data corresponding to the multispectral sensor can be cropped and geometrically deformed, and then the motion information between the first image data and the second image data is calculated and motion compensation is performed based on the motion information to achieve alignment of the first image data with the second image data in the target area. For example, the schematic diagram of the field of view corresponding to the multispectral sensor and the field of view corresponding to the color sensor is shown in FIG. Figure 2 As shown, the field of view angle 202 corresponding to the multi-spectral sensor is greater than the field of view angle 204 corresponding to the color sensor.

[0108] In this embodiment, the field of view angle corresponding to the multispectral sensor is greater than the field of view angle corresponding to the color sensor, which can ensure that the image data obtained by the multispectral sensor has sufficient cropping space and has a large room for alignment with the image data obtained by the color sensor, thereby facilitating the alignment between the first image data and the second image data and improving the alignment accuracy.

[0109] In one example, the flowchart of the image processing method is as follows: Figure 3 As shown, the image signal processor of the electronic device processes the image data in the first stage, the second stage and the third stage. The electronic device performs the first stage of processing on the first original domain image data 302 acquired by the multispectral sensor to obtain intermediate image data, and performs spectral reconstruction on the intermediate image data to obtain the first image data. The electronic device performs the first stage and the second stage of processing on the second original domain image data 304 acquired by the color sensor in sequence to obtain the second image data. The electronic device aligns the first image data with the second image data to obtain the third image data corresponding to the first image data, performs smoothing processing on the third image data to obtain the smoothed third image data, and performs the third stage of processing on the second image data according to the smoothed third image data to obtain the target image.

[0110] For example, the processing flow of the image processing method is described by taking the color sensor as an RGB sensor. The raw domain data collected and output by the multispectral sensor is subjected to the raw preprocessing (raw domain processing) corresponding to the first stage in the ISP chip, that is, the OB, lens shading correction, raw domain denoising and other processing are deducted to obtain the image data to be reconstructed; after the raw domain processing is completed, the image data to be reconstructed is sent to the multispectral algorithm unit for light source estimation, spectrum reconstruction and other processing to obtain the first image data, and the first image data is converted from the xyz color space to the RGB color space and sent to the CVP (Compute Vision Processor, computer vision processing) module of the ISP chip. Through optical flow estimation, geometric transformation and other processing, the result of the spectral sensor is aligned with the result of the RGB sensor. The aligned result is passed through the color restoration module (colorrendition) for smoothing and chromaticity edge and other post-processing (post refine), and the post-processing result is sent to the third stage of the ISP chip for processing to assist in color restoration processing of the image data collected by the RGB sensor. The processing flow for the image data collected by the RGB sensor includes: performing the first and second stage processing of the ISP chip on the raw domain data collected by the RGB sensor to obtain the second image data. That is, the result of the above post-processing is sent to the third stage processing flow of the ISP chip, and the second image data is subjected to color restoration processing according to the result of the post-processing, and the target image after color restoration is output.

[0111] For example, the function of an intelligent camera (AI-Camera) can be realized by an artificial intelligence algorithm. The AI-Camera can identify scene information of the shooting scene based on the second image data, and the electronic device can assist in spectral reconstruction and color restoration and other processing in combination with the scene information. Figure 4 As shown, AI-Camera identifies the scene information corresponding to the second image data, and can perform spectral reconstruction on the image data to be reconstructed to obtain the first image data in combination with the identified scene information, and can also perform image smoothing in color restoration processing in combination with the scene information.

[0112] In another example, the flowchart of the image processing method is as follows Figure 5As shown, the image signal processor of the electronic device processes the image data in the original domain and in the YUV domain. The electronic device performs original domain processing on the first original domain image data 502 acquired by the multispectral sensor to obtain image data to be reconstructed, and performs spectral reconstruction on the image data to be reconstructed to obtain first image data. The electronic device performs original domain processing on the second original domain image data 504 acquired by the color sensor to obtain second image data. The electronic device aligns the first image data with the second image data to obtain third image data corresponding to the first image data, performs image smoothing processing on the third image data to obtain smoothed third image data, and performs YUV domain processing on the second image data according to the smoothed third image data to obtain the target image.

[0113] In an actual application scenario, a partitioned multispectral device can be connected to the ISP chip of the currently used color sensor imaging, and the image data obtained by the multispectral sensor can be aligned with the image data obtained by the color sensor through a computer vision processor. That is, the spectral information obtained by the multispectral sensor is used to assist the image data obtained by the color sensor in color restoration to achieve color restoration accuracy.

[0114] In the above embodiment, the image data obtained by the color sensor is subjected to color restoration processing through the spectral information obtained by the multispectral sensor, which can fully combine the advantages of the partitioned multispectral sensor and the color sensor and improve the color restoration accuracy; in addition, in terms of hardware, the imaging link of the multispectral sensor can be introduced into the existing ISP chip platform for color sensor imaging, and different ISP chip platforms can be compatible to achieve an effective combination of color sensors and multispectral sensors, give play to their respective advantages, improve the color restoration effect of camera imaging, and can adapt to sensors of different focal lengths in the camera to achieve color restoration effects at all focal lengths.

[0115] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0116] Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations on the image processing method above, and will not be repeated here.

[0117] In an exemplary embodiment, Figure 6 As shown, an image processing device is provided, comprising: a first image data processing module 602, a second image data processing module 604, an image data alignment module 606 and a color restoration processing module 608, wherein:

[0118] A first image data processing module 602 is used to perform a first preprocessing on the first original domain image data acquired by the multispectral sensor to obtain first image data;

[0119] A second image data processing module 604 is used to perform a second preprocessing on the second original domain image data acquired by the color sensor to obtain second image data;

[0120] An image data alignment module 606, configured to align the first image data with the second image data to obtain third image data corresponding to the first image data;

[0121] The color restoration processing module 608 is used to perform color restoration processing on the second image data according to the third image data to obtain a target image.

[0122] In some embodiments, the image signal processor of the electronic device processes image data in stages including a first stage, a second stage, and a third stage; the first image data processing module 602 is also used to perform the first stage of processing on the first original domain image data to obtain intermediate image data; and perform spectral reconstruction on the intermediate image data to obtain first image data.

[0123] In some embodiments, the image signal processor of the electronic device processes image data in stages including a first stage, a second stage, and a third stage; the second image data processing module 604 is also used to sequentially perform the first stage and the second stage on the second original domain image data to obtain second image data.

[0124] In some embodiments, the image signal processor of the electronic device processes image data in the process of raw domain processing and YUV domain processing; the first image data processing module 602 is also used to perform raw domain processing on the first raw domain image data to obtain image data to be reconstructed; and perform spectral reconstruction on the image data to be reconstructed to obtain first image data.

[0125] In some embodiments, the first image data processing module 602 is further used to identify scene information of the shooting scene based on the second image data through an artificial intelligence algorithm; and perform spectral reconstruction on the image data to be reconstructed in combination with the scene information to obtain the first image data.

[0126] In some embodiments, the image signal processor of the electronic device processes image data in a process including raw domain processing and YUV domain processing; the second image data processing module 604 is further used to perform raw domain processing on the second raw domain image data to obtain second image data.

[0127] In some embodiments, the image data alignment module 606 is also used to calculate motion information between pixels at corresponding positions in the first image data and the second image data through a computer vision processor; and to perform motion compensation on the first image data according to the motion information through the computer vision processor to obtain third image data corresponding to the first image data.

[0128] In some embodiments, the image signal processor of the electronic device processes image data in stages including a first stage, a second stage, and a third stage; the color restoration processing module 608 is also used to smooth the third image data to obtain smoothed third image data; and the second image data is processed in the third stage according to the smoothed third image data to obtain a target image.

[0129] In some embodiments, the image signal processor of the electronic device processes image data in the process of raw domain processing and YUV domain processing; the color restoration processing module 608 is also used to smooth the third image data to obtain smoothed third image data; and perform YUV domain processing on the second image data according to the smoothed third image data to obtain a target image.

[0130] In some embodiments, the color restoration processing module 608 is further used to identify scene information of the shooting scene based on the second image data through an artificial intelligence algorithm; and to smooth the third image data in combination with the scene information to obtain smoothed third image data.

[0131] In some embodiments, the color restoration processing module 608 is further used to perform color estimation on pixels in the third image data to obtain pixel color estimation values; and obtain smoothed third image data according to the pixel color estimation values ​​of each pixel in the third image data.

[0132] In some embodiments, the color restoration processing module 608 is also used to perform block processing on the third image data to obtain multiple image blocks; perform feature extraction on each image block to obtain image block features; calculate the similarity between each image block based on the image block features; match each image block based on the similarity between the image blocks to obtain matching blocks; and obtain smoothed third image data based on the matching blocks.

[0133] Each module in the above-mentioned image processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in an electronic device in the form of hardware, or can be stored in a memory in an electronic device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0134] In an exemplary embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, an image processing method is implemented. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the electronic device casing, or an external keyboard, touchpad or mouse.

[0135] Those skilled in the art will understand that Figure 7 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0136] In an exemplary embodiment, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned image processing method embodiment when executing the computer program.

[0137] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned image processing method embodiment are implemented.

[0138] In some embodiments, a computer program product is provided, including a computer program, which implements the steps in the above-mentioned image processing method embodiment when executed by a processor.

[0139] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0140] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0141] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0142] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the electronic equipment includes a multi-spectral sensor and a color sensor; the method includes: Performing a first preprocessing on the first original domain image data acquired by the multispectral sensor to obtain first image data; Performing a second preprocessing on the second original domain image data acquired by the color sensor to obtain second image data; Aligning the first image data with the second image data to obtain third image data corresponding to the first image data; The second image data is subjected to color restoration processing according to the third image data to obtain a target image.

2. The method according to claim 1, characterized in that The image signal processor of the electronic device processes the image data in stages including a first stage, a second stage and a third stage; the first preprocessing of the first original domain image data acquired by the multispectral sensor to obtain the first image data includes: Performing the first stage of processing on the first original domain image data to obtain intermediate image data; Perform spectral reconstruction on the intermediate image data to obtain first image data.

3. The method according to claim 1 or 2, characterized in that: The image signal processor of the electronic device processes the image data in stages including a first stage, a second stage and a third stage; the second preprocessing of the second original domain image data acquired by the color sensor to obtain the second image data includes: The second original domain image data is processed in the first stage and the second stage in sequence to obtain second image data.

4. The method according to claim 1, characterized in that: The image signal processor of the electronic device processes the image data in a process including raw domain processing and YUV domain processing; the first preprocessing of the first raw domain image data acquired by the multispectral sensor to obtain the first image data includes: Performing the original domain processing on the first original domain image data to obtain image data to be reconstructed; Perform spectral reconstruction on the image data to be reconstructed to obtain first image data.

5. The method according to claim 4, characterized in that The performing spectral reconstruction on the image data to be reconstructed to obtain first image data includes: identifying scene information of the shooting scene based on the second image data by an artificial intelligence algorithm; In combination with the scene information, spectral reconstruction is performed on the image data to be reconstructed to obtain first image data.

6. The method according to claim 1, 4 or 5, characterized in that: The image signal processor of the electronic device processes the image data in a process including raw domain processing and YUV domain processing; the second preprocessing of the second raw domain image data acquired by the color sensor to obtain the second image data includes: The second original domain image data is subjected to the original domain processing to obtain second image data.

7. The method according to claim 1, characterized in that The aligning the first image data with the second image data to obtain third image data corresponding to the first image data includes: Calculate motion information between pixels at corresponding positions in the first image data and the second image data by a computer vision processor; Performing motion compensation on the first image data according to the motion information through a computer vision processor to obtain third image data corresponding to the first image data.

8. The method according to claim 1, characterized in that: The image signal processor of the electronic device processes the image data in stages including a first stage, a second stage and a third stage; performing color restoration processing on the second image data according to the third image data to obtain a target image includes: performing smoothing processing on the third image data to obtain smoothed third image data; The third stage of processing is performed on the second image data according to the smoothed third image data to obtain a target image.

9. The method according to claim 1, characterized in that: The image signal processor of the electronic device processes the image data in a process including raw domain processing and YUV domain processing; and performing color restoration processing on the second image data according to the third image data to obtain a target image, including: performing smoothing processing on the third image data to obtain smoothed third image data; The YUV domain processing is performed on the second image data according to the smoothed third image data to obtain a target image.

10. The method according to claim 8 or 9, characterized in that: The step of smoothing the third image data to obtain smoothed third image data includes: identifying scene information of the shooting scene based on the second image data by an artificial intelligence algorithm; In combination with the scene information, the third image data is smoothed to obtain smoothed third image data.

11. The method according to claim 8 or 9, characterized in that: The step of smoothing the third image data to obtain smoothed third image data includes: Performing color estimation on pixels in the third image data to obtain pixel color estimation values; The smoothed third image data is obtained according to the pixel color estimation value of each pixel in the third image data.

12. The method according to claim 8 or 9, characterized in that: The step of smoothing the third image data to obtain smoothed third image data includes: Performing block processing on the third image data to obtain a plurality of image blocks; Performing feature extraction on each of the image blocks to obtain image block features; Calculate the similarity between each image block according to the image block features; Matching each image block according to the similarity between the image blocks to obtain a matching block; According to the matching block, smoothed third image data is obtained.

13. The method according to claim 1, characterized in that The field of view angle corresponding to the multispectral sensor is greater than the field of view angle corresponding to the color sensor.

14. An image processing device, characterized in that: The device comprises: A first image data processing module, used for performing a first preprocessing on the first original domain image data acquired by the multispectral sensor to obtain first image data; A second image data processing module, used for performing a second preprocessing on the second original domain image data acquired by the color sensor to obtain second image data; An image data alignment module, used for aligning the first image data with the second image data to obtain third image data corresponding to the first image data; The color restoration processing module is used to perform color restoration processing on the second image data according to the third image data to obtain a target image.

15. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 13 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images