Image processing method and electronic equipment

Abstract

The invention discloses an image processing method and electronic equipment, and belongs to the technical field of camera shooting. The method comprises the steps that an RGB imaging sensor and a multispectral imaging sensor are controlled to shoot the same shooting scene, and a first image and a second image are obtained respectively; obtaining a comprehensive vector based on the first pixel value of the first image and the second pixel value of the second image; obtaining a target pixel value based on the comprehensive vector and a preset calibration matrix; wherein the preset calibration matrix is obtained based on an RGB calibration image shot by an RGB imaging sensor and a multispectral calibration image shot by a multispectral imaging sensor; and generating a target image based on the target pixel value.

Description

Technical Field

[0001] This application belongs to the field of camera technology, specifically relating to an image processing method and an electronic device. Background Technology

[0002] Currently, with the continuous development of electronic device technology, more and more electronic devices are equipped with imaging devices, enabling users to capture images. However, as users' demand for high-quality images increases, electronic devices equipped with a single imaging device often struggle to meet user needs. Therefore, more and more electronic devices are incorporating multiple imaging devices simultaneously, such as RGB imaging sensors and multispectral imaging sensors, to acquire both RGB and multispectral images.

[0003] Accordingly, how to ensure more accurate color information when generating target images based on RGB and multispectral images has become a technical problem that needs to be solved when taking pictures. Summary of the Invention

[0004] The purpose of this application is to provide an image processing method and electronic device that can obtain more accurate color information.

[0005] In a first aspect, embodiments of this application provide an image processing method, the method comprising:

[0006] The RGB imaging sensor and the multispectral imaging sensor are controlled to capture images of the same scene, resulting in a first image and a second image, respectively.

[0007] A comprehensive vector is obtained based on the first pixel value of the first image and the second pixel value of the second image;

[0008] The target pixel value is obtained based on the comprehensive vector and the preset calibration matrix; wherein, the preset calibration matrix is ​​obtained based on the RGB calibration image captured by the RGB imaging sensor and the multispectral calibration image captured by the multispectral imaging sensor;

[0009] Generate a target image based on the target pixel values.

[0010] Secondly, embodiments of this application provide an image processing apparatus, the apparatus comprising:

[0011] The first control module is used to control the RGB imaging sensor and the multispectral imaging sensor to capture the same shooting scene, and obtain the first image and the second image respectively;

[0012] The first generation module is used to obtain a comprehensive vector based on the first pixel value of the first image and the second pixel value of the second image;

[0013] The second generation module is used to obtain the target pixel value based on the comprehensive vector and the preset calibration matrix; wherein, the preset calibration matrix is ​​obtained based on the RGB calibration image captured by the RGB imaging sensor and the multispectral calibration image captured by the multispectral imaging sensor;

[0014] The third generation module is used to generate a target image based on the target pixel values.

[0015] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores a program or instructions that can run on the processor, and the program or instructions, when executed by the processor, implement the steps of the above-described image processing method.

[0016] Fourthly, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the above-described image processing method.

[0017] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the above-described image processing method.

[0018] Sixthly, embodiments of this application provide a computer program product, which is stored in a storage medium and executed by at least one processor to implement the above-described image processing method.

[0019] In this embodiment, an RGB imaging sensor and a multispectral imaging sensor are controlled to capture images of the same scene, obtaining a first image and a second image respectively. A composite vector is obtained based on the first pixel value of the first image and the second pixel value of the second image. A target pixel value is obtained based on the composite vector and a preset calibration matrix. The preset calibration matrix is ​​obtained based on the RGB calibration image captured by the RGB imaging sensor and the multispectral calibration image captured by the multispectral imaging sensor. A target image is generated based on the target pixel value. Since the composite vector is generated based on the pixel values ​​of pixels in the first and second images, which are captured by the RGB imaging sensor and the multispectral imaging sensor respectively, the composite vector can simultaneously represent the color information collected by the RGB imaging sensor and the multispectral imaging sensor. This allows for more efficient use of the color information collected by both sensors during the generation of the target image, resulting in more accurate color information and improved image processing performance.

[0020] Meanwhile, in this embodiment, the target pixel value can be obtained by combining the comprehensive vector and the preset calibration matrix, and then the target image can be obtained based on the target pixel value. In this way, image processing efficiency can be ensured to a certain extent. Attached Figure Description

[0021] Figure 1 This is a flowchart of the steps of an image processing method provided in an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of a calibration image provided in an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of a calibration matrix generation process provided in an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of image segmentation provided in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of a processing flow provided in an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of an aligned image provided in an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of another processing flow provided in an embodiment of this application;

[0028] Figure 8 This is a block diagram of an image processing apparatus provided in an embodiment of this application;

[0029] Figure 9 This is one of the structural schematic diagrams of the electronic device provided in the embodiments of this application;

[0030] Figure 10 This is the second schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0032] The terms "first," "second," etc., used in this application's specification are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0033] This application provides an image processing method applicable to electronic devices, such as mobile phones, computers, and cameras. These devices include RGB imaging sensors and multispectral imaging sensors. In practical applications, RGB imaging sensors offer higher spatial resolution, capturing clearer image details. However, the spectral response of RGB sensors differs significantly from the human eye's photosensitive spectral response, resulting in lower color accuracy and less precise color reproduction. Multispectral imaging sensors, by coating a monochrome array, increase spectral channels and optimize the spectral response, enabling the capture of multiple spectral bands and achieving higher spectral resolution. Furthermore, the spectral response of multispectral imaging sensors is closer to that of the human eye, resulting in higher color accuracy. However, the spatial resolution of multispectral imaging sensors is often lower than that of RGB imaging sensors, resulting in a lower spatial resolution.

[0034] Accordingly, embodiments of this application use a multispectral imaging sensor to assist the RGB imaging sensor in obtaining higher quality images. To this end, embodiments of this application provide an image processing method to obtain more accurate color information while generating a target image based on both RGB and multispectral images, ensuring effective image processing.

[0035] The image processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0036] Figure 1 This is a flowchart of the steps of an image processing method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes:

[0037] Step 101: Control the RGB imaging sensor and the multispectral imaging sensor to capture the same scene, and obtain the first image and the second image respectively.

[0038] The first image is an RGB image captured using an RGB imaging sensor, and the second image is a multispectral image captured using a multispectral imaging sensor. Both images are obtained by simultaneously capturing the same scene using the RGB and multispectral imaging sensors. Specifically, during the actual shooting process, in response to a shooting command, the RGB imaging sensor captures an image to obtain an RGB image, and the multispectral imaging sensor captures an image to obtain a multispectral image. The captured RGB and multispectral images are used as the first and second images, respectively. The shooting command can be user-triggered; for example, a user can trigger the shooting command by clicking a shooting button.

[0039] Furthermore, during the capture of the first image, after the RGB imaging sensor acquires the raw image information, it undergoes de-mosaic processing to obtain the final RGB image. In this way, each pixel in the RGB image corresponds to linear RGB information, meaning each pixel has channel values ​​for the three color channels: R, G, or B. Correspondingly, during the capture of the second image, after the multispectral imaging sensor acquires the raw image information, it undergoes de-mosaic processing to obtain the final multispectral image. Each pixel in the multispectral image corresponds to spectral information, meaning each pixel has channel values ​​for multiple spectral channels, facilitating processing. Here, a spectral channel refers to a spectral response channel. The number of spectral channels is denoted by k, and the specific value of k is determined by the hardware design of the multispectral imaging sensor. For example, assuming the multispectral imaging sensor responds to wavelengths of 400nm, 410nm, 420nm…700nm, then k is 31.

[0040] Step 102: Obtain a composite vector based on the first pixel value of the first image and the second pixel value of the second image.

[0041] In this embodiment, the first pixel refers to any pixel in the first image; specifically, a pixel in the first image can be considered as a first pixel. The second pixel refers to any pixel in the second image; specifically, a pixel in the second image can be considered as a second pixel. The channel values ​​of the first pixel include the channel values ​​of the R, G, and B color channels of the first pixel, and the channel values ​​of the second pixel include the channel values ​​of multiple spectral channels of the second pixel. The first pixel value includes the pixel values ​​of each color channel of the first pixel, and the second pixel value includes the pixel values ​​of each color channel of the second pixel. In this step, the pixel values ​​of each color channel of the first pixel in the RGB image and the pixel values ​​of each color channel of the second pixel in the multispectral image can be combined to generate a vector as a comprehensive vector. The color channels of the first pixel can be the R, G, and B channels, and the pixel values ​​of each color channel of the first pixel are the respective channel values ​​of the R, G, and B channels. The color channels of the second pixel can be the aforementioned k spectral channels, and the pixel values ​​of each color channel of the second pixel are the respective channel values ​​of the k spectral channels. This comprehensive vector can represent the channel values ​​of the first pixel and the corresponding second pixel.

[0042] Step 103: Obtain the target pixel value based on the comprehensive vector and the preset calibration matrix; wherein, the preset calibration matrix is ​​obtained based on the RGB calibration image captured by the RGB imaging sensor and the multispectral calibration image captured by the multispectral imaging sensor.

[0043] In this embodiment, the preset calibration matrix can be a matrix pre-generated and stored in the electronic device. For example, a preset calibration matrix can be pre-set for the electronic device before it leaves the factory. This preset calibration matrix is ​​used to correct the first pixel based on the channel value represented by the comprehensive vector, thereby obtaining the target pixel value corresponding to the first pixel. Specifically, matrix operations can be performed based on the comprehensive vector corresponding to the first pixel and the preset calibration matrix to obtain the target pixel value corresponding to the first pixel, thereby achieving color correction for the first pixel. The RGB calibration image and multispectral calibration image are pre-captured using the aforementioned RGB and multispectral devices. The preset calibration matrix is ​​generated based on the actual pixel value and pixel prediction value corresponding to the pixel in the RGB and multispectral calibration images. Adjustment is made through the preset calibration matrix to generate pixel prediction values ​​that are closer to the actual pixel values. This ensures that accurate target pixel values ​​can be generated during calibration based on the preset calibration matrix and the comprehensive vector, thereby ensuring image processing efficiency.

[0044] Step 104: Generate a target image based on the target pixel values.

[0045] The target image can be an image in the RGB color space, and the target pixel value can also be called the target color value. The target pixel value can be the tristimulus value of the CIE XYZ color space corresponding to the first pixel. The target pixel value can include the stimulus values ​​of the three dimensions of X, Y, and Z.

[0046] In this embodiment, an RGB imaging sensor and a multispectral imaging sensor are controlled to capture images of the same scene, obtaining a first image and a second image respectively. A composite vector is obtained based on the first pixel value of the first image and the second pixel value of the second image. A target pixel value is obtained based on the composite vector and a preset calibration matrix. The preset calibration matrix is ​​obtained based on the RGB calibration image captured by the RGB imaging sensor and the multispectral calibration image captured by the multispectral imaging sensor. A target image is generated based on the target pixel value. Since the composite vector is generated based on the pixel values ​​of pixels in the first and second images, which are captured by the RGB imaging sensor and the multispectral imaging sensor respectively, the composite vector can simultaneously represent the color information collected by the RGB imaging sensor and the multispectral imaging sensor. This allows for more efficient use of the color information collected by both sensors during the generation of the target image, resulting in more accurate color information and improved image processing performance.

[0047] Meanwhile, in this embodiment, the target pixel value can be obtained by combining the comprehensive vector and the preset calibration matrix, and then the target image can be obtained based on the target pixel value. In this way, image processing efficiency can be ensured to a certain extent.

[0048] Optionally, prior to step 103 above, the embodiments of this application may further include the following steps:

[0049] Step S21: Control the RGB imaging sensor to capture an image of the color chart to obtain an RGB calibration image, and control the multispectral imaging sensor to capture an image of the color chart to obtain a multispectral calibration image.

[0050] Step S22: Segment the RGB calibration image and the multispectral calibration image to obtain color patch region segmentation pairs.

[0051] Step S23: Generate a calibration composite vector based on the pixel values ​​of the pixels in the color block region segmentation pair.

[0052] Step S24: Generate pixel prediction values ​​for the color patch region segmentation pairs based on the calibration synthesis vector and the initial matrix.

[0053] Step S25: Based on the difference information between the pixel prediction value and the actual pixel value of the corresponding color block, the initial matrix is ​​iteratively optimized to obtain the preset calibration matrix.

[0054] In this embodiment, a preset calibration matrix can be predetermined in the color calibration process of the multispectral imaging sensor and the RGB imaging sensor. The execution subject of the above steps S21 to S25 can be the electronic device itself or other devices, and this embodiment of the invention does not limit this.

[0055] Specifically, the RGB calibration image and multispectral calibration image are obtained by simultaneously capturing images of a color chart using an RGB imaging sensor and a multispectral imaging sensor. The color chart, as the standard subject, includes multiple color patches. The number of color patches can be set as needed; for example, the number of color patches can be greater than or equal to k+3, where 3 represents the number of color channels in the RGB image. Correspondingly, both the RGB calibration image and the multispectral calibration image include the image region corresponding to each color patch. During the capture of the RGB calibration image, after the RGB imaging sensor acquires the original image information of the color chart, the original image information is de-mosaiced to obtain the final RGB calibration image. Each pixel in the RGB calibration image corresponds to linear RGB information. Similarly, during the capture of the multispectral calibration image, after the multispectral imaging sensor acquires the original image information of the color chart, the original image information is de-mosaiced to obtain the final multispectral calibration image. Each pixel in the multispectral image corresponds to spectral information, facilitating subsequent processing.

[0056] Color patch region segmentation pairs consist of image regions corresponding to the same color patch in both the RGB calibration image and the multispectral calibration image. The RGB calibration image can be divided into multiple image regions corresponding to each color patch, and the multispectral calibration image can also be divided into multiple image regions corresponding to each color patch. Image regions corresponding to the same color patch in both the RGB and multispectral calibration images form a color patch region segmentation pair for that color patch. Figure 2 This is a schematic diagram of a calibration image provided in an embodiment of this application, such as... Figure 2 As shown, assuming the color chart includes 6 color patches, the corresponding image regions in both the RGB calibration image and the multispectral calibration image are 6 color patches. These 6 color patches are denoted as color patch 1 to color patch 6. Figure 2 In the image, b1 to b6 represent the image regions corresponding to color patches 1 to 6 in the RGB calibration image. Figure 2In this context, a1 to a6 represent the image regions corresponding to color patches 1 to 6 in the multispectral calibration image. The image region corresponding to color patch j in the RGB calibration image and the image region corresponding to color patch j in the multispectral calibration image form a color patch region segmentation pair for color patch j. Here, j∈[1, m], and m represents the total number of color patches. In this example, m = 6.

[0057] Further, a calibration synthesis vector is generated based on the pixel values ​​of the pixels in the two image regions of the color patch region segmentation pair corresponding to color patch j. This calibration synthesis vector represents the pixel values ​​of the pixels in these two image regions. Next, matrix operations are performed between the calibration synthesis vector of the color patch and the initial matrix to obtain the pixel prediction value of the color patch region segmentation pair corresponding to the color patch. Based on the actual pixel value corresponding to the color patch and the pixel prediction value of the color patch region segmentation pair corresponding to the color patch, the difference between the pixel prediction value and the actual pixel value is determined, and the initial matrix is ​​iteratively optimized accordingly. After completing the iteration operation, the initial matrix is ​​determined as the preset calibration matrix and recorded in the electronic device. The actual pixel value and the pixel prediction value can be referred to as the actual color value and the color prediction value, respectively. Both the actual pixel value and the pixel prediction value are tristimulus values ​​in the CIEXYZ color space. The actual pixel value corresponding to each color patch can be pre-measured manually; that is, the actual pixel value is the measured value. A color measurement device can be used to measure the CIE tristimulus value of each color patch in the color chart, which is used as the actual pixel value corresponding to each color patch, thus ensuring the accuracy of the actual pixel value. Here, the actual pixel value corresponding to the color block is the actual pixel value corresponding to the color block region segmentation pair of that color block. For example, for the j-th color block, the actual pixel value corresponding to color block j can be represented as (X... j Y j Z j ).

[0058] In this embodiment, an RGB imaging sensor and a multispectral imaging sensor are controlled to capture images of a color chart, resulting in two calibration images. A calibration composite vector is generated based on the pixel values ​​of pixels in the color patch region segmentation pairs for each color patch in these two calibration images. Based on the calibration composite vector and an initial matrix, pixel prediction values ​​for the color patch region segmentation pairs are generated. The initial matrix is ​​then iteratively optimized based on the difference between the actual pixel values ​​corresponding to each color patch and the pixel prediction values ​​of the color patch region segmentation pairs for each color patch, resulting in a preset calibration matrix. The difference information can be determined based on the actual pixel values ​​corresponding to each color patch and the pixel prediction values ​​of the color patch region segmentation pairs for each color patch, and is used to characterize the difference between the pixel prediction values ​​and the actual pixel values.

[0059] In this embodiment of the application, during the generation of the preset calibration matrix, the initial matrix is ​​iteratively updated based on the difference information between the pixel prediction value of the color block region segmentation pair of each color block and the actual pixel value corresponding to each color block. This allows the calibration matrix obtained after iterative optimization to accurately measure the difference between the pixel prediction value and the actual pixel value, thereby ensuring the image processing effect of the subsequent use of the calibration matrix to a certain extent.

[0060] Furthermore, by segmenting color regions into blocks and determining the predicted pixel values ​​for subsequent processing, the processing load during the generation of the preset calibration matrix can be reduced to some extent, thereby improving processing efficiency. Simultaneously, using color blocks as units facilitates the measurement and acquisition of actual pixel values. Moreover, by using the electronic device's own RGB and multispectral imaging sensors to acquire RGB and multispectral calibration images and generating the preset calibration matrix, the final generated preset calibration matrix can be ensured to be more compatible with the electronic device, thus improving the processing effect when subsequently using the preset calibration matrix to generate the target image.

[0061] Optionally, step S23 may specifically include:

[0062] Step S231: Based on the pixel values ​​of each color channel of the pixel in the color block region segmentation, determine the statistical values ​​corresponding to each color channel to obtain the calibration components of each color channel.

[0063] Step S232: Based on the calibration components of each color channel, obtain the calibration composite vector.

[0064] Step S233: Concatenate the calibration component corresponding to the first region with the calibration component corresponding to the second region to obtain the calibration composite vector.

[0065] In this embodiment, scene segmentation can be performed on the RGB calibration image and the multispectral calibration image according to color blocks. For example, pixels of the same color in the RGB calibration image can be grouped into the same image region to obtain an image region corresponding to a color block. Here, one image region in the RGB calibration image is considered a first region. The RGB calibration image is divided into multiple first regions. Similarly, pixels of the same color in the multispectral calibration image can be grouped into the same image region to obtain an image region corresponding to a color block. The multispectral calibration image is divided into multiple second regions. Here, one image region in the multispectral calibration image is considered a second region. The six image regions in the RGB calibration image can be respectively designated as first region 1 to first region 6, and the six image regions in the multispectral calibration image can be respectively designated as second region 1 to second region 6. Referring to the above... Figure 2 Region 1 to Region 6 are... Figure 2b1~b6 in the middle, and the second region 1~second region 6 are the... Figure 2 In the diagram, a1 to a6. Color block region segmentation pairs can also be called color block segmentation pairs. The color block region segmentation pair corresponding to color block j includes the first region j and the second region j.

[0066] For color patch j, the statistical values ​​corresponding to each color channel can be determined based on the pixel values ​​of all pixels in the first region j, thus obtaining the calibration components of the RGB color channels in the first region j, i.e., the calibration components corresponding to the first region j. Similarly, based on the pixel values ​​of all pixels in the second region j, the statistical values ​​corresponding to each color channel can be determined, thus obtaining the calibration components of the k color channels (i.e., spectral channels) in the second region j, i.e., the calibration components corresponding to the second region j. The calibration components corresponding to the first region j and the calibration components corresponding to the second region j together constitute the calibration components corresponding to the color patch region segmentation pair based on color patch j.

[0067] In the first region, the color channels of the pixels can include R, G, and B channels, and the pixel value of each color channel of the pixel in the first region is the channel value of each of the R, G, and B channels. In the second region, the color channels of the pixels can include k spectral channels, and the pixel value of each color channel of the pixel in the second region is the channel value of each of the k spectral channels.

[0068] Specifically, for the first region j, three statistical values ​​can be obtained, which can be called RGB statistical values. For the second region j, k statistical values ​​can be obtained, which can be called spectral statistical values. That is, for any first region, three statistical values ​​are obtained, and for any second region, k statistical values ​​are obtained. Correspondingly, for any color patch region segmentation pair, k+3 statistical values ​​are obtained.

[0069] For any color channel in the first region j, the statistical value corresponding to that color channel refers to a statistical value obtained by statistically analyzing the pixel values ​​(i.e., channel values) of all pixels in the first region j corresponding to that color channel according to a preset statistical method. The preset statistical method can be set as needed, and this embodiment does not limit it. For example, the preset statistical method can be average statistics, median statistics, or variance statistics, etc. Assuming the first region j includes 900 pixels, for the color channel R, the average value of the R channel values ​​of these 900 pixels can be calculated as the statistical value corresponding to the R channel. The statistical values ​​corresponding to the three color channels of the first region j can be represented as I_rgb. i,j Where i is an integer greater than 0 and less than or equal to 3, that is, the statistical value corresponding to the 3 color channels is represented as I_rgb. 1,j 、I_rgb 2,j ,I_rgb3,j

[0070] For any color channel in the second region j, the statistical value corresponding to that color channel refers to a statistical value obtained by statistically analyzing the pixel values ​​(i.e., channel values, such as light intensity values) of all pixels in the second region j corresponding to that color channel according to a preset statistical method. Assuming the second region j contains 900 pixels, for color channel 1, the average light intensity value of spectral channel 1 for these 900 pixels can be calculated as the statistical value corresponding to spectral channel 1. The statistical values ​​corresponding to the k color channels of the second region j can be represented as I_spec. i,j Where i is an integer greater than 0 and less than or equal to k, that is, the statistical value corresponding to each of the k color channels is represented as I_spec. 1,j I_spec 2,j ...I_spec k,j That is, the calibration component corresponding to the first region j of color block j is (I_rgb). 1,j 、I_rgb 2,j ,I_rgb 3,j The calibration component corresponding to the second region j of color block j is (I_spec). 1,j I_spec 2,j ...I_spec k,j ).

[0071] The calibration components can be concatenated as column vectors, following the order of the calibration components corresponding to the second region and the calibration components corresponding to the first region, to obtain the calibration composite vector. For example, the calibration composite vector can be represented as:

[0072]

[0073] In this embodiment, statistical values ​​for each color channel are determined based on the pixel values ​​of the pixels in each color channel of the pixel segmented by the color block region, thus obtaining the calibration components for each color channel. Based on the calibration components of each color channel, a calibration composite vector can be obtained, which to a certain extent ensures the efficiency of calibration composite vector generation.

[0074] Optionally, step S24 may specifically include: step S24a, multiplying the calibration synthesis vector and the initial matrix to obtain the pixel prediction value of the color block region segmentation pair.

[0075] In this embodiment, the initial matrix is ​​a 3×(k+3) matrix, and the initial values ​​of each element in the initial matrix can be arbitrarily set. For example, a random algorithm can be used to generate initial values ​​for each element in the initial matrix. By iteratively optimizing the initial matrix, the optimal solution for each element is obtained.

[0076] Specifically, the calibration synthesis vector is in column vector form, representing a (k+3)×1 matrix. Let M denote the initial matrix. The pixel prediction value of the color patch region segmentation pair for color patch j can be obtained by left-multiplying matrix M by the calibration synthesis vector corresponding to color patch j.

[0077]

[0078] Among them, X j_p Y j_p and Z j_p This represents the pixel prediction value of the color block region segmentation pair for color block j.

[0079] In this embodiment, the pixel prediction value of the color block region segmentation pair can be calculated by directly multiplying the calibration synthesis vector and the initial matrix, which has a high efficiency in determining the pixel prediction value.

[0080] Optionally, step S25 above may specifically include:

[0081] Step S251: Determine the overall color difference value based on the pixel prediction value of the color block region segmentation pair of each color block and the actual pixel value; the overall color difference value is used to characterize the difference information.

[0082] Step S252: If the overall color difference value does not meet the preset condition, adjust the element values ​​of the initial matrix and return to the step of generating the pixel prediction value of the color block region segmentation pair based on the calibration comprehensive vector and the initial matrix to continue execution until the overall color difference value meets the preset condition.

[0083] Specifically, in the optimization algorithm used in this application embodiment, for any color patch, the color difference value between the predicted pixel value of the color patch region segmentation pair and the actual pixel value of the color patch can be determined first. Then, based on the color difference values ​​of all color patches, the overall color difference value is determined. For example, let X... j ,Y j Z j Let j represent the actual pixel value of color block j. The overall color difference value can be calculated using the following loss function:

[0084]

[0085] Here, function f is used to calculate the color difference between the predicted pixel value and the actual pixel value of the color patch region segmentation pair for color patch j. Function g is used to calculate the overall color difference value based on the color difference values ​​of all color patches. The specific calculation methods of functions f and g can be set as needed. For example, function f can be used to calculate the absolute value of the difference between the predicted pixel value and the actual pixel value, and function g can be used to calculate the sum of the color difference values ​​of all color patches to obtain the overall color difference value. The larger the overall color difference value, the greater the gap represented by the difference information; the smaller the overall color difference value, the smaller the gap represented by the difference information.

[0086] Figure 3 This is a schematic diagram of a calibration matrix generation process provided in an embodiment of this application, such as... Figure 3 As shown, first, a multispectral calibration image and an RGB calibration image are acquired. Then, the multispectral calibration image is segmented into a second region, and the RGB calibration image is segmented into a first region. The calibration components corresponding to the second region and the first region are obtained. The actual pixel values ​​corresponding to each color patch are then obtained. Next, the calibration components corresponding to the first and second regions are concatenated to form a calibration composite vector. Finally, based on the calibration composite vector and the initial matrix, pixel prediction values ​​for the color patch region segments are generated. Based on the difference between the pixel prediction values ​​and the actual pixel values, the initial matrix is ​​iteratively optimized to obtain a preset calibration matrix.

[0087] Furthermore, the preset conditions can be pre-set. For example, the preset conditions could be that the overall color difference value is less than a preset threshold, or that the number of times the overall color difference value is calculated reaches a preset number. The overall color difference value is calculated once in each iteration; therefore, the number of calculations of the overall color difference value can represent the iteration rounds of the initial matrix. If the number of calculations of the overall color difference value reaches the preset number, or the overall color difference value is less than the preset threshold, then the loss function is considered to be at its minimum, the current overall color difference value is at its minimum, and the initial matrix has reached its optimal solution. Accordingly, optimization can be stopped, and the current initial matrix can be determined as the preset calibration matrix.

[0088] If the overall color difference value does not meet the preset condition, it can be considered that the initial matrix has not yet reached the optimal solution. Accordingly, the element values ​​of the initial matrix can be adjusted. Specifically, the element values ​​in the initial matrix can be changed through optimization algorithms. For example, the gradient value of the loss function can be calculated, and the adjustment magnitude can be determined based on the preset step size and the gradient value. Then, the adjustment magnitude is subtracted from the element values ​​in the initial matrix. After that, return to step S23 to continue execution, that is, start a new round of iteration.

[0089] In this embodiment, the overall color difference value is used to represent the difference information. The overall color difference value is determined based on the pixel prediction values ​​and actual pixel values ​​of the color block region segmentation pairs for each color block. If the overall color difference value does not meet the preset conditions, the element values ​​of the initial matrix are adjusted, and a new round of iteration is initiated until the overall color difference value meets the preset conditions. In this way, through multiple iterations, the element values ​​of the initial matrix are continuously optimized, ensuring the accuracy of the obtained preset calibration matrix.

[0090] Optionally, step 102 above may specifically include:

[0091] Step 1021: Take the pixel values ​​of each color channel of the first pixel as the first component, and determine the second component based on the pixel values ​​of each color channel of the second pixel corresponding to the first pixel.

[0092] Step 1022: Concatenate the first component and the second component to obtain the composite vector.

[0093] In this embodiment, for any pixel in the first image (i.e., any first pixel), a first component is obtained. The first component includes the pixel values ​​of the R, G, and B color channels of the first pixel, that is, the first component of a first pixel includes the RGB values ​​of the pixel in the first image. Let n represent the pixel index of the first pixel, and the value of n can be any pixel index included in the first image. The first component of the first pixel n can be represented as (I_rgb 1,n 、I_rgb 2,n ,I_rgb 3,n ).

[0094] The second component can include k values ​​determined based on the pixel values ​​of all second pixels corresponding to the first pixel. Correspondingly, these two components can be concatenated as column vectors in the order of second component minus first component to obtain a composite vector. The composite vector represents a matrix of size (k+3)×1.

[0095] In this embodiment, the pixel values ​​of each color channel of the first pixel are used as the first component, and the second component is determined based on the pixel values ​​of each color channel of the second pixel corresponding to the first pixel. Concatenating the first and second components yields the composite vector, which to some extent ensures the efficiency of composite vector generation.

[0096] Optionally, in one implementation, prior to step 1021 above, the embodiments of this application further include:

[0097] Step S31: Perform scene segmentation on the first image and the second image respectively to obtain corresponding scene region segmentation pairs.

[0098] The above-described step of concatenating the first component and the second component to obtain the composite vector may specifically include:

[0099] Step 1021a: Based on the scene region segmentation pair, the first component and the second component are concatenated to obtain the comprehensive vector.

[0100] Specifically, the scene region segmentation pair includes scene regions corresponding to the same scene in the first image and the second image. A preset semantic segmentation algorithm can be used to segment the scene in the first image and the second image respectively. The semantic segmentation algorithm can divide the scene according to preset rules; for example, it can divide it according to natural landscapes, such as dividing it into greenery, blue sky, etc. This application embodiment does not impose such limitations. A scene region in the first image is considered a first scene region, and a scene region in the second image is considered a second scene region. Since the first image and the second image are segmented using the same semantic segmentation algorithm, the scene regions included in the first image are the same as those included in the second image; that is, these multiple first scene regions correspond one-to-one with multiple second scene regions. Figure 4 This is a schematic diagram of image segmentation provided in an embodiment of this application, such as... Figure 4 As shown, both the first and second images are divided into two scene regions: a leaf region and a non-leaf region. The leaf region and non-leaf region in the second image are denoted as c1 and c2, respectively, while the leaf region and non-leaf region in the first image are denoted as d1 and d2, respectively. c1 and d1 correspond to each other, forming a scene region segmentation pair. Similarly, c2 and d2 correspond to each other, forming another scene region segmentation pair. This indicates that the first and second scene regions of the same scene correspond to each other.

[0101] For any first pixel, the first scene region to which the first pixel belongs is designated as the first target region, and the second scene region representing the same scene as the first target region is designated as the second target region. The first target region and the second target region form a scene region segmentation pair corresponding to the first pixel. The pixels included in the second target region are the second pixels corresponding to the first pixel. Next, based on the pixel values ​​of each color channel of all pixels in the second target region of the second image, statistical values ​​corresponding to each color channel are determined, resulting in k statistical values. These k statistical values ​​constitute the second component.

[0102] For any color channel in the second target region, the statistical value corresponding to that color channel refers to a statistical value obtained by statistically analyzing the pixel values ​​of all pixels in the second target region corresponding to that color channel according to a preset statistical method. Let p represent the second scene region corresponding to the first target region to which the first pixel n belongs (i.e., the second target region corresponding to the first pixel n). The statistical value determined based on the pixel values ​​of each color channel of all pixels in the second target region p can be represented as I_spec. i,p That is, the statistical values ​​corresponding to the k color channels in the second target region p are represented as I_spec 1,p I_spec 2,n ...I_spec k,p Correspondingly, the second component corresponding to the first pixel n can be represented as (I_spec). 1,p I_spec 2,n ...I_spec k,p The second component corresponding to the first pixel refers to the second component determined based on the pixel values ​​of each color channel of the second pixel corresponding to the first pixel.

[0103] Figure 5 This is a schematic diagram of a processing flow provided in an embodiment of this application, such as... Figure 5 As shown, a second image and a first image are acquired first. The second image is divided into a second scene region, and the first image is divided into a first scene region. For any first pixel, the pixel value of the color channel of the first pixel is obtained as the first component. Based on the pixel values ​​of the color channels of all pixels in the second target region of the second image, the statistical value corresponding to each color channel is determined as the second component. The first component and the second component are concatenated to obtain a comprehensive vector. The comprehensive vector and a preset calibration matrix are multiplied by matrix multiplication to obtain the target pixel value corresponding to the first pixel. Based on the target pixel values ​​corresponding to each first pixel, the target image is generated.

[0104] In practical applications, RGB imaging sensors employ a Bayer filter array (CFA), typically with each large pixel containing one red (R), one blue (B), and two green (G) subpixels (i.e., RGGB arrangement) to capture information in red, green, and blue. Multispectral sensors, however, to capture richer spectral information, use more channels, with at least one subpixel per spectral channel to capture light in a specific wavelength range. This means that for the same sensor area, a multispectral sensor needs to allocate more subpixels to capture different wavelengths, resulting in a decrease in spatial resolution for each wavelength. Therefore, the resolution of multispectral images is often lower than that of RGB images.

[0105] Since the resolution of multispectral images is lower than that of RGB images, this implementation does not require alignment. Scene segmentation is performed on the first and second images to obtain corresponding scene region segments. The second component is determined using all pixels in the second target region corresponding to the first target region to which the first pixel belongs in the scene region segmentation pair. This ensures that the second component is obtained for each first pixel, enabling the generation of target pixel values ​​for each first pixel subsequently, thus ensuring effective image processing.

[0106] Accordingly, the combined vector obtained by concatenation can be represented as:

[0107]

[0108] In this embodiment, a composite vector is generated for each pixel in the first image. In this implementation, the second component concatenated in the composite vectors corresponding to each first pixel within the same first target region is the same. Accordingly, after determining the second component for a first pixel within the first target region, this second component can be stored. When processing other first pixels within the first target region subsequently, the stored second component is directly read. This reduces computational load and improves processing efficiency.

[0109] Optionally, embodiments of this application also include:

[0110] Step S41: Align the second image with the first image pixel by pixel.

[0111] The step of determining the second component based on the pixel values ​​of each color channel of the second pixel corresponding to the first pixel may specifically include:

[0112] Step 1021b: Obtain the pixel values ​​of each color channel of the second pixel in the second image whose pixel number matches the target number, and use them as the second component; the target number is the pixel number of the first pixel.

[0113] In this embodiment, a super-resolution process can be performed on the second image based on a preset processing algorithm to align the pixels of the second image with those of the first image. The preset processing algorithm can be set as needed. For example, it can be a traditional algorithm or an artificial intelligence algorithm, such as bilinear interpolation, bicubic interpolation, or a convolutional neural network interpolation algorithm. This embodiment does not impose any limitations on this. Figure 6 This is a schematic diagram of an aligned image provided in an embodiment of this application, such as... Figure 6As shown, the first image and the second image have the same number of pixel rows and columns. By aligning the pixels of the second image with those of the first image, each first pixel corresponds to a second pixel in the second image. For example, the first pixel in the e-th row and f-th column of the first image corresponds to the second pixel in the e-th row and f-th column of the second image.

[0114] Furthermore, the pixel index can represent the position of the pixel in the image. The pixel index can be the row and column coordinates of the pixel; for example, the pixel index can be represented as (e, f). Alternatively, pixel indices can be assigned row by row in order from left to right and top to bottom. The pixel index of the first pixel can be used as the target index to obtain the pixel values ​​of each color channel of the second pixel in the second image with the target index, resulting in k pixel values. These k pixel values ​​constitute the second component. The pixel row and pixel column of the second pixel with the target index are the same as those of the first pixel. For example, assuming the first pixel n is located in row e and column f of the first image, the channel values ​​of each spectral channel of the second pixel located in row e and column f of the second image can be obtained, resulting in k channel values. These k channel values ​​are represented as I_spec. 1,n I_spec 2,n ...I_spec k,n That is, the second component corresponding to the first pixel n can be represented as (I_spec). 1,n I_spec 2,n ...I_spec k,n ).

[0115] Accordingly, the combined vector obtained by concatenation can be represented as:

[0116]

[0117] In this embodiment, a comprehensive vector is generated for each pixel in the first image. In this implementation, the vector is concatenated pixel-by-pixel; that is, the comprehensive vector corresponding to each first pixel is obtained by concatenating the first component of that first pixel with the second component determined based on the second pixel whose pixel number matches the target number. The second pixel whose pixel number matches the target number is the second pixel corresponding to that first pixel.

[0118] Figure 7 This is another processing flow diagram provided in the embodiments of this application, such as... Figure 7As shown, the process involves first acquiring a second image, aligning the second image with the first image pixel by pixel, and then acquiring the first image again. For any first pixel, the pixel values ​​of each color channel of the first pixel are obtained as the first component. The pixel values ​​of each color channel of the second pixel in the second image whose pixel number matches the target number are also obtained as the second component. The first and second components are concatenated to obtain a composite vector. The composite vector and a preset calibration matrix are then multiplied to obtain the target pixel value corresponding to the first pixel. Based on the target pixel values ​​corresponding to each first pixel, the target image is generated.

[0119] In this implementation, the second image is first pixel-aligned with the first image, ensuring that the second image has the same number of pixels as the first image. The pixel values ​​of each color channel of the second pixel whose pixel index matches that of the first pixel are then obtained as the second component. This allows the second component to be obtained pixel-by-pixel for subsequent stitching, ensuring that the second component is obtained for each first pixel. This, in turn, enables the generation of the target pixel value for each first pixel, ensuring optimal image processing results.

[0120] Optionally, step 103 above may specifically include:

[0121] Step 1031: Multiply the integrated vector and the preset calibration matrix to obtain the target pixel value.

[0122] Specifically, the composite vector is in column vector form, representing a matrix of size (k+3)×1. Let M represent the preset calibration matrix. For the first pixel n, the composite vector corresponding to the first pixel n can be left-multiplied by the preset calibration matrix to obtain the pixel prediction value (X) corresponding to the first pixel n. n_p Y n_p Z n_p The predicted pixel value corresponding to the first pixel is the target pixel value corresponding to the first pixel.

[0123] In this embodiment, the target pixel value corresponding to the first pixel can be calculated by directly multiplying the comprehensive vector and the preset calibration matrix, which is highly efficient in determining the target pixel value.

[0124] Compared to methods that rely solely on multispectral image information for correction, ignoring the color accuracy enhancement provided by the RGB sensor, this embodiment uses the pixel values ​​of the color channels of pixels in the first and second images acquired by the RGB imaging sensor and the multispectral imaging sensor, respectively, to generate a first component and a second component, which are then concatenated to obtain a composite vector. The composite vector is multiplied by the calibration matrix to generate the target pixel value, and the target image is generated based on the target pixel value. Since the first component is obtained based on the color information of the first image (i.e., the RGB values ​​of the pixels), and the second component is obtained based on the spectral information of the multispectral image (i.e., the channel values ​​of k spectral channels), the RGB color information of each pixel in the target image is essentially obtained by combining the color information of the first image and the spectral information of the multispectral image for correction. This simultaneously leverages the color calibration capabilities of both sensors and makes fuller use of the color information of the first image.

[0125] Optionally, the target pixel value corresponding to each first pixel can be converted into an RGB color space representation to obtain the RGB value corresponding to each first pixel. For example, an initial RGB value can be obtained by calculating the product of the transformation matrix corresponding to the specified RGB space and the target pixel value corresponding to the first pixel. Then, the calculated initial RGB value is gamma-corrected using the gamma parameter of the specified RGB space to obtain the RGB value corresponding to the first pixel. Gamma correction can compensate for the non-linear characteristics of the display, ensuring that the final generated target image is correctly displayed on the display.

[0126] The specified RGB color space can be selected on demand based on the device attributes of the electronic device; for example, it can be sRGB, display P3, BT.2020, etc. The transformation matrix corresponding to the specified RGB color space can be pre-generated based on the primary color coordinates and white point coordinates of the specified RGB color space. The primary color coordinates refer to the chromaticity coordinates of red, green, and blue in the RGB color space. Different RGB color spaces have different primary color coordinates. The white point coordinates refer to the chromaticity coordinates of the white point, used to characterize the white obtained when the three primary colors of red, green, and blue are mixed with equal intensity in the RGB color space. Finally, the target image is obtained based on the RGB value corresponding to each first pixel. For example, each first pixel in the first image is set to the calculated RGB value corresponding to that first pixel to obtain the target image.

[0127] In this embodiment, there is no need to reconstruct high-precision spectral reflectance from multispectral data, thus resulting in faster processing speed. Furthermore, since the RGB images acquired by the RGB imaging sensor themselves possess high spatial resolution information, this embodiment determines the target pixel value for each pixel in the first image, enabling the generated target image to have high spatial resolution. Ultimately, a target image with higher spatial resolution and color accuracy is obtained, improving the image processing effect.

[0128] See Figure 8 This application provides a block diagram of an image processing apparatus, the apparatus comprising:

[0129] The first control module 801 is used to control the RGB imaging sensor and the multispectral imaging sensor to capture the same shooting scene, and obtain the first image and the second image respectively;

[0130] The first generation module 802 is used to obtain a comprehensive vector based on the first pixel value of the first image and the second pixel value of the second image;

[0131] The second generation module 803 is used to obtain the target pixel value based on the comprehensive vector and the preset calibration matrix; wherein, the preset calibration matrix is ​​obtained based on the RGB calibration image captured by the RGB imaging sensor and the multispectral calibration image captured by the multispectral imaging sensor;

[0132] The third generation module 804 is used to generate a target image based on the target pixel value.

[0133] Optionally, the preset calibration matrix is ​​obtained through the following module:

[0134] The second control module is used to control the RGB imaging sensor to capture images of the color chart to obtain an RGB calibration image, and to control the multispectral imaging sensor to capture images of the color chart to obtain a multispectral calibration image.

[0135] The segmentation module is used to segment the RGB calibration image and the multispectral calibration image to obtain color patch region segmentation pairs;

[0136] The fourth generation module is used to generate a calibration comprehensive vector based on the pixel values ​​of the pixels in the color block region segmentation.

[0137] The fifth generation module is used to generate pixel prediction values ​​for the color patch region segmentation pairs based on the calibration synthesis vector and the initial matrix;

[0138] The processing module is used to iteratively optimize the initial matrix based on the difference information between the pixel prediction value and the actual pixel value of the corresponding color block to obtain the preset calibration matrix.

[0139] Optionally, the fourth generation module is specifically used for:

[0140] Based on the pixel values ​​of each color channel of the pixel in the color block region segmentation, the statistical values ​​corresponding to each color channel are determined, and the calibration components of each color channel are obtained.

[0141] The calibration composite vector is obtained based on the calibration components of each color channel.

[0142] Optionally, the fifth generation module is specifically used for:

[0143] The pixel prediction values ​​of the color patch region segmentation pair are obtained by multiplying the calibration synthesis vector and the initial matrix.

[0144] Optionally, the first pixel value includes the pixel values ​​of each color channel of the first pixel, and the second pixel value includes the pixel values ​​of each color channel of the second pixel; the first generation module 202 is specifically used for:

[0145] The pixel values ​​of each color channel of the first pixel are used as the first component, and the second component is determined based on the pixel values ​​of each color channel of the second pixel corresponding to the first pixel.

[0146] The first component and the second component are concatenated to obtain the composite vector.

[0147] Optionally, the apparatus further includes: a segmentation module, used to perform scene segmentation on the first image and the second image respectively to obtain corresponding scene region segmentation pairs;

[0148] The first generation module 802 is further configured to:

[0149] Based on the scene region segmentation pair, the first component and the second component are concatenated to obtain the comprehensive vector.

[0150] Optionally, the second generation module 803 is specifically used to: perform matrix multiplication on the comprehensive vector and the preset calibration matrix to obtain the target pixel value.

[0151] The image processing device described above has the same advantages over related technologies as the image processing method described in the foregoing embodiments, and will not be repeated here.

[0152] The image processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device. The image processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems, this application embodiment does not specifically limit the device. The image processing device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method embodiments achieve the same technical effect, and will not be described again here to avoid repetition.

[0153] In some embodiments, such as Figure 9 As shown in the illustration, this application also provides an electronic device 900, including a processor 901 and a memory 902. The memory 902 stores a program or instructions that can run on the processor 901. When the program or instructions are executed by the processor 901, they implement the various steps of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here. It should be noted that the electronic device in this application includes the aforementioned mobile electronic device and non-mobile electronic device.

[0154] Figure 10This is a schematic diagram of the hardware structure of another electronic device according to an embodiment of this application. The electronic device 1000 includes, but is not limited to, components such as: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010. Those skilled in the art will understand that the electronic device 1000 may also include a power supply (such as a battery) to power the various components. The power supply can be logically connected to the processor 1010 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Figure 10 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0155] The processor 1010 controls the RGB imaging sensor and the multispectral imaging sensor to capture images of the same scene, obtaining a first image and a second image respectively; a comprehensive vector is obtained based on the first pixel value of the first image and the second pixel value of the second image; a target pixel value is obtained based on the comprehensive vector and a preset calibration matrix; wherein the preset calibration matrix is ​​obtained based on the RGB calibration image captured by the RGB imaging sensor and the multispectral calibration image captured by the multispectral imaging sensor; and a target image is generated based on the target pixel value.

[0156] In some embodiments, the processor 1010 is further configured to: before obtaining the target pixel value based on the composite vector and the preset calibration matrix:

[0157] The RGB imaging sensor is controlled to capture an image of the color chart to obtain an RGB calibration image, and the multispectral imaging sensor is controlled to capture an image of the color chart to obtain a multispectral calibration image;

[0158] The RGB calibration image and the multispectral calibration image are segmented to obtain color patch region segments;

[0159] A calibration synthesis vector is generated based on the pixel values ​​of the pixels in the color block region segmentation.

[0160] Pixel prediction values ​​for the color patch region segmentation pairs are generated based on the calibration synthesis vector and the initial matrix.

[0161] Based on the difference between the predicted pixel value and the actual pixel value of the corresponding color block, the initial matrix is ​​iteratively optimized to obtain the preset calibration matrix.

[0162] In some embodiments, the processor 1010 is further configured to: determine the statistical value corresponding to each color channel based on the pixel value of each color channel of the pixel in the color block region segmentation, and obtain the calibration component of each color channel; and obtain the calibration synthesis vector based on the calibration component of each color channel.

[0163] In some embodiments, the processor 1010 is further configured to: perform matrix multiplication on the calibration synthesis vector and the initial matrix to obtain the pixel prediction value of the color block region segmentation pair.

[0164] In some embodiments, the first pixel value includes the pixel values ​​of each color channel of the first pixel, and the second pixel value includes the pixel values ​​of each color channel of the second pixel; the processor 1010 is further configured to: use the pixel values ​​of each color channel of the first pixel as a first component, and determine a second component based on the pixel values ​​of each color channel of the second pixel corresponding to the first pixel; and concatenate the first component and the second component to obtain the composite vector.

[0165] In some embodiments, before taking the pixel values ​​of each color channel of the first pixel as the first component and taking the pixel values ​​of each color channel of the second pixel corresponding to the first pixel as the second component, the processor 1010 is further configured to: perform scene segmentation on the first image and the second image respectively to obtain corresponding scene region segmentation pairs; and concatenate the first component and the second component based on the scene region segmentation pairs to obtain the comprehensive vector.

[0166] In some embodiments, the processor 1010 is further configured to: perform matrix multiplication on the composite vector and the preset calibration matrix to obtain the target pixel value.

[0167] The electronic device and the image processing method described in the foregoing embodiments have the same advantages over related technologies, which will not be repeated here.

[0168] It should be understood that, in this embodiment, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042. The GPU 10041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1006 may include a display panel 10061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0169] The memory 1009 can be used to store software programs and various data. The memory 1009 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1009 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1009 in this embodiment includes, but is not limited to, these and any other suitable types of memory. Processor 1010 may include one or more processing units; in some embodiments, processor 1010 integrates an application processor and a modem processor, wherein the application processor primarily handles operations related to the operating system, user interface, and applications, while the modem processor primarily handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may not be integrated into processor 1010.

[0170] This application also provides a readable storage medium storing a program or instructions. When executed by a processor, the program or instructions implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, further details are omitted here. The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk. This application also provides a chip including a processor and a communication interface coupled to the processor. The processor runs the program or instructions to implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, further details are omitted here. It should be understood that the chip mentioned in this application can also be called a system-on-a-chip (SoC), system-on-a-chip (SoC), chip system, or system-on-a-chip (SoC). This application also provides a computer program product stored in a storage medium. This program product is executed by at least one processor to implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, further details are omitted here.

[0171] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0172] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0173] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: The RGB imaging sensor and the multispectral imaging sensor are controlled to capture images of the same scene, resulting in a first image and a second image, respectively. A comprehensive vector is obtained based on the first pixel value of the first image and the second pixel value of the second image; The target pixel value is obtained based on the comprehensive vector and the preset calibration matrix; wherein, the preset calibration matrix is ​​obtained based on the RGB calibration image captured by the RGB imaging sensor and the multispectral calibration image captured by the multispectral imaging sensor; Generate a target image based on the target pixel values.

2. The method according to claim 1, characterized in that, Before obtaining the target pixel value based on the comprehensive vector and the preset calibration matrix, the method further includes: The RGB imaging sensor is controlled to capture an image of the color chart to obtain an RGB calibration image, and the multispectral imaging sensor is controlled to capture an image of the color chart to obtain a multispectral calibration image; The RGB calibration image and the multispectral calibration image are segmented to obtain color patch region segments; A calibration synthesis vector is generated based on the pixel values ​​of the pixels in the color block region segmentation. Pixel prediction values ​​for the color patch region segmentation pairs are generated based on the calibration synthesis vector and the initial matrix. Based on the difference between the predicted pixel value and the actual pixel value of the corresponding color block, the initial matrix is ​​iteratively optimized to obtain the preset calibration matrix.

3. The method according to claim 2, characterized in that, The step of generating a calibration synthesis vector based on the pixel values ​​of the pixels in the color block region segmentation includes: Based on the pixel values ​​of each color channel of the pixel in the color block region segmentation, the statistical values ​​corresponding to each color channel are determined, and the calibration components of each color channel are obtained. The calibration composite vector is obtained based on the calibration components of each color channel.

4. The method according to claim 2, characterized in that, The step of generating pixel prediction values ​​for the color patch region segmentation pairs based on the calibration synthesis vector and the initial matrix includes: The pixel prediction values ​​of the color patch region segmentation pair are obtained by multiplying the calibration synthesis vector and the initial matrix.

5. The method according to any one of claims 1-4, characterized in that, The first pixel value includes the pixel values ​​of each color channel of the first pixel, and the second pixel value includes the pixel values ​​of each color channel of the second pixel; the comprehensive vector obtained based on the first pixel value of the first image and the second pixel value of the second image includes: The pixel values ​​of each color channel of the first pixel are used as the first component, and the second component is determined based on the pixel values ​​of each color channel of the second pixel corresponding to the first pixel. The first component and the second component are concatenated to obtain the composite vector.

6. The method according to claim 5, characterized in that, Before taking the pixel values ​​of each color channel of the first pixel as the first component, and taking the pixel values ​​of each color channel of the second pixel corresponding to the first pixel as the second component, the method further includes: Scene segmentation is performed on the first image and the second image respectively to obtain corresponding scene region segmentation pairs; The concatenation of the first component and the second component to obtain the composite vector includes: Based on the scene region segmentation pair, the first component and the second component are concatenated to obtain the comprehensive vector.

7. The method according to any one of claims 1-4, characterized in that, The process of obtaining the target pixel value based on the comprehensive vector and the preset calibration matrix includes: The target pixel value is obtained by multiplying the comprehensive vector and the preset calibration matrix.

8. An image processing apparatus, characterized in that, The device includes: The first control module is used to control the RGB imaging sensor and the multispectral imaging sensor to capture the same shooting scene, and obtain the first image and the second image respectively; The first generation module is used to obtain a comprehensive vector based on the first pixel value of the first image and the second pixel value of the second image; The second generation module is used to obtain the target pixel value based on the comprehensive vector and the preset calibration matrix; wherein, the preset calibration matrix is ​​obtained based on the RGB calibration image captured by the RGB imaging sensor and the multispectral calibration image captured by the multispectral imaging sensor; The third generation module is used to generate a target image based on the target pixel values.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the image processing method as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the image processing method as described in any one of claims 1-7.

Images