Image processing method and device, electronic equipment, storage medium and program product

Abstract

This application relates to an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. The method includes: acquiring first spectral image data and second spectral image data; the first spectral image data and the second spectral image data have different brightness levels; performing image fusion on regions in the first image data and corresponding regions in the second image data to obtain region fusion data; and statistically analyzing the region fusion data to obtain target spectral data. This method enables information complementarity among various spectral channels at different brightness levels, helping to improve the accuracy of spectral channel information and giving the target spectral data a high dynamic range.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] As the photography capabilities of electronic devices become increasingly sophisticated, it is necessary to shoot in specific shooting scenarios to meet diverse shooting needs.

[0003] However, electronic devices use image sensors to acquire images. Since image sensors have a certain threshold for the brightness they can acquire, it is difficult to fully capture the brightness of the scene being photographed, which may result in missing information. Summary of the Invention

[0004] This application provides an image processing method, apparatus, electronic device, and computer-readable storage medium to improve the accuracy of spectral channel information and increase the dynamic range of target spectral data.

[0005] In a first aspect, this application provides an image processing method, comprising:

[0006] Acquire first spectral image data and second spectral image data; the brightness of the first spectral image data and the second spectral image data are different.

[0007] Image fusion is performed on the region in the first image data and the corresponding region in the second image data to obtain region fusion data;

[0008] The target spectral data is obtained by statistically analyzing the fused data of the region.

[0009] Secondly, this application also provides an image processing apparatus, comprising:

[0010] The acquisition module is used to acquire first spectral image data and second spectral image data; the brightness of the first spectral image data and the second spectral image data are different;

[0011] The fusion module is used to perform image fusion on the region in the first spectral image data and the corresponding region in the second spectral image data to obtain region fusion data;

[0012] The restoration module is used to statistically analyze the fused data of the region to obtain the target spectral data.

[0013] Thirdly, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the image processing method of the first aspect.

[0014] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method of the first aspect.

[0015] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the image processing method of the first aspect.

[0016] The aforementioned image processing methods, apparatuses, electronic devices, computer-readable storage media, and computer program products acquire first spectral image data and second spectral image data. On one hand, since both the first and second spectral image data contain multiple spectral channels, their dimensions are relatively richer, enabling the acquisition of more accurate information. On the other hand, because the overall brightness of the two spectral image data differs, even if a certain spectral channel reaches its maximum or minimum value under one overall brightness level, preventing further information capture, information about that spectral channel can still be provided through the other overall brightness level. Therefore, fusion in this way allows for information complementarity among various spectral channels under different brightness levels. Based on this, image fusion is performed according to the corresponding regions of the two spectral image data to obtain regional fusion data. This allows each region to be processed separately, forming regional-level data. This richer data hierarchy helps improve the accuracy of spectral channel information. Finally, the regional fusion data is statistically analyzed, transforming the regional-level data into global-level target spectral data, thereby improving the dynamic range. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is an application environment diagram of an image processing method in one embodiment;

[0019] Figure 2 This is a flowchart illustrating an image processing method in one embodiment;

[0020] Figure 3 This is a schematic diagram of the spectral channel array of a repeating unit in one embodiment;

[0021] Figure 4 This is a schematic diagram of the spectral channels of a partition in one embodiment;

[0022] Figure 5This is a schematic diagram of the array for each group of channels in one embodiment;

[0023] Figure 6 This is a schematic diagram of spectral channel fusion for each region in one embodiment;

[0024] Figure 7 This is a schematic diagram illustrating the specific process of an image processing method in one embodiment;

[0025] Figure 8 This is a structural block diagram of an image processing device in one embodiment;

[0026] Figure 9 This is a diagram of the internal structure of an electronic device in one embodiment. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0028] In the field of multispectral image sensors, there exist image sensors with multiple regions and multiple spectral channels. The hardware solution for these sensors utilizes pixel arrays and resin dyes. The primary purpose of this image sensor in mobile phones is to assist the camera in white balance (AWB) and color reproduction. While multispectral image sensors assist RGB cameras, the hardware investment in multispectral image sensors is significantly lower than that of RGB cameras, and the performance of the image sensor in a multispectral image sensor is also far inferior to that of an RGB camera. This performance difference leads to insufficient dynamic range in multispectral image sensors, making it difficult to perform the corresponding camera functions well in high dynamic range scenes. Therefore, this application is necessary.

[0029] The image processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, electronic device 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Electronic device 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0030] In some exemplary embodiments, such as Figure 2 As shown, an image processing method is provided, which is applied to... Figure 1 Taking electronic device 102 as an example, this method can also be applied to server 104. The method includes the following steps 202 to 206.

[0031] Step 202: Obtain first spectral image data and second spectral image data; the brightness of the first spectral image data and the second spectral image data are different.

[0032] The first and second spectral image data are acquired through multiple spectral channels. These multiple spectral channels can belong to the acquisition channels configured in the image sensor. An image sensor is a photosensitive element used for image acquisition. An image sensor can acquire spectral information in different bands for analysis of each band. Functionally, an image sensor includes a main image sensor and an auxiliary image sensor, which are used to acquire spectral information in different ranges. When the main image sensor includes an RGB camera module, the auxiliary image sensor has a partitioned multi-channel module with multiple spectral channels; conversely, when the main image sensor has a partitioned multi-channel module with multiple spectral channels, the auxiliary image sensor includes an RGB camera module.

[0033] Spectral channels are channels used to acquire information according to preset wavelengths. Each spectral channel provides information for one wavelength band, therefore the brightness influence between different spectral channels is small, and the corresponding channel values ​​can be determined from different perspectives. The brightness of each spectral channel is information at the channel level, and the brightness of each spectral channel is used to characterize the information acquired by the spectral channel.

[0034] Spectral channels can be distinguished from two perspectives: frequency band and processing method. Each spectral channel is for signal acquisition of the spectrum in the same frequency band. Each spectral channel can be configured with multiple spectral channels. For example, in the spectral detection unit, the spectral channel used for red light can be configured with multiple spectral channels at different positions. Each spectral pixel channel has an independent signal acquisition and processing method, and the signal acquisition and processing method of each spectral channel can be the same or different.

[0035] Image acquisition using an image sensor with multiple spectral channels yields first and second spectral image data. Correspondingly, these are image data with multiple spectral channels, and the brightness of the second spectral image data differs from that of the first spectral image data at different image levels. Because the overall brightness of these two types of spectral image data differs at the image level, even if a certain spectral channel reaches its maximum or minimum value under one overall brightness level, preventing further information capture, information about that spectral channel can still be provided by the other overall brightness level. Therefore, fusion in this way allows for the complementary information from various spectral channels at different brightness levels.

[0036] From the perspective of image meaning and acquisition process, both the first and second spectral image data include at least one image, and the acquisition process for both the first and second spectral image data is performed at least once. Therefore, it is possible to provide more types of brightness and information for fusion.

[0037] From a relative perspective, in at least two frames of spectral image data, any number of frames with different brightness levels are considered the first and second spectral image data to each other; in this case, the first and second spectral image data are relative concepts. At least two frames of spectral image data can also be three frames of spectral image data; these three frames are, in order of increasing brightness, underexposed image data (EV-), normally exposed image data (EV0), and overexposed image data (EV+); or, in order of decreasing brightness, normally exposed image data (EV0), underexposed image data (EV-), and severely underexposed image data (EV--). Taking underexposed, normally exposed, and overexposed image data as an example, any two frames of underexposed, normally exposed, and overexposed image data are considered the first and second spectral image data to each other, resulting in intermediate region fusion data, i.e., the fusion result of any two frames of image data; and the fusion result of any two frames of image data and another frame of image data are considered the first and second spectral image data to each other, and then fusion is performed again to obtain the region fusion data used in step 206.

[0038] For example, the multiple spectral channels include, but are not limited to, a red channel for acquiring red light, a green channel for acquiring green light, a blue channel for acquiring blue light, a cyan channel for acquiring cyan light, a magenta channel for acquiring magenta light, a yellow channel for acquiring yellow light (Y), and a violet channel for acquiring violet light. Specifically, the frequency band that the red channel can acquire is 620-700 nanometers (nm); the frequency band that the green channel can acquire is 500-565 nanometers (nm); the frequency band that the blue channel can acquire is 440-485 nanometers (nm); the frequency band that the cyan channel can acquire is 485-500 nanometers (nm); the frequency band that the magenta channel can acquire is 625-740 nm, and the superposition portion between 440-485 nm; the frequency band that the yellow channel can acquire is 565-570 nm; and the frequency band that the violet channel can acquire is 380-450 nm.

[0039] In the image acquisition implementation, the brightness values ​​of multiple spectral channels can be obtained by filtering with at least six bandpass filters. These at least six bandpass filters include three filters corresponding to the RGB color space and a filter corresponding to the CMY color space. The three filters corresponding to the RGB color space are used to transmit red (R), green (G), and blue (B) light, respectively, while the three filters corresponding to the CMY color space are used to transmit cyan (C), magenta (M), and yellow (Y) light, respectively. For example, the multiple spectral channels may also include an infrared filter (IR filter) for preserving infrared light and a clear filter for preserving various spectra. The multiple spectral channels can also be eight channels, including the three filters corresponding to the RGB color space, the filter corresponding to the CMY color space, the infrared filter, and the clear filter, which can be characterized as RGBCMY+clear+IR. The multiple spectral channels may also involve other color spaces or relationships between different color spaces, thus forming nine bands or other numbers of spectral channels.

[0040] In some embodiments, acquiring first spectral image data and second spectral image data includes: after filtering out first light signals that do not belong to the spectral channel using filters disposed at different positions on the image sensor, converting the filtered first light signals into electrical signals at each position using photoelectric sensors to obtain first spectral image data; and after filtering out second light signals that do not belong to the spectral channel using filters disposed at different positions on the image sensor, converting the filtered second light signals into electrical signals at each position using photoelectric sensors to obtain second spectral image data. Thus, by using filters to filter out light signals, the circuit and processor process relatively less data, improving efficiency.

[0041] In some embodiments, acquiring first spectral image data and second spectral image data includes: obtaining first spectral image data by exposure for a first exposure duration; and obtaining second spectral image data by exposure for a second exposure duration; wherein the first exposure duration and the second exposure duration are different. Thus, by using different exposure durations, the brightness of the spectral image data is controlled to ensure processing efficiency.

[0042] Step 204: Perform image fusion on the corresponding regions in the first spectral image data and the second spectral image data to obtain region fusion data.

[0043] The region fusion data includes a set of spectral channel values ​​provided by the spectral channels within each region. Each spectral channel within a region is defined according to its location within a physical area of ​​the image sensor. For example, the x-coordinate of each spectral channel within a region is within a preset x-coordinate range, and the y-coordinate of each spectral channel within a region is within a preset y-coordinate range.

[0044] Each region's spectral channels can also be divided according to a preset number of regions. Each region's spectral channels include a set of spectral channels, and each set of spectral channels includes multiple spectral channels. For example, an image sensor has 200w pixel spectral channels, the image resolution acquired by the image sensor is 1600*1200 pixels, the preset number of regions is 8*6, a set of spectral channels has 3x3 spectral channels, and each set of spectral channels has 9 spectral channels, and these 9 spectral channels are of different types; in this case, 1600*1200 divided by 8*6 partitions results in 200*200 spectral channels per region; therefore, 200*200 divided by 3x3, each region includes 66*66 = 4356 spectral channels; where the spectral channels of a certain repeating unit of the image sensor are as follows: Figure 3As shown, it includes multiple sets of spectral channels 310, each set of spectral channels 310 including 9 spectral channels. Each spectral channel is of a different type, thus forming spectral channels 301, 302, 303, 304, 305, 306, 307, 308, and 309; the spectral channels of the 8*6 region are as follows... Figure 4 As shown, each area includes, but is not limited to, area 401, area 402, and area 403, etc., and each group of channels is as follows: Figure 5 As shown. The fusion process of regional fusion data is as follows. Figure 6 As shown, the first image data from a long exposure and the second image data from a short exposure are used to provide information about different locations.

[0045] Region fusion data is the result of fusing each corresponding region from the first image data and the second image data. Within each region, multiple spectral channels still exist, and each spectral channel is fused separately. The number of spectral channel types in the region fusion data remains unchanged.

[0046] In some embodiments, image fusion is performed on regions of the first spectral image data and corresponding regions of the second spectral image data to obtain region fusion data, including: dividing the first image data and the second image data into M regions according to the positions of the spectral channels on the image sensor; fusing the spectral channel values ​​of the first image data in the i-th region and the spectral channel values ​​of the second image data in the i-th region according to the spectral channels to obtain fused data for each spectral channel of the i-th region; i is a positive integer less than or equal to M, and M is a positive integer.

[0047] Step 206: Statistically fuse regional data to obtain target spectral data.

[0048] The target spectral data is the overall spectral data formed based on the fused data from each region. Regarding the number of channels in the target spectral data, it is the cumulative number of spectral channels from each region. As for the spectral channel values, the formation process of the target spectral data includes statistical analysis of the spectral channel values ​​for each spectral channel in the fused regional data.

[0049] In the process of obtaining the target spectral data, the channel values ​​of the corresponding spectral channels in the regional fusion data can be statistically analyzed, or the proportion of channel values ​​of different spectral channels can be analyzed, such as the ratio of different types of spectral channels in the regional fusion data. Of course, other data used to calculate white balance results or color reproduction data can also be analyzed, without limitation.

[0050] In some embodiments, obtaining target spectral data by statistically fusing regional data includes: statistically analyzing the target parameters in the regional fusion data to obtain statistical results of the target parameters; and generating target spectral data according to the statistical results and their positions. Target parameters include, but are not limited to, channel values ​​of spectral channels and the proportion of channel values ​​of different spectral channels.

[0051] In the aforementioned image processing method, first and second spectral image data are acquired. On one hand, both first and second spectral image data contain multiple spectral channels, thus offering richer dimensions and enabling the acquisition of more accurate information. On the other hand, because the overall brightness of these two spectral image data differs, even if a certain spectral channel reaches its maximum or minimum value under one overall brightness level, preventing further information capture, information about that spectral channel can still be obtained through the other overall brightness level. Therefore, fusion in this way allows for complementary information from various spectral channels under different brightness levels. Based on this, image fusion is performed according to the corresponding regions of the two spectral image data to obtain region-fused data. This allows each region to be processed separately, forming region-level data. This richer data hierarchy helps improve accuracy. Finally, the region-fused data is statistically analyzed, transforming the region-level data into global-level target spectral data, thereby improving the dynamic range.

[0052] In some embodiments, the method further includes: acquiring initial spectral image data; and, if the brightness of the initial spectral image data meets the adjustment conditions, performing the steps described above for acquiring the first spectral image data and the second spectral image data.

[0053] The initial spectral image data is at least one image with multiple spectral channels. In terms of brightness, the initial spectral image data is less bright than the first spectral image data and more bright than the second image data, so that the brightness of the initial spectral image data is moderate.

[0054] In terms of acquisition method, the acquisition method of the initial spectral image data is the same as that of the first spectral image data, and the exposure time of the initial spectral image data is different from that of the first spectral image data, so that the brightness of the initial spectral image data is different from that of the first spectral image data.

[0055] The adjustment conditions are the acquisition conditions for spectral image data with different brightness levels. If the brightness of the initial spectral image data meets the adjustment conditions, it can be determined based on the global brightness of the initial spectral image data or its local brightness. Alternatively, the brightness can also be determined based on the number and distribution of spectral channels. Furthermore, when fusing the first and second spectral image data, the initial spectral image data can also be fused.

[0056] In this embodiment, initial spectral image data is acquired first, and then conditional judgment is made based on the brightness of the initial spectral image data. This reduces the frequency of image acquisition and image fusion of spectral image data, thereby improving processing efficiency.

[0057] In some embodiments, the step of determining whether the brightness of the initial spectral image data meets the adjustment conditions includes: counting the number of spectral channels whose brightness values ​​meet the brightness conditions in the initial spectral image data to obtain the number of spectral channels that meet the brightness conditions; if the number of spectral channels that meet the brightness conditions meets a preset quantity condition, then the brightness of the initial spectral image data meets the adjustment conditions.

[0058] The number of spectral channels is a statistical result of the brightness values ​​of each spectral channel. Since the brightness of a spectral channel belongs to the local level, while the number of spectral channels belongs to the global level of the initial spectral image data, the statistical process achieves a transformation from the local level to the global level. This number of spectral channels can also be unrelated to the type of spectral channel, thus improving statistical efficiency. Furthermore, when the brightness conditions of different spectral channels vary, and the number of spectral channels is unrelated to the type of spectral channel, it helps to improve both the efficiency and accuracy of the statistics.

[0059] Brightness conditions are set for the brightness values ​​of spectral channels. They are used to filter spectral channels whose brightness values ​​fall within abnormal brightness ranges, thereby enabling more accurate analysis of local brightness information. The brightness value of a spectral channel is one type of channel value; channel values ​​may also include attributes such as channel coordinates, channel distribution, and channel type.

[0060] The preset quantity conditions include global quantity conditions. Therefore, when the number of spectral channels exceeds the corresponding threshold, there is a global level of brightness anomaly. At this time, the brightness of the initial spectral image data meets the adjustment conditions. The preset quantity conditions may also include conditions based on the location of the corresponding spectral channels, so as to further form different local levels based on the distribution information of the corresponding locations, thereby more accurately determining whether the brightness of the initial spectral image data meets the adjustment conditions.

[0061] In some embodiments, the preset quantity condition can be a preset quantity ratio. Correspondingly, if the number of spectral channels that meet the brightness condition meets the preset quantity condition, then the brightness of the initial spectral image data meets the adjustment condition, including: if the ratio between the number of spectral channels that meet the brightness condition and the total number of spectral channels in the spectral image data is greater than the preset quantity ratio, then the brightness of the initial spectral image data meets the adjustment condition.

[0062] In some embodiments, in the initial spectral image data, the number of spectral channels whose brightness values ​​meet the brightness conditions is counted to obtain the total number of spectral channels that meet the brightness conditions. This includes: identifying spectral channels whose brightness values ​​exceed a critical value in the initial spectral image data; and accumulating the number of spectral channels whose brightness values ​​exceed the critical value to obtain the total number of spectral channels that meet the brightness conditions. This accumulation method improves statistical efficiency.

[0063] In some embodiments, the number of spectral channels whose brightness values ​​meet the brightness conditions in the initial spectral image data is statistically analyzed to obtain the total number of spectral channels meeting the brightness conditions. This includes: determining the brightness histogram for each group of spectral channels in the initial spectral image data; and accumulating the number of spectral channels whose brightness histograms exceed a critical value to obtain the total number of spectral channels meeting the brightness conditions. Thus, a new histogram layer is added between the spectral channels and the global layer, improving the accuracy of the statistics.

[0064] In this embodiment, the number of spectral channels whose brightness values ​​meet the brightness conditions in the initial spectral image data is counted to filter out spectral channels that can more accurately determine the spectral channels related to the adjustment conditions, thereby achieving local level judgment. By filtering out the number of spectral channels that meet the brightness conditions, the saturation degree of the spectral channels at the global level can be more accurately reflected. Therefore, when the number of spectral channels meets the preset quantity condition, global level judgment can be achieved to more accurately determine whether the brightness of the initial spectral image data meets the adjustment conditions.

[0065] In some embodiments, the spectral channels whose brightness values ​​satisfy the brightness conditions include at least one of a first spectral channel and a second spectral channel; the brightness value of the first spectral channel is less than a first critical value, the brightness value of the second spectral channel is equal to a second critical value, and the second critical value is greater than the first critical value.

[0066] The first spectral channel is a spectral channel that provides insufficient information, i.e., an underexposed spectral channel. Because the brightness of the first spectral channel is less than a first threshold, it provides too little information. Therefore, it is necessary to fuse different spectral image data to provide information from different regions through long and short exposures. For example, the first spectral channel is a spectral channel with a brightness value less than 5, i.e., low-quality pixels.

[0067] The second spectral channel is the spectral channel that provides information up to its saturation value at the corresponding brightness, i.e., the overexposed spectral channel. Because the channel value cannot provide more information after reaching its maximum value, it is necessary to fuse different spectral image data to provide information in different regions through long and short exposures. For example, the second spectral channel is the spectral channel whose brightness value equals its saturation value, i.e., the saturated pixel.

[0068] The first threshold value is a luminance value used to judge luminance quality. Spectral channels with luminance values ​​less than the first threshold value are designated as first spectral channels, while spectral channels with luminance values ​​greater than the first threshold value provide only moderate information and are therefore not considered first spectral channels.

[0069] The second threshold value is a luminance value used to determine the degree of luminance saturation. Spectral channels with luminance values ​​equal to the second threshold value are considered second spectral channels, while spectral channels with luminance values ​​less than the second threshold value provide only moderate information and are therefore not considered second spectral channels.

[0070] The spectral channel whose brightness value meets the brightness condition includes at least one of the first spectral channel and the second spectral channel. This means that the spectral channel whose brightness value meets the brightness condition can be the first spectral channel, the second spectral channel, or both. In the case where both the first spectral channel and the second spectral channel are included, the number of spectral channels of the first spectral channel and the number of spectral channels of the second spectral channel can be counted separately, and then it can be determined whether the number of spectral channels of these two types of spectral channels meets the preset quantity condition. If either one meets the preset quantity condition, then the brightness of the initial spectral image data meets the adjustment condition.

[0071] In this embodiment, the spectral channel whose brightness value meets the brightness condition is a spectral channel whose brightness value exceeds the normal brightness range. Therefore, both the first spectral channel used to represent underexposure and the second spectral channel used to represent overexposure can be used as spectral channels that meet the brightness condition. As a result, the shooting channels can be adaptively adjusted to meet the adjustment conditions for different scenarios, so as to control the fusion process more accurately and flexibly.

[0072] In some embodiments, image fusion is performed on regions of the first spectral image data and corresponding regions of the second spectral image data to obtain region fusion data. This includes: processing the luminance value of the j-th spectral channel in the i-th region of the first spectral image data and the corresponding luminance fusion weight, as well as the luminance value of the j-th spectral channel in the i-th region of the second spectral image data and the corresponding luminance fusion weight, to obtain fusion data of the j-th spectral channel in the i-th region; where i is a positive integer and j is a positive integer.

[0073] The luminance fusion weight is a weight value determined based on the luminance value. It is used to adjust the luminance value, thereby achieving data fusion of two spectral image data sets. By determining the luminance fusion weight based on the luminance value of each spectral channel, the fused data from different spectral channels can be adaptively adjusted.

[0074] In some embodiments, processing is performed on the brightness value of the j-th spectral channel in the i-th region of the first spectral image data and the corresponding brightness fusion weight, and on the brightness value of the j-th spectral channel in the i-th region of the second spectral image data and the corresponding brightness fusion weight, to obtain fused data of the j-th spectral channel in the i-th region. This includes: mapping the brightness value of the j-th spectral channel in the i-th region of the first spectral image data to a first brightness fusion weight, and adjusting the brightness value of the j-th spectral channel in the i-th region of the first spectral image data according to the first brightness fusion weight to obtain a first brightness value to be accumulated for the j-th spectral channel in the i-th region; mapping the brightness value of the j-th spectral channel in the i-th region of the second spectral image data to a second brightness fusion weight, and adjusting the brightness value of the j-th spectral channel in the i-th region of the second spectral image data according to the second brightness fusion weight to obtain a second brightness value to be accumulated for the j-th spectral channel in the i-th region; and accumulating the first brightness value to be accumulated and the second brightness value to be accumulated to obtain fused data of the j-th spectral channel in the i-th region.

[0075] In this embodiment, the brightness value of each spectral channel has a corresponding brightness fusion weight. Therefore, the fusion data of each spectral channel is formed through the brightness fusion weight at the local level, thereby ensuring the accuracy of the fusion data.

[0076] In some embodiments, the brightness value includes a first brightness value and a second brightness value in different brightness ranges; the first brightness value is less than the second brightness value; the first brightness value is positively correlated with the brightness fusion weight; and the second brightness value is negatively correlated with the brightness fusion weight.

[0077] A brightness interval is the distribution of brightness values ​​for each spectral channel. Different brightness intervals exhibit different rules of variation in their brightness values. There are at least two brightness intervals: the maximum value in the interval containing the first brightness value is less than the minimum value in the interval containing the second brightness value, ensuring that the first brightness value is less than the second brightness value. Within the brightness interval containing the first brightness value, there can also be multiple sub-intervals to further refine the brightness fusion weights.

[0078] The first brightness value represents a case where the brightness is too low, indicating a significant degree of noise influence. Therefore, the first brightness value is positively correlated with the brightness fusion weight. The lower the first brightness value, the greater the noise in the spectral channel to which it belongs, resulting in a lower confidence level. Consequently, the first brightness value is positively correlated with the corresponding brightness fusion weight, making the brightness fusion weight positively correlated with the confidence level. Conversely, the higher the first brightness value, the lower the noise in the spectral channel to which it belongs, resulting in a higher confidence level. Therefore, the first brightness value is positively correlated with the corresponding brightness fusion weight, making the brightness fusion weight positively correlated with the confidence level.

[0079] The second brightness value represents a situation where the brightness value is too high. This indicates that the degree to which the second brightness value approaches the saturation value has a greater impact, and therefore, the second brightness value is positively correlated with the brightness fusion weight. When the second brightness value is higher, the spectral channel to which the second brightness value belongs is closer to the saturation value. Therefore, the ratio of the second brightness value to the brightness values ​​of other spectral channels is more likely to deviate from the preset ratio, thus affecting the confidence level. Consequently, the second brightness value is negatively correlated with the corresponding brightness fusion weight, ensuring that the brightness fusion weight is positively correlated with the confidence level. Conversely, when the second brightness value is lower, the spectral channel to which the second brightness value belongs is further away from the saturation value. Therefore, the ratio of the second brightness value to the brightness values ​​of other spectral channels is more likely to conform to the preset ratio, also affecting the confidence level. Thus, the second brightness value is negatively correlated with the corresponding brightness fusion weight, ensuring that the brightness fusion weight is positively correlated with the confidence level.

[0080] The first and second brightness values ​​are set for each spectral channel, and the brightness value for each spectral channel can be either the first or the second brightness value. When the first and second brightness values ​​belong to different ranges, the brightness fusion weights are controlled more precisely, thereby ensuring the accuracy of the regional fusion data.

[0081] Since the first and second brightness values ​​are set for each spectral channel, each spectral channel in the first spectral image data can be both the first and second brightness values. Similarly, each spectral channel in the second spectral image data can also be both the first and second brightness values. Furthermore, because the first and second spectral image data represent different brightness levels at the same overall level, their brightness values ​​are not identical at the same pixel location.

[0082] In this embodiment, different brightness fusion weight change rules exist for brightness values ​​in different intervals, so as to control the changes in brightness values ​​more finely and thus ensure that the accuracy of regional fusion data is relatively high.

[0083] In some embodiments, when the first brightness value is less than the first threshold, the brightness fusion weight is the minimum preset value; when the first brightness value is greater than the second threshold and the first brightness value is less than the third threshold, the brightness fusion weight is the maximum preset value; when the second brightness value is greater than the fourth threshold, the brightness fusion weight is the minimum preset value; the first threshold, the second threshold, the third threshold and the fourth threshold are from small to large, and the minimum preset value is less than the maximum preset value.

[0084] The first threshold represents the level of noise that affects the brightness for normal use. If the first brightness value is less than the first threshold, it indicates excessive noise, and the corresponding brightness fusion weight is the minimum preset value. If the first brightness value is greater than the first threshold and less than the second threshold, it indicates relatively low noise, and the corresponding brightness fusion weight is positively correlated with the first brightness value.

[0085] The second threshold is a smaller brightness threshold indicating whether the noise level is sufficiently low and whether the spectral channels are sufficiently close to their saturation values. The third threshold is a larger brightness threshold indicating whether the noise level is sufficiently low and whether the spectral channels are sufficiently close to their saturation values. When the first brightness value is between the second and third thresholds, it indicates that the noise is low and the spectral channels are at their minimum close to their saturation values, thus the brightness fusion weight is the minimum preset value.

[0086] The fourth threshold is a threshold set for the second brightness value, used to represent the critical situation where noise is low and the spectral channels are not yet saturated. Since the second brightness value is negatively correlated with the brightness fusion weight, when the second brightness value is greater than the fourth threshold, the degree to which the second brightness value approaches the saturation value has too much of an impact. Therefore, the brightness fusion weight is set to the minimum preset value.

[0087] The first, second, and third thresholds are all thresholds set for the first luminance value. Luminance values ​​less than the third threshold are all considered first luminance values. These thresholds are used to adjust the rules governing the change of the first luminance value, further improving the accuracy of the luminance fusion weight. Correspondingly, the interval containing the first luminance value is refined into at least three cases: 1) The first luminance value of a certain spectral channel is less than the first threshold, in which case the luminance fusion weight is the minimum preset value; 2) The first luminance value of a certain spectral channel is between the first and second thresholds, in which case the first luminance value and the luminance fusion weight are positively correlated, and this positive correlation can be linear or non-linear; 3) The first luminance value of a certain spectral channel is between the second and third thresholds, in which case the luminance fusion weight is the maximum preset value.

[0088] The third and fourth thresholds are thresholds set for the second brightness value. Besides the case where the second brightness value is greater than the fourth threshold, the second brightness value can also be a brightness value between the third and fourth thresholds. In this case, the second brightness value is negatively correlated with the brightness fusion weight; this negative correlation can be linear or non-linear.

[0089] The minimum preset value is the minimum value of the luminance fusion weight, indicating that the luminance value of this spectral channel has the least impact on fusion. The minimum preset value can be, but is not limited to, 0. The maximum preset value is the maximum value of the luminance fusion weight, indicating that the luminance value of this spectral channel has the greatest impact on fusion. The maximum preset value can be, but is not limited to, 100%.

[0090] For example, the first threshold is 5, the second threshold is 20, the third threshold is 900, the fourth threshold is 950, the minimum preset value is 0, and the maximum preset value is 100%. Therefore, the brightness fusion weight corresponding to the first brightness value less than 5 is 0, the brightness fusion weight corresponding to the first brightness value between 20 and 900 is 100%, and the brightness fusion weight corresponding to the second brightness value greater than 950 is 0.

[0091] In this embodiment, since the first brightness value and the second brightness value are in different brightness ranges and have different variation rules, both have a brightness fusion weight corresponding to the minimum preset value. At the same time, when the first brightness value is between the second threshold and the third threshold, the noise is relatively low and does not affect the brightness ratio between different spectral channels. Therefore, its confidence is directly set to the maximum preset value to ensure accuracy.

[0092] In some embodiments, the fused data of the j-th spectral channel in the i-th region of the first spectral image data and the corresponding luminance fusion weight are processed according to the luminance value of the j-th spectral channel in the i-th region of the second spectral image data and the corresponding luminance fusion weight, to obtain the fused data of the j-th spectral channel in the i-th region, including:

[0093] The fused data of the j-th spectral channel in the i-th region of the first spectral image data, the corresponding luminance fusion weight and gain coefficient fusion weight, and the luminance value of the j-th spectral channel in the i-th region of the second spectral image data are processed to obtain the fused data of the j-th spectral channel in the i-th region.

[0094] Among them, the gain coefficient fusion weight is negatively correlated with the gain coefficient of the brightness value.

[0095] Gain coefficient fusion weights are weight values ​​determined based on the gain coefficient. These weights are used to adjust brightness values, thereby achieving data fusion of two spectral image data sets. The gain coefficient fusion weights are determined based on the gain coefficient of each spectral channel, allowing for adaptive adjustment of the fused data from different spectral channels. The gain coefficient represents the degree of gain applied to brightness values ​​within the image sensor. For the same brightness value, different gain coefficients result in different confidence levels; a higher gain coefficient indicates a lower confidence level for the same brightness value.

[0096] In some embodiments, the fused data of the j-th spectral channel in the i-th region of the first spectral image data, the corresponding brightness fusion weight and gain coefficient fusion weight, and the brightness value, corresponding brightness fusion weight and gain coefficient fusion weight of the j-th spectral channel in the i-th region of the second spectral image data are processed to obtain the fused data of the j-th spectral channel in the i-th region, including:

[0097] The brightness value of the j-th spectral channel in the i-th region of the first spectral image data is mapped to the first brightness fusion weight, and the first gain coefficient fusion weight is determined according to the gain coefficient of the first spectral image data.

[0098] Based on the first luminance fusion weight and the first gain coefficient fusion weight, the luminance value of the j-th spectral channel in the i-th region of the first spectral image data is adjusted to obtain the first weighted luminance value of the j-th spectral channel in the i-th region.

[0099] The brightness value of the j-th spectral channel in the i-th region of the second spectral image data is mapped to the second brightness fusion weight, and the second gain coefficient fusion weight is determined according to the gain coefficient of the second spectral image data.

[0100] Based on the second luminance fusion weight and the second gain coefficient fusion weight, the luminance value of the j-th spectral channel in the i-th region of the second spectral image data is adjusted to obtain the second weighted luminance value of the j-th spectral channel in the i-th region.

[0101] The first weighted brightness value and the second weighted brightness value are summed to obtain the fused data of the j-th spectral channel in the i-th region.

[0102] For example, based on the brightness value, the corresponding brightness fusion weight, and the gain coefficient fusion weight, region 1 (Raw_T1_Block1) has M spectral channels, which is equivalent to providing M sets of pixel values. The expression for fusing the j-th spectral channel of the i-th region is as follows:

[0103]

[0104] Where Channel j is the fusion result of the j-th spectral channel; Pixel_j is the brightness value of the j-th spectral channel; W_DNj is the brightness fusion weight corresponding to the j-th spectral channel; and W_ratioj is the gain coefficient fusion weight corresponding to the j-th spectral channel.

[0105] In this embodiment, the brightness value of each spectral channel has a corresponding brightness fusion weight and gain coefficient fusion weight. Therefore, the influence of the image sensor's confidence in the brightness value is incorporated into the fusion weight process. Thus, through more detailed brightness fusion weight and gain coefficient fusion weight at the local level, fusion data for each spectral channel is formed, thereby ensuring higher accuracy of the fusion data.

[0106] In some embodiments, statistically analyzing regional fusion data to obtain target spectral data includes: statistically analyzing the fusion data of each region according to spectral channels to obtain channel values ​​for various spectral channels; and converting the channel values ​​of various spectral channels into spectral data to obtain target spectral data.

[0107] The channel values ​​of a spectral channel include at least the brightness value, and may also include the processed result of the brightness value. Because spectral channels have attributes such as channel coordinate position, channel distribution, and channel type, the corresponding channel value distribution can be determined when statistically analyzing the channel values ​​of a spectral channel.

[0108] With the channel values ​​of various spectral channels obtained, the channel values ​​of the global level of the image can be represented by the channel values ​​of various spectral channels, and thus spectral data can be formed according to the position information of the corresponding spectral channels.

[0109] In some embodiments, statistical analysis is performed on the fused data of each region according to the spectral channels to obtain the channel values ​​of various spectral channels, including: converting the position of the spectral channels in the fused data of each region into the position in the image hierarchy to obtain the channel values ​​of various spectral channels.

[0110] In some embodiments, statistical analysis is performed on the fused data of each region according to the spectral channels to obtain the channel values ​​of various spectral channels, including: accumulating the brightness values ​​of each spectral channel in the fused regional data to obtain the channel values ​​of various spectral channels.

[0111] In some embodiments, converting the channel values ​​of various spectral channels into spectral data to obtain target spectral data includes: determining the corresponding positions of the channel values ​​of various spectral channels in the spectral image data according to their locations; and associating the channel values ​​of various spectral channels with their corresponding positions to obtain the target spectral data. Thus, the position of the region fusion data is changed to its position in the target spectral data, and through adjusting the positional relationships, a spectral data conversion from a local to a global level is achieved.

[0112] In this embodiment, the fused data for each region is statistically analyzed according to the spectral channels, so that the channel values ​​at the regional level are changed to the channel values ​​at the global level; the channel values ​​of various spectral channels are converted into spectral data, so that the channel values ​​at the global level are restored to the target spectral data.

[0113] In some embodiments, the method further includes: determining target environmental data based on target spectral data; and adjusting the color temperature of the target spectral data based on the target environmental data.

[0114] Target environment data represents the shooting scene of the image sensor. Since different environments correspond to different color temperatures, the appropriate color temperature can be determined based on the target environment data to adjust the color temperature of the target spectral data. Target environment data may include, but is not limited to, an identifier of the shooting scene and the parameters corresponding to that identifier.

[0115] In some embodiments, determining target environment data based on target spectral data includes: determining the shooting environment corresponding to the target spectral data based on a comparison result between the target spectral data and a preset environment threshold; and determining target environment data based on the shooting environment.

[0116] In some embodiments, adjusting the color temperature of target spectral data based on target environmental data includes: determining a corrected color value corresponding to the target environmental data; determining the deviation between the target spectral data and the corrected color value; and adjusting the color temperature of the target spectral data based on the deviation until the deviation reaches a minimum. The deviation may be a color matrix, and correspondingly, the methods for adjusting the color temperature of the target spectral data include, but are not limited to, gradient descent, mapping tables, neural networks, or other optimization methods.

[0117] In some embodiments, adjusting the color temperature of target spectral data based on target environmental data includes: determining a color temperature adjustment coefficient mapped from the target spectral data in a lookup table corresponding to the target environmental data; and adjusting the color temperature of the target spectral data based on the color temperature adjustment coefficient.

[0118] In this embodiment, target environment data is determined based on target spectral data to more accurately determine the shooting environment through multiple spectral channels. Furthermore, color temperature adjustment of the target spectral data based on the target environment data allows for a better match between the color temperature of the target spectral data and the actual shooting scene. Simultaneously, applying color temperature based on the partitioned target spectral data eliminates the need for adaptive adjustments using first and second spectral image data, resulting in less data adjustment and optimizing computational power and consumption. Additionally, multi-channel color temperature allows for more accurate determination of the image partition color temperature, aiding in color reproduction and resolving image data mixing issues.

[0119] In some embodiments, determining target environment data based on target spectral data includes: determining the proportion of spectral channel values ​​in the target spectral data; if a shooting scene is determined to exist based on the proportion of spectral channel values, using the environmental data of the shooting scene as target environment data; if at least two shooting scenes are determined to exist based on the proportion of spectral channel values, determining target environment data based on image detection data corresponding to at least two shooting scenes.

[0120] The spectral channel value ratio includes the difference between channel values ​​of different spectral channels. Based on the spectral channel dimension, the spectral channel value ratio adds a corresponding dimension for scene judgment, allowing for more accurate assessment of environmental data. The spectral channel value ratio may include, but is not limited to, the ratio of channel values ​​for each spectral channel, or the ratio of channel values ​​for some different spectral channels. For example, if the target spectral data has N spectral channels, the ratio of the brightness values ​​of the N spectral channels can be used as the spectral channel value ratio; alternatively, the ratio of the brightness values ​​of Z spectral channels can be used, where Z is a positive integer less than N, and N is a positive integer.

[0121] Shooting scenes include potential scene elements when the image sensor acquires images. Shooting scenes include, but are not limited to: object scenes and weather scenes; object scenes can include locations such as studios, offices, and parks, as well as real-life scenes such as portraits and green plants; weather scenes can include scenes with natural light and artificial light, and natural light can be daytime, dusk, or sunset scenes. Due to the limited number of spectral channels, it is difficult to accurately reflect the characteristics of natural light, therefore, there are cases where the proportion of spectral channel values ​​is determined and there are at least two shooting scenes.

[0122] Image detection data refers to detection data that is in a different dimension from the proportion of spectral channel values. Image detection data can be, but is not limited to, the result of text recognition or image recognition, and can also be audio data or text input.

[0123] In some embodiments, when a shooting scene is determined to exist based on the proportion of spectral channel values, environmental data of the shooting scene is used as target environmental data. This includes: when the confidence level between the proportion of spectral channel values ​​and the environmental features of the shooting scene reaches a preset value, the environmental data representing the shooting scene is used as the target environmental data. Thus, by determining the confidence level using environmental data, the range of options is increased.

[0124] In some embodiments, when a shooting scene is determined to exist based on the proportion of spectral channel values, environmental data of the shooting scene is used as target environmental data. This includes: when the similarity between the proportion of spectral channel values ​​and the environmental features of the shooting scene reaches a preset value, the environmental data representing the shooting scene is used as the target environmental data. Thus, by determining similarity through environmental data, shooting scenes can be directly filtered based on similarity, improving processing efficiency.

[0125] In some embodiments, determining the existence of at least two shooting scenarios based on the proportion of spectral channel values ​​includes: determining the feature difference between the proportion of spectral channel values ​​and the proportion of preset spectral channel values ​​contained in multiple shooting scenarios; and determining, among the multiple shooting scenarios, that the feature difference between at least two shooting scenarios is less than a preset value. Thus, the shooting scenarios are represented by the feature of the proportion of spectral channel values, and used to determine the feature difference, so as to accurately determine the existence of at least two shooting scenarios.

[0126] In some embodiments, determining the existence of at least two shooting scenarios based on the proportion of spectral channel values ​​includes: determining the feature difference between the proportion of spectral channel values ​​and the proportion of preset spectral channel values ​​contained in multiple shooting scenarios; calculating the difference based on the feature difference between at least two shooting scenarios in multiple shooting scenarios to obtain the shooting scenario difference; and determining that the shooting scenario difference is less than a preset value. Therefore, based on the determined feature difference, the feature difference is calculated again to more accurately determine the shooting scenario difference between at least two shooting scenarios, and to more accurately determine the existence of at least two shooting scenarios through the shooting scenario difference.

[0127] Taking the shooting scene of sunset or green plants in the nine spectral channels as an example, since the proportion of sunset and green plants in the nine channels of spectral response is similar, the corresponding image detection data is used for adaptive detection, thereby adjusting the confidence of the two shooting scenes of sunset and green plants, and determining the target environment data by adjusting the results.

[0128] In this embodiment, by adjusting the spectral channel value ratio, the dimensionality of the spectral channels can be further increased, enabling accurate determination of the target shooting scene. Furthermore, to address the issue of insufficient dimensionality of the spectral channels, image detection data corresponding to at least two shooting scenes are used to more accurately determine the target environment data.

[0129] In some embodiments, the method further includes: determining imaging parameters based on the target spectral data.

[0130] Image capture parameters are parameters used for image acquisition. These parameters include, but are not limited to, image signal processing parameters, such as white balance and color reproduction for the captured image. When target spectral data is obtained based on first and second spectral image data acquired by an auxiliary sensor, this target spectral data is used as the image capture parameters for the main sensor.

[0131] The process of determining the photographing parameters can be carried out by adjusting the parameters of the photographed image based on the target spectral data, so as to display the adjusted image.

[0132] Therefore, since the dynamic range of the target spectral data has been improved, the shooting parameters can be determined to be more suitable for the actual scene, so that the captured image is more in line with the real scene.

[0133] In one exemplary embodiment, such as Figure 7 As shown, taking double exposure as an example, the fusion processing method of multispectral partitioning is illustrated. The fusion processing method includes: step 701, acquiring initial spectral image data; step 702, determining that the brightness of the initial spectral image data meets the adjustment conditions; step 703, acquiring first spectral image data; step 704, acquiring second spectral image data; step 705, fusing the first spectral image data and the second spectral image data to obtain regional fusion data; step 706, statistically analyzing the spectral information of the regional fusion data; step 707, restoring the spectral information to target spectral image data; and step 708, performing color temperature adaptation on the target spectral image data.

[0134] The above-mentioned acquisition of initial spectral image data, after determining that the brightness of the initial spectral image data meets the adjustment conditions, involves acquiring first spectral image data and then acquiring second spectral image data, including:

[0135] Acquire initial spectral image data;

[0136] In the initial spectral image data, the number of spectral channels whose brightness values ​​meet the brightness conditions is counted to obtain the number of spectral channels that meet the brightness conditions; if the number of spectral channels that meet the brightness conditions meets the preset quantity condition, then the brightness of the initial spectral image data meets the adjustment condition; the spectral channels whose brightness values ​​meet the brightness conditions include at least one of the first spectral channel and the second spectral channel; the brightness value of the first spectral channel is less than the first threshold value, the brightness value of the second spectral channel is equal to the second threshold value, and the second threshold value is greater than the first threshold value.

[0137] If the brightness of the initial spectral image data meets the adjustment conditions, the acquisition of the first spectral image data and the second spectral image data is performed.

[0138] The aforementioned method of fusing first spectral image data and second spectral image data to generate region fusion data includes:

[0139] The fused data of the j-th spectral channel in the i-th region of the first spectral image data, the corresponding luminance fusion weight and gain coefficient fusion weight, and the luminance value of the j-th spectral channel in the i-th region of the second spectral image data are processed to obtain the fused data of the j-th spectral channel in the i-th region.

[0140] The brightness value includes a first brightness value and a second brightness value in different brightness ranges; the first brightness value is less than the second brightness value; the first brightness value is positively correlated with the brightness fusion weight; the second brightness value is negatively correlated with the brightness fusion weight; when the first brightness value is less than a first threshold, the brightness fusion weight is the minimum preset value; when the first brightness value is greater than a second threshold and the first brightness value is less than a third threshold, the brightness fusion weight is the maximum preset value; when the second brightness value is greater than a fourth threshold, the brightness fusion weight is the minimum preset value; the first threshold, second threshold, third threshold, and fourth threshold are from smallest to largest, and the minimum preset value is less than the maximum preset value.

[0141] Among them, the gain coefficient fusion weight is negatively correlated with the gain coefficient of the brightness value.

[0142] The spectral information of the above-mentioned statistical region fusion data is used to restore the spectral information into target spectral image data, including: statistically analyzing the fusion data of each region according to the spectral channels to obtain the channel values ​​of various spectral channels; and converting the channel values ​​of various spectral channels into spectral data to obtain target spectral data.

[0143] The above-mentioned color temperature adaptation of the target spectral image data includes:

[0144] Determine the proportion of spectral channel values ​​in the target spectral data;

[0145] When a shooting scene is determined to exist based on the proportion of spectral channel values, the environmental data of the shooting scene is used as the target environmental data.

[0146] If at least two shooting scenarios are determined based on the proportion of spectral channel values, target environment data is determined based on the image detection data corresponding to the at least two shooting scenarios.

[0147] Based on the target environment data, the color temperature of the target spectral data is adjusted.

[0148] Therefore, by introducing image fusion functionality into the partitioned multispectral mode, target spectral image data can be generated through multiple exposures, thereby improving the dynamic range of the partitioned multispectral mode and better adapting to the operation of RGB cameras. Furthermore, the multiple exposure method used in partitioned multispectral mode allows for setting different exposure times (T1, T2, T3, etc.) for each partition.

[0149] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0150] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.

[0151] In one exemplary embodiment, such as Figure 8 As shown, an image processing apparatus is provided, comprising:

[0152] Acquisition module 802 is used to acquire first spectral image data and second spectral image data; the brightness of the first spectral image data and the second spectral image data are different;

[0153] The fusion module 804 is used to perform image fusion on the region of the first spectral image data and the corresponding region in the second spectral image data to obtain region fusion data;

[0154] The restoration module 806 is used to statistically analyze the fused data of the region to obtain the target spectral data.

[0155] In one embodiment, the fusion module 804 is configured to:

[0156] The fused data of the j-th spectral channel in the i-th region of the first spectral image data and the corresponding luminance fusion weight are processed together with the luminance value of the j-th spectral channel in the i-th region of the second spectral image data to obtain the fused data of the j-th spectral channel in the i-th region; i is a positive integer and j is a positive integer.

[0157] In one embodiment, the brightness value includes a first brightness value and a second brightness value in different brightness ranges; the first brightness value is less than the second brightness value; the first brightness value is positively correlated with the brightness fusion weight; and the second brightness value is negatively correlated with the brightness fusion weight.

[0158] In one embodiment, when the first brightness value is less than a first threshold, the brightness fusion weight is a minimum preset value; when the first brightness value is greater than a second threshold and less than a third threshold, the brightness fusion weight is a maximum preset value; when the second brightness value is greater than a fourth threshold, the brightness fusion weight is the minimum preset value; the first threshold, the second threshold, the third threshold, and the fourth threshold are in ascending order, and the minimum preset value is less than the maximum preset value.

[0159] In one embodiment, the fusion module 804 is configured to:

[0160] The fused data of the j-th spectral channel in the i-th region of the first spectral image data, the corresponding brightness fusion weight and gain coefficient fusion weight, and the brightness value of the j-th spectral channel in the i-th region of the second spectral image data are processed to obtain the fused data of the j-th spectral channel in the i-th region.

[0161] The gain coefficient fusion weight is negatively correlated with the gain coefficient of the brightness value.

[0162] In one embodiment, the restoration module 806 is configured to:

[0163] According to the spectral channels, the fused data of each region is statistically analyzed to obtain the channel values ​​of each spectral channel;

[0164] The channel values ​​of the various spectral channels are converted into spectral data to obtain the target spectral data.

[0165] In one embodiment, the acquisition module 802 is used for:

[0166] Acquire initial spectral image data;

[0167] If the brightness of the initial spectral image data meets the adjustment conditions, the step of acquiring the first spectral image data and the second spectral image data is performed.

[0168] In one embodiment, the acquisition module 802 is used for:

[0169] In the initial spectral image data, the number of spectral channels whose brightness values ​​meet the brightness conditions is counted to obtain the number of spectral channels that meet the brightness conditions.

[0170] If the number of spectral channels that meet the brightness condition meets the preset quantity condition, then the brightness of the initial spectral image data meets the adjustment condition.

[0171] In one embodiment, the spectral channel whose brightness value satisfies the brightness condition includes at least one of a first spectral channel and a second spectral channel; the brightness value of the first spectral channel is less than a first threshold value, the brightness value of the second spectral channel is equal to a second threshold value, and the second threshold value is greater than the first threshold value.

[0172] In one embodiment, the restoration module 806 is configured to:

[0173] Based on the target spectral data, determine the target environmental data;

[0174] Based on the target environment data, the color temperature of the target spectral data is adjusted.

[0175] In one embodiment, the restoration module 806 is configured to:

[0176] Determine the proportion of spectral channel values ​​in the target spectral data;

[0177] If a shooting scene is determined to exist based on the ratio of the spectral channel values, the environmental data of the shooting scene is used as the target environmental data.

[0178] If at least two shooting scenarios are determined based on the ratio of the spectral channel values, the target environment data is determined based on the image detection data corresponding to the at least two shooting scenarios.

[0179] In one embodiment, the acquisition module 802 is used for:

[0180] When the exposure is performed for the first exposure time, the first spectral image data is obtained;

[0181] When the exposure is performed for the second exposure time, the second spectral image data is obtained;

[0182] The first exposure duration and the second exposure duration are different.

[0183] In one embodiment, the apparatus further includes a parameter determination module, the parameter determination module being configured to:

[0184] Based on the target spectral data, determine the imaging parameters.

[0185] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the electronic device in hardware form or independent of it, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to each module.

[0186] In one exemplary embodiment, an electronic device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, this electronic device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the electronic device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the electronic device, or external keyboards, touchpads, or mice, etc.

[0187] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0188] In one embodiment, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0189] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0190] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0191] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0192] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0193] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0194] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method, characterized by, The method includes: Acquire first spectral image data and second spectral image data; the brightness of the first spectral image data and the second spectral image data are different. Image fusion is performed on the region in the first image data and the corresponding region in the second image data to obtain region fusion data; The target spectral data is obtained by statistically analyzing the fused data of the region.

2. The method of claim 1, wherein, The step of fusing the regions in the first spectral image data and the corresponding regions in the second spectral image data to obtain region fusion data includes: The fused data of the j-th spectral channel in the i-th region of the first spectral image data and the corresponding luminance fusion weight are processed together with the luminance value of the j-th spectral channel in the i-th region of the second spectral image data to obtain the fused data of the j-th spectral channel in the i-th region; i is a positive integer and j is a positive integer.

3. The method of claim 2, wherein, The brightness value includes a first brightness value and a second brightness value in different brightness ranges; the first brightness value is less than the second brightness value; the first brightness value is positively correlated with the brightness fusion weight; and the second brightness value is negatively correlated with the brightness fusion weight.

4. The method of claim 3, wherein, When the first brightness value is less than the first threshold, the brightness fusion weight is the minimum preset value; when the first brightness value is greater than the second threshold and less than the third threshold, the brightness fusion weight is the maximum preset value. When the second brightness value is greater than the fourth threshold, the brightness fusion weight is the minimum preset value; the first threshold, the second threshold, the third threshold and the fourth threshold are in ascending order, and the minimum preset value is less than the maximum preset value.

5. The method of claim 3, wherein, The step of processing the luminance value and corresponding luminance fusion weight of the j-th spectral channel in the i-th region of the first spectral image data, and the luminance value and corresponding luminance fusion weight of the j-th spectral channel in the i-th region of the second spectral image data to obtain the fused data of the j-th spectral channel in the i-th region includes: The fused data of the j-th spectral channel in the i-th region of the first spectral image data, the corresponding brightness fusion weight and gain coefficient fusion weight, and the brightness value of the j-th spectral channel in the i-th region of the second spectral image data are processed to obtain the fused data of the j-th spectral channel in the i-th region. The gain coefficient fusion weight is negatively correlated with the gain coefficient of the brightness value.

6. The method of claim 1, wherein, The statistical analysis of the fused regional data yields target spectral data, including: According to the spectral channels, the fused data of each region is statistically analyzed to obtain the channel values ​​of each spectral channel; The channel values ​​of the various spectral channels are converted into spectral data to obtain the target spectral data.

7. The method of claim 1, wherein, The method further includes: Acquire initial spectral image data; If the brightness of the initial spectral image data meets the adjustment conditions, the step of acquiring the first spectral image data and the second spectral image data is performed.

8. The method according to claim 7, characterized in that, The determination step for whether the brightness of the initial spectral image data meets the adjustment conditions includes: In the initial spectral image data, the number of spectral channels whose brightness values ​​meet the brightness conditions is counted to obtain the number of spectral channels that meet the brightness conditions. If the number of spectral channels that meet the brightness condition meets the preset quantity condition, then the brightness of the initial spectral image data meets the adjustment condition.

9. The method according to claim 8, characterized in that, The spectral channels whose brightness values ​​meet the brightness conditions include at least one of a first spectral channel and a second spectral channel; the brightness value of the first spectral channel is less than a first critical value, the brightness value of the second spectral channel is equal to a second critical value, and the second critical value is greater than the first critical value.

10. The method according to claim 1, characterized in that, The method further includes: Based on the target spectral data, determine the target environmental data; Based on the target environment data, the color temperature of the target spectral data is adjusted.

11. The method according to claim 10, characterized in that, The step of determining the target environmental data based on the target spectral data includes: Determine the proportion of spectral channel values ​​in the target spectral data; If a shooting scene is determined to exist based on the ratio of the spectral channel values, the environmental data of the shooting scene is used as the target environmental data. If at least two shooting scenarios are determined based on the ratio of the spectral channel values, the target environment data is determined based on the image detection data corresponding to the at least two shooting scenarios.

12. The method according to claim 1, characterized in that, The acquisition of the first spectral image data and the second spectral image data includes: When the exposure is performed for the first exposure time, the first spectral image data is obtained; When the exposure is performed for the second exposure time, the second spectral image data is obtained; The first exposure duration and the second exposure duration are different.

13. The method according to claim 1, characterized in that, The method further includes: Based on the target spectral data, determine the imaging parameters.

14. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire first spectral image data and second spectral image data; the brightness of the first spectral image data and the second spectral image data are different; The fusion module is used to perform image fusion on the region in the first spectral image data and the corresponding region in the second spectral image data to obtain region fusion data; The restoration module is used to statistically analyze the fused data of the region to obtain the target spectral data.

15. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 13.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 13.

17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 13.

Images