Multi - spectral Image Enhancement Method, Device, Computer Equipment and Storage Medium

Abstract

The present application relates to a multi-spectral image enhancement method, apparatus, computer device, and storage medium. The method includes: obtaining low-frequency information and high-frequency information of a target image; statistically obtaining brightness level information of a first information region and brightness level information of a second information region based on the low-frequency information, and respectively obtaining a first histogram and a second histogram; according to the first histogram and the second histogram, performing stretching processing on the first histogram, and calculating brightness mapping information according to the stretched first histogram; performing enhancement processing on the low-frequency information according to the brightness mapping information to obtain low-frequency enhanced information; reconstructing the brightness information of the target image according to the low-frequency enhanced information and the high-frequency information to obtain brightness reconstruction information; and combining the brightness reconstruction information and the color information of the visible light image to obtain a multi-spectral fusion image. Through the present application, the interference of noise in the low-frequency information on the low-frequency enhanced information is eliminated, and the quality of the multi-spectral fusion image is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and particularly to a multi-spectral image enhancement method, apparatus, computer device, and storage medium. Background Art

[0002] During a medical operation, it is often necessary to observe the target site and perform processing with the aid of a shooting and display system, and the quality of medical images and videos has become one of the concerns in the medical system. In a multi-spectral endoscope, visible light can observe a normal view, while fluorescence can better detect diseased tissues that are invisible to the naked eye. Multi-spectral fusion combines visible light and non-visible light images and displays them on one picture, which can help surgeons initially locate and examine during the operation.

[0003] For multi-spectral fusion, visible light and non-visible light are "superimposed" together. Since it is the fusion of different spectral bands, non-visible light has a relatively narrow spectrum, and when reflected onto the image, there will be a problem of low contrast. After being fused with visible light, it will also affect the contrast of the fused image. During the process of enhancing the contrast of the fused image, it is easily affected by interference factors, such as being affected by noise, resulting in poor enhancement effect of the contrast.

[0004] Currently, for the problem of low quality of multi-spectral fusion images in related technologies, no effective solution has been proposed. Summary of the Invention

[0005] Based on this, in view of the above technical problems, it is necessary to provide a multi-spectral image enhancement method, apparatus, computer device, and storage medium that can improve the quality of multi-spectral fusion images.

[0006] In a first aspect, the present application provides a multi-spectral image enhancement method, including:

[0007] Obtaining low-frequency information and high-frequency information of a target image; wherein, the target image includes a visible light image and a non-visible light image;

[0008] Based on the low-frequency information, statistically obtaining the brightness level information of a first information region and the brightness level information of a second information region, and respectively obtaining a first histogram and a second histogram; wherein, the average gradient value of the first information region is greater than the average gradient value of the second information region;

[0009] According to the first histogram and the second histogram, performing stretching processing on the first histogram, and calculating brightness mapping information according to the stretched first histogram;

[0010] Enhancing the low-frequency information according to the brightness mapping information to obtain low-frequency enhanced information;

[0011] Reconstruct the luminance information of the target image according to the low-frequency enhancement information and the high-frequency information to obtain luminance reconstruction information;

[0012] Merge the luminance reconstruction information and the color information of the visible light image to obtain a multispectral fusion image.

[0013] In one embodiment, stretching the first histogram according to the first histogram and the second histogram includes:

[0014] Calculate a stretching value according to the first histogram and the second histogram;

[0015] Divide the first histogram into multiple different luminance regions and determine the luminance ratios corresponding to the different luminance regions;

[0016] In each of the different luminance regions, update the first histogram according to the stretching value and the corresponding luminance ratio to obtain the stretched first histogram.

[0017] In one embodiment, based on the low-frequency information, statistically obtain the luminance level information of the first information region and the luminance level information of the second information region to obtain a first histogram and a second histogram respectively, including:

[0018] In the low-frequency information, statistically obtain the gradient value of each pixel and compare the gradient value with a preset threshold;

[0019] Divide the first pixels with the gradient value greater than the preset threshold into the first information region, and divide the second pixels with the gradient value not greater than the preset threshold into the second information region;

[0020] In the first information region, statistically obtain the number of the first pixels at each luminance level to obtain the first histogram, and in the second information region, statistically obtain the number of the second pixels at each luminance level to obtain the second histogram.

[0021] In one embodiment, calculating a stretching value according to the first histogram and the second histogram includes:

[0022] Calculate the first average information intensity of the first information region according to the gradient values of the first pixels and the total number of the first pixels;

[0023] Calculate the second average information intensity of the second information region according to the gradient values of the second pixels and the total number of the second pixels;

[0024] Calculate a stretching parameter based on the first average information intensity, the second average information intensity, and the number of the first pixels;

[0025] Calculate a stretching value based on the stretching parameter.

[0026] In one embodiment, calculating a stretching parameter based on the first average information intensity, the second average information intensity, and the number of the first pixels includes:

[0027] Calculate the root mean square of the first average information intensity and the second average information intensity;

[0028] Calculate a stretching coefficient based on the first average information intensity and the root mean square;

[0029] Calculate a stretching strength based on the number of the first pixels and the root mean square.

[0030] In one embodiment, calculating a stretching value based on the stretching parameter includes:

[0031] Determine a weight coefficient according to the stretching coefficient;

[0032] Perform a weighted sum of the stretching strength and the values of each brightness level in the first histogram according to the weight coefficient to obtain the stretching value.

[0033] In one embodiment, calculating brightness mapping information based on the first histogram after stretching processing includes:

[0034] Determine a target histogram in the first histogram; wherein, the target histogram includes a plurality of the first histograms arranged in the first n brightness levels in descending order of brightness level; wherein, n is a positive integer;

[0035] Divide the sum value of the target histogram by the sum value of all the first histograms, and then multiply the obtained quotient by the maximum brightness level value to obtain a brightness enhancement level value corresponding to each brightness input level value.

[0036] In one embodiment, reconstructing the brightness information of the target image based on the low-frequency enhancement information and the high-frequency information to obtain brightness reconstruction information includes:

[0037] Add the low-frequency enhancement information of the last layer to the high-frequency information of the same layer to obtain the low-frequency enhancement information of the upper layer;

[0038] Add the low-frequency enhancement information of the upper layer to the high-frequency information of the same layer to obtain the low-frequency enhancement information of the upper-upper layer, and so on, until the low-frequency enhancement information of the first layer is added to the high-frequency information of the same layer to obtain the luminance reconstruction information.

[0039] In one embodiment, the low-frequency information includes visible low-frequency information and non-visible low-frequency information; alternatively, the low-frequency information includes low-frequency fusion information obtained by fusing visible low-frequency information and non-visible low-frequency information.

[0040] The high-frequency information includes visible high-frequency information and non-visible high-frequency information; alternatively, the high-frequency information includes high-frequency fusion information obtained by fusing visible high-frequency information and non-visible high-frequency information.

[0041] In one embodiment, obtaining the low-frequency information and high-frequency information of the target image includes:

[0042] Obtain the luminance information of the target image;

[0043] On the basis of maintaining the original resolution of the target image, perform frequency-domain decomposition on the luminance information to respectively obtain visible low-frequency information and visible high-frequency information, as well as non-visible low-frequency information and non-visible high-frequency information.

[0044] In a second aspect, the present application provides a multi-spectral image enhancement device, and the device includes:

[0045] An acquisition module for acquiring the low-frequency information and high-frequency information of a target image; wherein, the target image includes a visible light image and a non-visible light image;

[0046] A statistics module for statistically obtaining the luminance level information of a first information region and the luminance level information of a second information region based on the low-frequency information, respectively obtaining a first histogram and a second histogram; wherein, the average gradient value of the first information region is greater than the average gradient value of the second information region;

[0047] A stretching module for stretching the first histogram according to the first histogram and the second histogram, and calculating luminance mapping information according to the stretched first histogram;

[0048] An enhancement module for enhancing the low-frequency information according to the luminance mapping information to obtain low-frequency enhancement information;

[0049] A reconstruction module for reconstructing the luminance information of the target image according to the low-frequency enhancement information and the high-frequency information to obtain luminance reconstruction information;

[0050] A merging module, configured to merge the luminance reconstruction information and the color information of the visible light image to obtain a multispectral fusion image.

[0051] In a third aspect, the present application provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the method described in the first aspect above are implemented.

[0052] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0053] For the above multispectral image enhancement method, device, computer device, and storage medium, when calculating the histogram, the first information region and the second information region are distinguished, the first histogram and the second histogram are respectively statistically calculated based on these two regions, and based on the first histogram and the second histogram, stretching processing is performed on the first histogram. In this way, during stretching, it is possible to mainly stretch the texture, avoid noise, and prevent the noise in the low-frequency information from being wrongly stretched, thereby eliminating the interference of the noise in the low-frequency information on the low-frequency enhancement information and improving the quality of the multispectral fusion image. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a hardware structure block diagram of a terminal for the multispectral image enhancement method in an embodiment;

[0055] Figure 2 It is a flowchart of the multispectral image enhancement method in an embodiment;

[0056] Figure 3 It is a schematic diagram of luminance information reconstruction in an embodiment;

[0057] Figure 4 It is a schematic diagram of a target image in an embodiment;

[0058] Figure 5 It is a schematic diagram of Laplacian pyramid decomposition in the related art;

[0059] Figure 6 It is a schematic diagram of frequency domain decomposition in an embodiment;

[0060] Figure 7 It is a flowchart of the multispectral image enhancement method in another embodiment;

[0061] Figure 8 It is a structure block diagram of the multispectral image enhancement device in an embodiment;

[0062] Figure 9 It is an internal structure diagram of a computer device in an embodiment. Detailed Implementation Manner

[0063] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] Unless otherwise defined, the technical terms or scientific terms involved in the present application shall have the general meaning understood by those with ordinary skills in the technical field to which the present application belongs. In the present application, words such as "a", "one", "kind", "the", "these" and the like do not indicate a limitation in quantity, and they can be singular or plural. The terms "including", "comprising", "having" and any variants thereof involved in the present application are intended to cover non-exclusive inclusion; for example, a process, method, system, product or device including a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent in these processes, methods, products or devices. The terms "connected", "coupled" and the like involved in the present application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly connected. The "plurality" involved in the present application means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships may exist. For example, "A and / or B" may represent: A exists alone, A and B exist simultaneously, and B exists alone. Usually, the character " / " indicates that the associated objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in the present application are only used to distinguish similar objects and do not represent a specific sorting of the objects.

[0065] The method embodiment provided in this embodiment can be executed on a terminal, a computer or a similar computing device. For example, it runs on a terminal. Figure 1 It is a hardware structure block diagram of a terminal of a multi-spectral image enhancement method according to an embodiment of the present application. As Figure 1 shown, the terminal may include one or more ( Figure 1 only one is shown in Figure 1 processors 101 for storing data and a memory 102. Among them, the processor 101 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may further include a transmission device 103 for communication functions and an input / output device 104. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above terminal. For example, the terminal may further include more or fewer components than Figure 1 shown inFigure 1 The different configurations shown.

[0066] The memory 102 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the multi-spectral image enhancement method in this embodiment. The processor 101 executes various functional applications and data processing by running the computer program stored in the memory 102, that is, implements the above method. The memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory 102 may further include a memory remotely disposed relative to the processor 101, and these remote memories can be connected to the terminal through a network. Examples of the above network include but are not limited to the Internet, enterprise intranet, local area network, mobile communication network, and combinations thereof.

[0067] The transmission device 103 is used to receive or send data via a network. The above network includes a wireless network provided by a communication provider of the terminal. In one instance, the transmission device 103 includes a network adapter (abbreviated as NIC), which can be connected to other network devices through a base station and thus can communicate with the Internet. In one instance, the transmission device 103 can be a radio frequency (abbreviated as RF) module, which is used to communicate with the Internet wirelessly.

[0068] In the traditional multi-spectral fusion method, due to the introduction of non-visible spectral brightness, there are image quality problems such as the overall image being hazy and having a low contrast in the fused image. The related technology provides a multi-spectral image enhancement method, the key design point of which is to use the method of histogram stretching on the input image to obtain a high-exposure image, then invert the high-exposure image to obtain a low-exposure image, and then use the Laplacian pyramid to fuse the high-exposure and low-exposure images to obtain the final enhanced image. However, this method directly performs histogram stretching on the original image, is easily interfered by noise, resulting in poor quality of the high-exposure image, and ultimately resulting in poor enhancement effect of the fused image.

[0069] Based on the analysis of the above situation, in one embodiment, as Figure 2 shown, a multi-spectral image enhancement method is provided. Taking the case where this method is applied to the Figure 1 terminal as an example, it includes the following steps:

[0070] Step S101, obtaining the low-frequency information and high-frequency information of the target image; wherein, the target image includes a visible light image and a non-visible light image.

[0071] Taking endoscope detection as an example, the target image includes visible light images and non-visible light images (fluorescent images) collected by the endoscope. Among them, the visible light images are used to provide an intuitive view, and the non-visible light images are used to enhance the contrast and help discover detailed information that may be missed.

[0072] Step S101 can be achieved through frequency domain decomposition. The methods include: performing multiple frequency domain filtering processes on the target image to decompose the luminance information into different frequency domain information of multiple layers. Dividing the image into component information in different frequency domains helps to perform targeted enhancement and fusion processing in each frequency domain information. The filters used in the frequency domain decomposition method include, but are not limited to, low-pass filters such as Gaussian and Butterworth, or edge-preserving filters such as bilateral filtering, guided filtering, weighted least squares filter, non-uniform local filter, double exponential edge smoothing filter, selective blur, and surface filter. Among them, the edge-preserving filter refers to a type of filter that preserves the edge information of the image during the filtering process. If f(y) represents performing a filtering operation on the input luminance image y, the low-frequency information obtained by 1-time decomposition is f(y), and the high-frequency information is y - f(y); the low-frequency information obtained by 2-time decomposition is f(f(y)), and the high-frequency information is f(y) - f(f(y)); the low-frequency information obtained by 3-time decomposition is f(f(f(y))), and the high-frequency information is f(f(y)) - f(f(f(y))). Finally, 1 low-frequency information f(f(f(y))) and 3 high-frequency information are respectively y - f(y), f(y) - f(f(y)), and f(f(y)) - f(f(f(y))).

[0073] In some embodiments, before performing step S101, color space conversion processing can also be performed on the target image. The color space conversion processing includes: converting the visible light image and the non-visible light image into a specified color space, and this color space needs to have a distinction between luminance and color information. For example, color spaces such as YUV, YCbCr, HSV, and Lab. Among them, in YUV, Y is the luminance information and UV is the color information; in YCbCr, Y is the luminance information and CbCr is the color information; in HSV, V is the luminance information and HS is the color information; in Lab, L is the luminance information and ab is the color information. This embodiment is applicable to any of the above color spaces.

[0074] Step S102, based on the low-frequency information, statistically obtain the luminance level information of the first information region and the luminance level information of the second information region, and respectively obtain the first histogram and the second histogram; wherein, the average gradient value of the first information region is greater than the average gradient value of the second information region.

[0075] In a histogram, the abscissa represents the brightness level (for example, the brightness level can take values in the range of 0 - X, where X can be the maximum pixel value of the target image or other preset values), and the ordinate represents the number of pixels corresponding to the brightness level. The larger the average gradient value, the richer the texture information. The average gradient value of the first information region is greater than that of the second information region, indicating that the texture information of the first information region is richer than that of the second information region.

[0076] The low-frequency information includes visible low-frequency information (i.e., the low-frequency information of the visible light image) and non-visible low-frequency information (i.e., the low-frequency information of the non-visible light image); alternatively, the low-frequency information includes low-frequency fusion information obtained by fusing visible low-frequency information and non-visible low-frequency information. The high-frequency information includes visible high-frequency information (i.e., the high-frequency information of the visible light image) and non-visible high-frequency information (i.e., the high-frequency information of the non-visible light image); alternatively, the high-frequency information includes high-frequency fusion information obtained by fusing visible high-frequency information and non-visible high-frequency information.

[0077] When statistically analyzing the histogram based on the low-frequency information, it is based on the low-frequency information of the last layer after frequency domain decomposition. Specifically, the low-frequency information of the last layer of the visible light image and the non-visible light image can be statistically analyzed separately first, and then the statistical results of the two are combined. Alternatively, the low-frequency information of the last layer of the visible light image and the non-visible light image can be fused first, and then the fused low-frequency information of the last layer is statistically analyzed. This application does not make any restrictions.

[0078] Step S103: According to the first histogram and the second histogram, perform stretching processing on the first histogram, and calculate the brightness mapping information based on the stretched first histogram.

[0079] When performing stretching processing on the first histogram according to the first histogram and the second histogram, it can be to first calculate the stretching value based on the first histogram and the second histogram, and then update the ordinate value of the first histogram to the corresponding stretching value.

[0080] After performing stretching processing on the first histogram, calculate the brightness mapping information from the stretched and updated first histogram. Determine the target histogram in the first histogram; where the target histogram includes multiple first histograms arranged in the order of brightness levels from high to low, and the first n brightness levels; where n is a positive integer; divide the sum value of the target histogram by the sum value of all the first histograms, and then multiply the obtained quotient by the maximum brightness level value to obtain the brightness enhancement level value corresponding to each brightness input level value.

[0081] The sum value of the target histogram refers to the accumulation of the values of the target histogram, and the sum of the first histogram of the first n brightness levels is calculated. Divide the sum value of the target histogram by the sum value of all the first histograms, and then multiply the obtained quotient by the maximum brightness level value to obtain the brightness enhancement level value corresponding to each brightness input level value, which is the brightness mapping information. Among them, the maximum brightness level value can be determined by the bit width of the target image. If the bit width is b, the maximum brightness level value is , and the range of the brightness level n is . Among them, is also the maximum pixel value of the target image. Exemplarily, assume that the bit width of the target image is 8 and its maximum brightness level value is 255, then the range of the brightness level n is [0, 255].

[0082] Step S104: Enhance the low-frequency information according to the brightness mapping information to obtain low-frequency enhanced information.

[0083] The brightness mapping information includes two types of data, namely the input brightness level value (i.e., the original brightness level value in the target image) and the enhanced brightness level value. Applying the brightness mapping information to the input low-frequency information will obtain the enhanced low-frequency information.

[0084] Step S105: Reconstruct the brightness information of the target image according to the low-frequency enhanced information and the high-frequency information to obtain brightness reconstruction information.

[0085] Brightness information reconstruction is the inverse process of multi-layer frequency domain decomposition: Add the low-frequency enhanced information of the last layer to the high-frequency information of the same layer to obtain the low-frequency enhanced information of the upper layer; Add the low-frequency enhanced information of the upper layer to the high-frequency information of the same layer to obtain the low-frequency enhanced information of the upper upper layer, and so on, until the low-frequency enhanced information of the first layer is added to the high-frequency information of the same layer to obtain the brightness reconstruction information, that is, the enhanced brightness information.

[0086] Figure 3 A schematic diagram of brightness information reconstruction is provided, as Figure 3 shown. Assume that the target image initially undergoes 3 times of frequency domain decomposition. Then, during brightness information reconstruction, start merging layer by layer from the low-frequency information of the last layer. First, add the low-frequency enhanced information of the 3rd layer to the high-frequency information of the 3rd layer to obtain the low-frequency information of the 2nd layer, then add it to the high-frequency information of the 2nd layer to obtain the low-frequency information of the 1st layer, and then add it to the high-frequency information of the 1st layer to obtain the final brightness reconstruction information.

[0087] Step S106: Merge the brightness reconstruction information and the color information of the visible light image to obtain a multi-spectral fusion image.

[0088] The luminance reconstruction information and the color information of the visible light image can be directly merged, or the color information of the visible light image can be adjusted first and then the adjusted color information is merged with the luminance reconstruction information. Among them, color adjustment includes, but is not limited to, color denoising, color mapping, color enhancement and other processes.

[0089] For the above steps S101 to S106, if the original histogram is directly stretched, it is difficult to distinguish texture and noise during stretching, and the noise will inevitably be stretched as texture. Therefore, in this embodiment, when calculating the histogram, the first information region and the second information region are distinguished, the first histogram and the second histogram are respectively counted based on these two regions, and based on the first histogram and the second histogram, the first histogram is stretched. In this way, during stretching, it is possible to mainly stretch the texture, avoid noise, and prevent the noise in the low-frequency information from being wrongly stretched, thereby eliminating the interference of the noise in the low-frequency information on the low-frequency enhancement information and improving the quality of the multi-spectral fusion image.

[0090] The related technology provides another endoscopic image enhancement method. It obtains the base layer image by performing a guided filter on the input image, subtracts it from the original image to obtain the detail layer image, performs block histogram stretching (CLAHE) on the base layer image and non-local means filtering (NLM) on the detail layer image respectively, performs segmented gain processing on the denoised detail layer, and finally adds the enhanced base layer and the processed detail layer to obtain the final enhanced image to improve the details in the fusion image. This scheme can avoid the problem of enhancing noise by performing block contrast enhancement on the decomposed low-frequency layer, but the block contrast is easily affected by the over-bright and over-dark information in the block, and there will be a problem of over-strong regional stretching.

[0091] Based on the analysis of the above situation, in one embodiment, in step S103, according to the first histogram and the second histogram, the stretching process of the first histogram can also be realized by the following method:

[0092] According to the first histogram and the second histogram, a stretching value is calculated; the first histogram is divided into multiple different brightness regions, and the brightness ratios corresponding to the different brightness regions are determined; within each different brightness region, according to the stretching value and the corresponding brightness ratio, the first histogram is updated to obtain the stretched first histogram.

[0093] In this embodiment, a preset value can be set. The first histogram is divided into two or more different brightness regions according to the preset value, and each different brightness region corresponds to a different brightness ratio. Within each different brightness region, the first histogram is stretched according to the stretching value and the corresponding brightness ratio, so as to achieve sub-region and sub-intensity stretching of the first histogram, ensuring that the enhanced low-frequency information is not affected by overexposed or underexposed regions. In addition, it should be noted that the histogram itself is arranged according to brightness levels (the abscissa is the brightness level, and the ordinate is the number of pixels belonging to that brightness level). The comfortable viewing area of the human eye is concentrated in the medium-brightness region. If only the medium-brightness region is considered for stretching, the brightness information of other regions will be compressed, reducing the image quality. Therefore, in this implementation, the first histogram is subdivided into multiple regions before stretching, ensuring the quality of the stretched image.

[0094] Exemplarily, the first histogram can be first divided into a high-brightness region, a medium-brightness region, and a low-brightness region according to the preset value, and then the stretched first histogram is calculated by combining the stretching value and the corresponding brightness ratio. Among them, stretching the value of each brightness level means updating the first histogram to the value obtained by multiplying the stretching value by the preset brightness ratio of each region. Specifically, traverse all brightness levels in the first histogram and stretch them in the following 3 cases:

[0095] (1) For the low-brightness region: If the brightness level is less than or equal to the low-brightness preset value, the first histogram is updated to the stretching value multiplied by the low-brightness ratio;

[0096] (2) For the high-brightness region: If the brightness level is greater than or equal to the high-brightness preset value, the first histogram is updated to the stretching value multiplied by the high-brightness ratio;

[0097] (3) For the medium-brightness region: If the brightness level is greater than the low-brightness preset value and less than the high-brightness preset value, the first histogram is updated to the stretching value.

[0098] The low-brightness ratio and the high-brightness ratio can be preset by the user to be between (0, 1) according to the usage environment. For example, the low-brightness ratio is 0.3 and the high-brightness ratio is 0.6. The role of this ratio is to control the enhancement amplitude of the low-brightness region and the high-brightness region. The smaller the brightness ratio, the greater the enhancement amplitude.

[0099] In one embodiment, in the above step S102, based on the low-frequency information, the brightness level information of the first information region and the brightness level information of the second information region are statistically obtained, and the first histogram and the second histogram are respectively obtained. This can be achieved by the following method:

[0100] In the low-frequency information, the gradient value of each pixel is statistically calculated and compared with a preset threshold; the first pixels with gradient values greater than the preset threshold are divided into a first information region, and the second pixels with gradient values not greater than the preset threshold are divided into a second information region; the number of each first pixel at each brightness level is statistically calculated in the first information region to obtain a first histogram, and the number of each second pixel at each brightness level is statistically calculated in the second information region to obtain a second histogram.

[0101] Further, according to the first histogram and the second histogram, a stretching value can be calculated, which can be achieved by the following method:

[0102] According to the gradient values of each first pixel and the total number of first pixels, the first average information intensity of the first information region is calculated; according to the gradient values of each second pixel and the total number of second pixels, the second average information intensity of the second information region is calculated; according to the first average information intensity, the second average information intensity, and the number of first pixels, a stretching parameter is calculated; according to the stretching parameter, a stretching value is calculated.

[0103] In this embodiment, the gradient value of a pixel can be the gradient value obtained under the gradient information in a certain dimension of the pixel, or the gradient value obtained by taking the average of the gradient information in several dimensions of the pixel. The more dimensions of the gradient information considered, the more accurate the pixel division will be. Exemplarily, in the target image, the gradient value obtained by taking the average of the gradient information at each pixel position in the three dimensions of horizontal, vertical, and diagonal is statistically calculated, and then the gradient value is compared with the preset threshold, which is divided into the following two cases:

[0104] (1) If the gradient value is greater than the preset threshold, the number corresponding to the pixel brightness level in the first histogram is incremented by 1, and the gradient value is accumulated as the first information intensity, and the total number of first pixels is incremented by 1.

[0105] (2) If the gradient value is less than or equal to the preset threshold, the gradient value is accumulated as the second information intensity, and the total number of second pixels is incremented by 1.

[0106] Traverse the low-frequency information, obtain the first histogram according to step 1), divide the first information intensity by the accumulated total number of first pixels to obtain the first average information intensity, obtain the second histogram according to step 2), and divide the second information intensity by the accumulated total number of second pixels to obtain the second average information intensity. Among them, the preset threshold can be an empirical value, for example, set to 20.

[0107] The following is the calculation formula for the average gradient value:

[0108]

[0109] Figure 4It is a schematic diagram of the target image, as Figure 4 shown, where p(m,n) is the pixel value of the low-frequency information at the m-th column and n-th row, Gh(m,n) is the horizontal gradient value at the m-th column and n-th row, Gv(m,n) is the vertical gradient value at the m-th column and n-th row, Gd(m,n) is the diagonal gradient value at the m-th column and n-th row, and G(m,n) is the average gradient value at the m-th column and n-th row.

[0110] The first histogram hist is initialized to 0, and its calculation method is that when G(m,n) is greater than thr1, hist(p(m,n)) is incremented by 1. Where thr1 is a preset threshold, which is set to 15 in general scenarios.

[0111] The first average information intensity S mean is the mean value of G(m,n) that meets G(m,n) > thr1.

[0112] The second average information intensity T mean is the mean value of G(m,n) that meets G(m,n) ≤ thr1.

[0113] In one embodiment, according to the first average information intensity, the second average information intensity, and the number of the first pixels, the stretching parameters are calculated, including:

[0114] Calculate the root mean square of the first average information intensity and the second average information intensity; according to the first average information intensity and the root mean square, calculate the stretching coefficient; according to the number of the first pixels and the root mean square, calculate the stretching strength.

[0115] In this embodiment, the stretching parameters include the stretching coefficient and the stretching strength. The stretching coefficient is calculated from the first average information intensity S mean , the second average information intensity T mean The calculation method is that the first average information intensity is divided by the root mean square of the first average information intensity and the second average information intensity:

[0116]

[0117]

[0118] Among them, β is the stretching coefficient, and its function is to control the proportion of enhancing the contrast according to the proportion of the first information in the low-frequency information. μ is the stretching strength, and its function is to control the intensity of enhancing the contrast according to the texture quantity of the low-frequency fusion information. The stretching strength μ is calculated from the first average information intensity S mean , the second average information intensity T mean , the first pixel statistical number C, and the stretching upper limit value thr2. The calculation method is that the first pixel statistical number C is divided by the first average information intensity S mean and the second average information intensity Tmean The root mean square is restricted to between 1 and the stretching upper limit value thr2 for its result range. Among them, takes the maximum value among them, takes the minimum value among them. It should be noted that for extreme scenarios, such as for images with rich textures, a larger stretching effect is desired, while for images with less texture, excessive stretching is not desired. To prevent over-stretching and insufficient stretching, the stretching upper limit value thr2 is set in this embodiment, which can ensure the basic stretching effect.

[0119] The stretching parameter of this embodiment is the root mean square of the first pixel statistical number C divided by the first average information intensity S mean and the second average information intensity T mean The root mean square. The first average information intensity and the second average information intensity are introduced herein, and the information statistically obtained from the first histogram is utilized, such that the calculated stretching parameter can be adaptively adjusted according to the texture distribution, so that the enhancement ratio and amplitude of the low-frequency information can be automatically adjusted according to the image content. Specifically, for images in different scenarios, appropriate stretching intensities can be provided. For example, for the two endoscopic detection scenarios of the nasal cavity and the abdominal cavity, by adopting the stretching parameter calculation method of this embodiment, the texture distribution of the image can be considered, so that the enhancement ratio and amplitude of the low-frequency information can be automatically adjusted according to the image content.

[0120] In one embodiment, according to the stretching parameter, a stretching value is calculated, including:

[0121] Determine a weight coefficient according to the stretching coefficient; perform weighted summation on the stretching intensity and the values of each brightness level in the first histogram according to the weight coefficient to obtain the stretching value.

[0122] The stretching value can be obtained by performing weighted summation on the stretching intensity and the first histogram according to the stretching coefficient:

[0123]

[0124] Among them, he(n) is the stretching value corresponding to the brightness level n, and hist(n) is the value corresponding to the brightness level n of the first histogram.

[0125] In one embodiment, obtaining the low-frequency information and high-frequency information of the target image includes:

[0126] Obtain the brightness information of the target image; on the basis of maintaining the original resolution of the target image, perform frequency domain decomposition on the brightness information to respectively obtain visible low-frequency information and visible high-frequency information, as well as non-visible low-frequency information and non-visible high-frequency information.

[0127] Figure 5 is a schematic diagram of Laplacian pyramid decomposition for related technologies,Figure 6 This is a schematic diagram of frequency domain decomposition for this embodiment. The frequency domain decomposition in this embodiment is different from Laplacian pyramid decomposition. During each decomposition process, no downsampling is performed, but the low-frequency information and high-frequency information of the original resolution size are maintained, and the generation of halos in the reconstructed image caused by enhanced processing of low-frequency information is suppressed. As Figure 6 shown, after one-time frequency domain decomposition, low-frequency information and high-frequency information with the same resolution size as the input image are obtained. After three times of frequency domain decomposition, 1 piece of low-frequency information and 3 pieces of high-frequency information are obtained.

[0128] In one embodiment, low-frequency information fusion and high-frequency information fusion can be implemented by the following methods:

[0129] (1) For low-frequency information fusion, the luminance information in the visible light image and the non-visible light image can be used as a guide to generate a luminance weight mapping table, and the low-frequency information in the visible light image and the non-visible light image is weighted to obtain low-frequency fusion information. For example, if the luminance weight mapping table is w, then the low-frequency fusion information lum Low is:

[0130]

[0131] where vis Low is the visible low-frequency information, and sp Low is the non-visible low-frequency information.

[0132] (2) For high-frequency information fusion, based on the corresponding visible high-frequency information and non-visible high-frequency information of each layer, a high-frequency weight table for each layer is obtained for high-frequency information fusion. For example, if the high-frequency weight table is , then the high-frequency fusion information is:

[0133]

[0134] where is the visible high-frequency information, is the non-visible high-frequency information, and n represents the information of the nth layer.

[0135] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are sequentially shown according to the indications of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same moment, but can be executed at different moments. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0136] In one embodiment, Figure 7 a flowchart of another multi-spectral image enhancement method is provided, as Figure 7 shown, and this process includes the following steps:

[0137] Step 1, convert the visible light image and the non-visible light image into a specified color space to obtain brightness information and color information respectively.

[0138] Step 2, perform multi-layer frequency domain decomposition on the visible brightness information and the non-visible brightness information to obtain visible low-frequency information, non-visible low-frequency information, visible high-frequency information, and non-visible high-frequency information.

[0139] Step 3, fuse the visible low-frequency information and the non-visible low-frequency information through the fusion rule of brightness mapping to obtain low-frequency fusion information; fuse the visible high-frequency information and the non-visible high-frequency information through different image frequency domain weight tables to obtain high-frequency fusion information.

[0140] Step 4, perform brightness and contrast enhancement processing on the low-frequency fusion information to obtain low-frequency enhanced information. Specifically, it includes histogram statistics, calculation of stretching parameters, regional stretching, and calculation of brightness mapping information.

[0141] Step 5, perform brightness reconstruction based on the low-frequency enhanced information and the high-frequency fusion information to obtain brightness reconstruction information.

[0142] Step 6, perform color adjustment processing such as color denoising and color mapping on the color information to obtain color enhanced information.

[0143] Step 7, combine the brightness reconstruction information and the color enhanced information to obtain a multi-spectral fusion image.

[0144] In this embodiment, through multi-layer frequency domain decomposition, the visible light image and the non-visible light image are decomposed into full-resolution low-frequency information and high-frequency information. The first histogram statistics are introduced into the low-frequency information. According to the judgment of the first information region and the second information region, the stretching parameters (stretching coefficient and stretching intensity) are calculated. The first histogram is stretched in regions and applied to the low-frequency information. This fusion enhancement process avoids being affected by noise, overexposed, and underexposed regions, and automatically adjusts the enhancement amplitude of the fused image. Specifically, during multi-layer frequency domain decomposition, the frequency domain information of full resolution is used for each decomposition to suppress the generation of halos in the reconstructed image during the low-frequency information enhancement process. When introducing the first histogram, the first information region and the second information region are distinguished according to the mean value of gradient information in different directions, avoiding the interference of noise in the low-frequency fusion information on the enhancement information statistics. When calculating the stretching coefficient and intensity, the first average information intensity and the second average information intensity are introduced, and the information statistically by the first histogram is used to enable the enhancement ratio and amplitude to be automatically adjusted according to the image content. During region-by-region stretching, according to the preset low-brightness and high-brightness thresholds, the first histogram is stretched in regions and intensities to ensure that the fusion enhancement is not affected by overexposed and underexposed regions.

[0145] In one embodiment, a multi-spectral image enhancement device is provided. This device is used to implement the above-mentioned embodiments and preferred implementation manners, and those that have been described will not be repeated here. The following terms such as "module", "unit", "sub-unit", etc. can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0146] Figure 8 is the structural block diagram of the multi-spectral image enhancement device of this embodiment, as Figure 8 shown, the device includes:

[0147] An acquisition module, configured to acquire the low-frequency information and high-frequency information of a target image; wherein, the target image includes a visible light image and a non-visible light image;

[0148] A statistics module, configured to statistically obtain the brightness level information of the first information region and the brightness level information of the second information region based on the low-frequency information, and respectively obtain a first histogram and a second histogram; wherein, the average gradient value of the first information region is greater than the average gradient value of the second information region;

[0149] A stretching module, configured to perform stretching processing on the first histogram according to the first histogram and the second histogram, and calculate brightness mapping information according to the stretched first histogram;

[0150] An enhancement module, configured to perform enhancement processing on the low-frequency information according to the brightness mapping information to obtain low-frequency enhanced information;

[0151] A reconstruction module, configured to reconstruct the luminance information of a target image according to low-frequency enhancement information and high-frequency information, so as to obtain luminance reconstruction information;

[0152] A merging module, configured to merge the luminance reconstruction information and the color information of a visible light image to obtain a multispectral fusion image.

[0153] It should be noted that the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation manners, and will not be elaborated herein. The above-mentioned respective modules may be functional modules or program modules, and may be implemented by software or by hardware. For the modules implemented by hardware, the above-mentioned respective modules may be located in the same processor; or the above-mentioned respective modules may also be located in different processors in any combined form.

[0154] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as Figure 9 shown. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Wherein, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a multispectral image enhancement method. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or may be a button, a trackball, or a touchpad provided on the housing of the computer device, or may also be an external keyboard, a touchpad, or a mouse, etc.

[0155] Those skilled in the art can understand that Figure 9 the structure shown in

[0156] In addition, in combination with the multi-spectral image enhancement method provided in the above embodiments, a storage medium can also be provided in this embodiment to implement it. A computer program is stored on the storage medium; when the computer program is executed by a processor, any one of the multi-spectral image enhancement methods in the above embodiments is implemented.

[0157] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties.

[0158] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the various embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the various embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the various embodiments provided in this application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0159] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0160] The above-described embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the patent of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several variations and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A multispectral image enhancement method, characterized in that: include: Acquire low-frequency information and high-frequency information of a target image; wherein the target image includes a visible light image and a non-visible light image; Based on the low-frequency information, the brightness level information of the first information area and the brightness level information of the second information area are counted to obtain a first histogram and a second histogram respectively; wherein the average gradient value of the first information area is greater than the average gradient value of the second information area, the first information area includes a first pixel, and the second information area includes a second pixel; According to the first histogram and the second histogram, stretching the first histogram, and calculating brightness mapping information according to the stretched first histogram; Performing enhancement processing on the low-frequency information according to the brightness mapping information to obtain low-frequency enhancement information; Reconstructing the brightness information of the target image according to the low-frequency enhancement information and the high-frequency information to obtain brightness reconstruction information; Combining the brightness reconstruction information with the color information of the visible light image to obtain a multi-spectral fusion image; The stretching process is performed on the first histogram according to the first histogram and the second histogram, including: Calculating a first average information intensity of the first information area according to the gradient value of each of the first pixels and the total number of the first pixels; calculating a second average information intensity of the second information area according to the gradient value of each of the second pixels and the total number of the second pixels; calculating a root mean square of the first average information intensity and the second average information intensity; calculating a stretching coefficient according to the first average information intensity and the root mean square; calculating a stretching intensity according to the number of the first pixels and the root mean square; Determine a weight coefficient according to the stretch coefficient; perform weighted summation of the stretch intensity and the values ​​of each brightness level in the first histogram according to the weight coefficient to obtain a stretch value; The stretched value is multiplied by the brightness ratio to obtain the stretched first histogram.

2. The multispectral image enhancement method according to claim 1, characterized in that: According to the first histogram and the second histogram, stretching the first histogram includes: Dividing the first histogram into a plurality of different brightness regions, and determining brightness ratios corresponding to the different brightness regions; In each of the different brightness regions, the first histogram is updated according to the stretching value and the corresponding brightness ratio to obtain the stretched first histogram.

3. The multispectral image enhancement method according to claim 1 or claim 2, characterized in that: The method comprises: calculating the brightness level information of the first information region and the brightness level information of the second information region based on the low-frequency information to obtain a first histogram and a second histogram respectively. The method comprises: In the low-frequency information, counting the gradient value of each pixel, and comparing the gradient value with a preset threshold; The first pixels whose gradient values ​​are greater than the preset threshold are divided into the first information region, and the second pixels whose gradient values ​​are not greater than the preset threshold are divided into the second information region; The number of each of the first pixels at each brightness level is counted in the first information region to obtain the first histogram, and the number of each of the second pixels at each brightness level is counted in the second information region to obtain the second histogram.

4. The multispectral image enhancement method according to claim 1 or claim 2, characterized in that: Calculating brightness mapping information according to the first histogram after the stretching process includes: Determining a target histogram in the first histogram; wherein the target histogram includes a plurality of the first histograms arranged in the first n brightness levels in order of brightness levels from high to low; wherein n is a positive integer; The sum of the target histogram is divided by the sum of all the first histograms, and the obtained quotient is multiplied by the maximum brightness level value to obtain the brightness enhancement level value corresponding to each brightness input level value.

5. The multispectral image enhancement method according to claim 1, characterized in that: Reconstructing the brightness information of the target image according to the low-frequency enhancement information and the high-frequency information to obtain the brightness reconstruction information includes: Add the low-frequency enhancement information of the last layer to the high-frequency information of the same layer to obtain the low-frequency enhancement information of the previous layer; The low-frequency enhancement information of the previous layer is added to the high-frequency information of the same layer to obtain the low-frequency enhancement information of the previous layer, and so on, until the low-frequency enhancement information of the first layer is added to the high-frequency information of the same layer to obtain the brightness reconstruction information.

6. The multispectral image enhancement method according to claim 1, characterized in that: The low-frequency information includes visible low-frequency information and invisible low-frequency information; or, the low-frequency information includes low-frequency fusion information obtained by fusion of visible low-frequency information and invisible low-frequency information; The high-frequency information includes visible high-frequency information and invisible high-frequency information; or, the high-frequency information includes high-frequency fusion information obtained by fusing visible high-frequency information with invisible high-frequency information.

7. The multispectral image enhancement method according to any one of claims 1, 2, 5 and 6, characterized in that: Obtain low-frequency and high-frequency information of the target image, including: Acquiring brightness information of the target image; On the basis of maintaining the original resolution of the target image, the brightness information is decomposed in the frequency domain to obtain visible low-frequency information and visible high-frequency information, as well as invisible low-frequency information and invisible high-frequency information.

8. A multispectral image enhancement device, characterized in that: The device comprises: An acquisition module, used to acquire low-frequency information and high-frequency information of a target image; wherein the target image includes a visible light image and a non-visible light image; A statistical module, configured to collect the brightness level information of the first information region and the brightness level information of the second information region based on the low-frequency information, and obtain a first histogram and a second histogram respectively; wherein the average gradient value of the first information region is greater than the average gradient value of the second information region, the first information region includes a first pixel, and the second information region includes a second pixel; a stretching module, configured to perform stretching processing on the first histogram according to the first histogram and the second histogram, and calculate brightness mapping information according to the first histogram after the stretching processing; An enhancement module, configured to perform enhancement processing on the low-frequency information according to the brightness mapping information to obtain low-frequency enhancement information; A reconstruction module, used to reconstruct the brightness information of the target image according to the low-frequency enhancement information and the high-frequency information to obtain brightness reconstruction information; A merging module, used for merging the brightness reconstruction information and the color information of the visible light image to obtain a multi-spectral fused image; The stretching process is performed on the first histogram according to the first histogram and the second histogram, including: Calculating a first average information intensity of the first information area according to the gradient value of each of the first pixels and the total number of the first pixels; calculating a second average information intensity of the second information area according to the gradient value of each of the second pixels and the total number of the second pixels; calculating a root mean square of the first average information intensity and the second average information intensity; calculating a stretching coefficient according to the first average information intensity and the root mean square; calculating a stretching intensity according to the number of the first pixels and the root mean square; Determine a weight coefficient according to the stretch coefficient; perform weighted summation of the stretch intensity and the values ​​of each brightness level in the first histogram according to the weight coefficient to obtain a stretch value; The stretched value is multiplied by the brightness ratio to obtain the stretched first histogram.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images