Multispectral image enhancement method and device, computer equipment and storage medium

Abstract

The invention relates to a multispectral image enhancement method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring low-frequency information and high-frequency information of a target image; performing statistics on brightness level information of the first information area and brightness level information of the second information area based on the low-frequency information to obtain a first histogram and a second histogram respectively; performing stretching processing on the first histogram according to the first histogram and the second histogram, and calculating brightness mapping information according to the stretched first histogram; performing enhancement processing on the low-frequency information according to the brightness mapping information to obtain low-frequency enhancement information; according to the low-frequency enhancement information and the high-frequency information, reconstructing brightness information of the target image to obtain brightness reconstruction information; and combining the brightness reconstruction information and the color information of the visible light image to obtain a multispectral fusion image. According to the invention, the interference of the noise in the low-frequency information on the low-frequency enhanced information is eliminated, and the quality of the multispectral fusion image is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a multispectral image enhancement method, device, computer equipment and storage medium. Background Art

[0002] During medical surgery, it is often necessary to use shooting and display systems to observe and process the target area. The quality of medical images and videos has become one of the concerns in the medical system. Visible light in multispectral endoscopes can observe normal views, while fluorescence can better detect diseased tissues that are invisible to the naked eye. Multispectral fusion combines visible light and non-visible light images in one image, which can help surgeons to perform preliminary positioning and inspection during surgery.

[0003] Multi-spectral fusion "superimposes" visible light and invisible light together. Since it is a fusion of different spectral bands, invisible light has a narrow spectrum, which will result in low contrast in the image. After fusion with visible light, it will also affect the contrast of the fused image. In the process of contrast enhancement of the fused image, it is easy to be affected by interference factors, such as noise, resulting in poor effect after contrast enhancement.

[0004] At present, no effective solution has been proposed for the problem of low quality of multispectral fusion images in related technologies. Summary of the invention

[0005] Based on this, it is necessary to provide a multispectral image enhancement method, device, computer equipment and storage medium that can improve the quality of multispectral fusion images in response to the above technical problems.

[0006] In a first aspect, the present application provides a multispectral image enhancement method, comprising: Acquire low-frequency information and high-frequency information of a target image; wherein the target image includes a visible light image and a non-visible light image; Based on the low-frequency information, the brightness level information of the first information area and the brightness level information of the second information area are counted to obtain a first histogram and a second histogram respectively; wherein the average gradient value of the first information area is greater than the average gradient value of the second information area; According to the first histogram and the second histogram, stretching the first histogram, and calculating brightness mapping information according to the stretched first histogram; Performing enhancement processing on the low-frequency information according to the brightness mapping information to obtain low-frequency enhancement information; Reconstructing the brightness information of the target image according to the low-frequency enhancement information and the high-frequency information to obtain brightness reconstruction information; The brightness reconstruction information and the color information of the visible light image are combined to obtain a multi-spectral fusion image.

[0007] In one embodiment, performing stretching processing on the first histogram according to the first histogram and the second histogram includes: Calculate a stretch value according to the first histogram and the second histogram; Dividing the first histogram into a plurality of different brightness regions, and determining brightness ratios corresponding to the different brightness regions; In each of the different brightness regions, the first histogram is updated according to the stretching value and the corresponding brightness ratio to obtain the stretched first histogram.

[0008] In one embodiment, the brightness level information of the first information area and the brightness level information of the second information area are statistically calculated based on the low-frequency information to obtain a first histogram and a second histogram respectively, including: In the low-frequency information, counting the gradient value of each pixel, and comparing the gradient value with a preset threshold; Classify the first pixel whose gradient value is greater than the preset threshold into the first information area, and classify the second pixel whose gradient value is not greater than the preset threshold into the second information area; The number of each of the first pixels at each brightness level is counted in the first information region to obtain the first histogram, and the number of each of the second pixels at each brightness level is counted in the second information region to obtain the second histogram.

[0009] In one embodiment, calculating the stretch value according to the first histogram and the second histogram includes: Calculating a first average information intensity of the first information area according to the gradient value of each of the first pixels and the total number of the first pixels; Calculating a second average information intensity of the second information area according to the gradient value of each of the second pixels and the total number of the second pixels; Calculating a stretching parameter according to the first average information intensity, the second average information intensity, and the number of the first pixels; The stretching value is obtained by calculation according to the stretching parameter.

[0010] In one embodiment, calculating the stretching parameter according to the first average information intensity, the second average information intensity and the number of the first pixels includes: Calculating a root mean square of the first average information strength and the second average information strength; Calculate a stretching coefficient according to the first average information intensity and the root mean square; The tensile strength is calculated according to the number of the first pixels and the root mean square.

[0011] In one embodiment, calculating the stretching value according to the stretching parameter includes: Determining a weight coefficient according to the stretch coefficient; The stretching strength and the values ​​of each brightness level in the first histogram are weighted and summed according to the weight coefficient to obtain the stretching value.

[0012] In one embodiment, calculating brightness mapping information according to the first histogram after stretching includes: Determining a target histogram in the first histogram; wherein the target histogram includes a plurality of the first histograms arranged in the first n brightness levels in order of brightness levels from high to low; wherein n is a positive integer; The sum of the target histogram is divided by the sum of all the first histograms, and the obtained quotient is multiplied by the maximum brightness level value to obtain the brightness enhancement level value corresponding to each brightness input level value.

[0013] In one embodiment, reconstructing the brightness information of the target image according to the low-frequency enhancement information and the high-frequency information to obtain the brightness reconstruction information includes: Add the low-frequency enhancement information of the last layer to the high-frequency information of the same layer to obtain the low-frequency enhancement information of the previous layer; The low-frequency enhancement information of the previous layer is added to the high-frequency information of the same layer to obtain the low-frequency enhancement information of the previous layer, and so on, until the low-frequency enhancement information of the first layer is added to the high-frequency information of the same layer to obtain the brightness reconstruction information.

[0014] In one embodiment, the low-frequency information includes visible low-frequency information and invisible low-frequency information; or, the low-frequency information includes low-frequency fusion information obtained by fusion of visible low-frequency information and invisible low-frequency information; The high-frequency information includes visible high-frequency information and invisible high-frequency information; or, the high-frequency information includes high-frequency fusion information obtained by fusing visible high-frequency information with invisible high-frequency information.

[0015] In one embodiment, obtaining low-frequency information and high-frequency information of a target image includes: Acquiring brightness information of the target image; On the basis of maintaining the original resolution of the target image, the brightness information is decomposed in the frequency domain to obtain visible low-frequency information and visible high-frequency information, as well as invisible low-frequency information and invisible high-frequency information.

[0016] In a second aspect, the present application provides a multispectral image enhancement device, the device comprising: An acquisition module, used to acquire low-frequency information and high-frequency information of a target image; wherein the target image includes a visible light image and a non-visible light image; A statistical module, configured to collect the brightness level information of the first information region and the brightness level information of the second information region based on the low-frequency information, and obtain a first histogram and a second histogram respectively; wherein the average gradient value of the first information region is greater than the average gradient value of the second information region; a stretching module, configured to perform stretching processing on the first histogram according to the first histogram and the second histogram, and calculate brightness mapping information according to the first histogram after the stretching processing; An enhancement module, configured to perform enhancement processing on the low-frequency information according to the brightness mapping information to obtain low-frequency enhancement information; A reconstruction module, used to reconstruct the brightness information of the target image according to the low-frequency enhancement information and the high-frequency information to obtain brightness reconstruction information; The merging module is used to merge the brightness reconstruction information and the color information of the visible light image to obtain a multi-spectral fused image.

[0017] In a third aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.

[0018] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0019] The above-mentioned multispectral image enhancement method, device, computer equipment and storage medium distinguish between the first information area and the second information area when calculating the histogram, respectively count the first histogram and the second histogram based on the two areas, and stretch the first histogram based on the first histogram and the second histogram. In this way, when stretching, the texture stretching can be mainly targeted to avoid noise, and the noise in the low-frequency information can be avoided from being stretched incorrectly, thereby eliminating the interference of the noise in the low-frequency information on the low-frequency enhancement information, and improving the quality of the multispectral fusion image. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The hardware structure block diagram of a terminal of a multi-spectral image enhancement method in one embodiment; Figure 2is a flow chart of a multispectral image enhancement method in one embodiment; Figure 3 is a schematic diagram of brightness information reconstruction in one embodiment; Figure 4 is a schematic diagram of a target image in one embodiment; Figure 5 A schematic diagram of Laplace pyramid decomposition for related technologies; Figure 6 is a schematic diagram of frequency domain decomposition in one embodiment; Figure 7 is a flow chart of a multispectral image enhancement method in another embodiment; Figure 8 is a structural block diagram of a multi-spectral image enhancement device in one embodiment; Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0022] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0023] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG. 1 is a hardware structure diagram of a terminal of a multispectral image enhancement method according to an embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 101 and memory 102 for storing data, wherein processor 101 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 103 and an input and output device 104 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0024] The memory 102 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the multispectral image enhancement method in this embodiment. The processor 101 executes various functional applications and data processing by running the computer program stored in the memory 102, that is, to implement the above method. The memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 102 may further include a memory remotely arranged relative to the processor 101, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0025] The transmission device 103 is used to receive or send data via a network. The above network includes a wireless network provided by a communication provider of the terminal. In one example, the transmission device 103 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 103 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0026] In traditional multispectral fusion methods, due to the introduction of non-visible spectral brightness, the fused image has image quality problems such as overall blurring and low contrast. The related art provides a multispectral image enhancement method, the design point of which is to use the histogram stretching method to obtain a high-exposure image for the input image, then negate the high-exposure image to obtain a low-exposure image, and then use the Laplacian pyramid to fuse the high-exposure and low-exposure images to obtain the final enhanced image. However, this method directly stretches the histogram of the original image, which is easily affected by noise, resulting in poor quality of the high-exposure image, and ultimately poor enhancement of the fused image.

[0027] Based on the analysis of the above situation, in one embodiment, if Figure 2 As shown, a multispectral image enhancement method is provided, which is applied to Figure 1 The terminal in is used as an example to illustrate, including the following steps: Step S101, obtaining low-frequency information and high-frequency information of a target image; wherein the target image includes a visible light image and a non-visible light image.

[0028] Taking endoscope inspection as an example, the target image includes visible light images and non-visible light images (fluorescence images) collected by the endoscope. Among them, the visible light image is used to provide an intuitive view, and the non-visible light image is used to enhance the contrast, which helps to find details that may be missed.

[0029] Step S101 can be implemented by frequency domain decomposition, and the method includes: performing multiple frequency domain filtering processes on the target image to decompose the brightness information into multiple layers of different frequency domain information. Dividing the image into component information of different frequency domains is helpful for targeted enhancement and fusion processing in each frequency domain information. The filters used in the frequency domain decomposition method include but are not limited to low-pass filters such as Gaussian and Butterworth, or edge-preserving filters such as bilateral filtering, guided filtering, weighted least squares filter, non-uniform local filter, double exponential edge smoothing filter, selective blur and surface filtering. Among them, the edge-preserving filter refers to a type of filter that retains the edge information of the image during the filtering process. If f(y) represents a filtering operation on the input brightness image y, the low-frequency information obtained by the first decomposition is f(y), and the high-frequency information is yf(y). The low-frequency information obtained by the second decomposition is f(f(y)), and the high-frequency information is f(y)-f(f(y)). The low-frequency information obtained by the third decomposition is f(f(f(y))), and the high-frequency information is f(f(y))-f(f(f(y))). Finally, the low-frequency information f(f(f(y))) and the three high-frequency information are yf(y), f(y)-f(f(y)), and f(f(y))-f(f(f(y))).

[0030] In some embodiments, before executing step S101, the target image may also be subjected to color space conversion processing. The color space conversion processing includes: converting the visible light image and the non-visible light image into a specified color space, which needs to be distinguished by brightness and color information, for example, color spaces such as YUV, YCbCr, HSV, and Lab. Among them, Y in YUV is brightness information and UV is color information, Y in YCbCr is brightness information and CbCr is color information, V in HSV is brightness information and HS is color information, and L in Lab is brightness information and ab is color information. This embodiment is applicable to any of the above color spaces.

[0031] Step S102, based on the low-frequency information, statistics are collected on the brightness level information of the first information area and the brightness level information of the second information area to obtain a first histogram and a second histogram respectively; wherein the average gradient value of the first information area is greater than the average gradient value of the second information area.

[0032] In the histogram, the horizontal axis represents the brightness level (for example, the brightness level can be in the range of 0-X, where X can be the maximum pixel value of the target image or other preset values), and the vertical axis represents the number of pixels corresponding to the brightness level. The larger the average gradient value, the richer the texture information. The average gradient value of the first information area is greater than the average gradient value of the second information area, which means that the texture information of the first information area is richer than that of the second information area.

[0033] The low-frequency information includes visible low-frequency information (i.e., low-frequency information of visible light images) and invisible low-frequency information (i.e., low-frequency information of non-visible light images); or, the low-frequency information includes low-frequency fusion information obtained by fusing visible low-frequency information with invisible low-frequency information. The high-frequency information includes visible high-frequency information (i.e., high-frequency information of visible light images) and invisible high-frequency information (i.e., high-frequency information of non-visible light images); or, the high-frequency information includes high-frequency fusion information obtained by fusing visible high-frequency information with invisible high-frequency information.

[0034] When the histogram is based on low-frequency information, the statistics are performed based on the low-frequency information of the last layer after frequency domain decomposition. Specifically, the last layer of low-frequency information of the visible light image and the non-visible light image can be respectively counted, and then the statistical results of the two can be merged. Alternatively, the last layer of low-frequency information of the visible light image and the non-visible light image can be fused first, and then the fused last layer of low-frequency information can be counted, which is not limited in this application.

[0035] Step S103: According to the first histogram and the second histogram, stretching processing is performed on the first histogram, and brightness mapping information is calculated according to the stretched first histogram.

[0036] When stretching the first histogram according to the first histogram and the second histogram, the stretching value may be calculated according to the first histogram and the second histogram, and then the ordinate value of the first histogram is updated to the corresponding stretching value.

[0037] After the first histogram is stretched, the brightness mapping information is calculated from the stretched and updated first histogram. A target histogram is determined in the first histogram; wherein the target histogram includes a plurality of first histograms arranged in the first n brightness levels in descending order of brightness levels; wherein n is a positive integer; the sum of the target histogram is divided by the sum of all the first histograms, and the quotient is multiplied by the maximum brightness level value to obtain the brightness enhancement level value corresponding to each brightness input level value.

[0038] The sum of the target histogram refers to the sum of the first histograms of the first n brightness levels calculated by accumulating the values ​​of the target histogram. The sum of the target histogram is divided by the sum of all the first histograms, and then the quotient is multiplied by the maximum brightness level value to obtain the brightness enhancement level value corresponding to each brightness input level value, which is the brightness mapping information. The maximum brightness level value can be determined by the bit width of the target image. If the bit width is b, the maximum brightness level value is , the range of brightness level n is .in, That is, the maximum pixel value of the target image. For example, assuming that the bit width of the target image is 8 and its maximum brightness level value is 255, the range of the brightness level n is [0, 255].

[0039] Step S104: enhance the low-frequency information according to the brightness mapping information to obtain low-frequency enhanced information.

[0040] The brightness mapping information includes two types of data, namely, the input brightness level value (i.e., the original brightness level value in the target image) and the enhanced brightness level value. Applying the brightness mapping information to the input low-frequency information will obtain the enhanced low-frequency information.

[0041] Step S105 : reconstructing the brightness information of the target image according to the low-frequency enhancement information and the high-frequency information to obtain brightness reconstruction information.

[0042] Brightness information reconstruction is the inverse process of multi-layer frequency domain decomposition: the low-frequency enhancement information of the last layer is added to the high-frequency information of the same layer to obtain the low-frequency enhancement information of the previous layer; the low-frequency enhancement information of the previous layer is added to the high-frequency information of the same layer to obtain the low-frequency enhancement information of the previous layer, and so on, until the low-frequency enhancement information of the first layer is added to the high-frequency information of the same layer to obtain the brightness reconstruction information, that is, the enhanced brightness information.

[0043] Figure 3 A schematic diagram of brightness information reconstruction is provided, such as Figure 3 As shown in the figure, assuming that the target image is initially decomposed in the frequency domain three times, when reconstructing the brightness information, the low-frequency information of the last layer is merged upward layer by layer. First, the low-frequency enhancement information of the third layer is added to the high-frequency information of the third layer to obtain the low-frequency information of the second layer, and then added to the high-frequency information of the second layer to obtain the low-frequency information of the first layer, and then added to the high-frequency information of the first layer to obtain the final brightness reconstruction information.

[0044] Step S106: merge the brightness reconstruction information and the color information of the visible light image to obtain a multi-spectral fusion image.

[0045] The brightness reconstruction information and the color information of the visible light image may be directly combined, or the color information of the visible light image may be adjusted first and then the adjusted color information and the brightness reconstruction information may be combined. The color adjustment includes but is not limited to color denoising, color mapping, color enhancement and other processing.

[0046] In the above steps S101 to S106, if the original histogram is stretched directly, the texture and noise will be difficult to distinguish during stretching, and the noise will inevitably be stretched as texture. For this reason, this embodiment distinguishes the first information area and the second information area when calculating the histogram, and the first histogram and the second histogram are respectively counted based on the two areas, and the first histogram is stretched based on the first histogram and the second histogram. In this way, when stretching, the texture can be stretched mainly to avoid noise, and the noise in the low-frequency information can be avoided from being stretched incorrectly, thereby eliminating the interference of the noise in the low-frequency information on the low-frequency enhancement information, and improving the quality of the multi-spectral fusion image.

[0047] The related technology provides another endoscopic image enhancement method, which obtains a base layer image by performing a guided filter on the input image, subtracts it from the original image to obtain a detail layer image, performs block histogram stretching (CLAHE) on the base layer image and non-local mean filtering (NLM) on the detail layer image, performs segmented gain processing on the denoised detail layer, and finally uses the enhanced base layer plus the processed detail layer to obtain the final enhanced image in order to enhance the details in the fusion image. This scheme performs block contrast enhancement on the low-frequency layer after decomposition, which can avoid the problem of enhanced noise, but the block contrast is easily affected by the over-brightness and over-darkness information in the block area, which will cause the problem of excessive regional stretching.

[0048] Based on the analysis of the above situation, in one embodiment, in step S103, the first histogram is stretched according to the first histogram and the second histogram, which can also be achieved by the following method: A stretching value is calculated based on the first histogram and the second histogram; the first histogram is divided into a plurality of different brightness regions, and the brightness ratios corresponding to the different brightness regions are determined; within each different brightness region, the first histogram is updated based on the stretching value and the corresponding brightness ratio to obtain a stretched first histogram.

[0049] In this embodiment, a preset value can be set, and the first histogram is divided into two or more different brightness areas through the preset value, each different brightness area corresponds to a different brightness ratio, and in each different brightness area, the first histogram is stretched according to the stretching value and the corresponding brightness ratio, so as to achieve regional and intensity stretching of the first histogram, and ensure that the enhanced low-frequency information is not affected by overexposure and dark areas. It should also be noted that the histogram itself is arranged according to the brightness level (the horizontal axis is the brightness level, and the vertical axis is the number of pixels belonging to the brightness level). The human eye observation comfort zone is concentrated in the medium-bright area. If only the stretching of the medium-bright area is considered, the brightness information of other areas will be compressed, and the image quality will be reduced. For this reason, this implementation subdivides the first histogram into multiple areas before stretching it, which ensures the quality of the stretched image.

[0050] For example, the first histogram can be divided into a high-brightness area, a medium-brightness area, and a low-brightness area according to a preset value, and then the stretched first histogram is calculated by combining the stretching value and the corresponding brightness ratio. Among them, stretching the value of each brightness level means updating the first histogram to the value obtained by multiplying the stretching value by the preset brightness ratio of each area. Specifically, all brightness levels in the first histogram are traversed and stretched in the following three cases: (1) For low-brightness areas: If the brightness level is less than or equal to the low-brightness preset value, the first histogram is updated to the stretch value multiplied by the low-brightness ratio; (2) For the highlight area: If the brightness level is greater than or equal to the highlight preset value, the first histogram is updated to the stretch value multiplied by the highlight ratio; (3) For the medium-bright area: If the brightness level is greater than the low-brightness preset value and less than the high-brightness preset value, the first histogram is updated to the stretched value.

[0051] The low brightness ratio and the high brightness ratio can be preset by the user to be between (0,1) according to the usage environment, for example, the low brightness ratio is 0.3 and the high brightness ratio is 0.6. The role of this ratio is to control the enhancement range of the low brightness area and the high brightness area. The smaller the brightness ratio, the greater the enhancement range.

[0052] In one embodiment, in the above step S102, based on the low-frequency information, the brightness level information of the first information area and the brightness level information of the second information area are statistically obtained to obtain the first histogram and the second histogram respectively, which can be achieved by the following method: In the low-frequency information, the gradient value of each pixel is counted, and the gradient value is compared with a preset threshold; the first pixel whose gradient value is greater than the preset threshold is divided into a first information area, and the second pixel whose gradient value is not greater than the preset threshold is divided into a second information area; the number of each first pixel at each brightness level in the first information area is counted to obtain a first histogram, and the number of each second pixel at each brightness level in the second information area is counted to obtain a second histogram.

[0053] Furthermore, the stretching value is calculated based on the first histogram and the second histogram, which can be achieved by the following method: The first average information intensity of the first information area is calculated according to the gradient value of each first pixel and the total number of first pixels; the second average information intensity of the second information area is calculated according to the gradient value of each second pixel and the total number of second pixels; the stretching parameter is calculated according to the first average information intensity, the second average information intensity and the number of first pixels; the stretching value is calculated according to the stretching parameter.

[0054] In this embodiment, the gradient value of a pixel can be the gradient value obtained under the gradient information of a certain dimension of the pixel, or the gradient value obtained by averaging the gradient information of several dimensions of the pixel. The more dimensions of gradient information considered, the more accurate the pixel division will be. For example, in the target image, the gradient value obtained by averaging the gradient information of the three dimensions of horizontal, vertical, and diagonal for each pixel position is counted, and then the gradient value is compared with the preset threshold, which is divided into the following two cases: (1) If the gradient value is greater than a preset threshold, the number of pixel brightness levels in the first histogram is accumulated by 1, and the gradient value is accumulated as the first information intensity, and the total number of first pixel statistics is accumulated by 1.

[0055] (2) If the gradient value is less than or equal to the preset threshold, the gradient value is accumulated as the second information strength, and the total number of second pixel statistics is accumulated as 1.

[0056] Traverse the low-frequency information, obtain the first histogram according to step 1), and divide the first information intensity by the total number of accumulated first pixels to obtain the first average information intensity, obtain the second histogram according to step 2), and divide the second information intensity by the total number of accumulated second pixels to obtain the second average information intensity. The preset threshold value can be an empirical value, for example, set to 20.

[0057] The following is the calculation formula for the average gradient value:

[0058] Figure 4 is a schematic diagram of the target image, such as Figure 4As shown, p(m,n) is the pixel value of the low-frequency information in the mth column and nth row, Gh(m,n) is the horizontal gradient value of the mth column and nth row, Gv(m,n) is the vertical gradient value of the mth column and nth row, Gd(m,n) is the diagonal gradient value of the mth column and nth row, and G(m,n) is the average gradient value of the mth column and nth row.

[0059] The first histogram hist is initialized to 0, and its calculation method is that when G(m,n) is greater than thr1, hist(p(m,n)) is accumulated by 1. Here, thr1 is the preset threshold, which is set to 15 in general scenarios.

[0060] The first average information strength S mean is the mean of G(m,n) such that G(m,n) is greater than thr1.

[0061] The second average information strength T mean is the mean of G(m,n) such that G(m,n) is less than or equal to thr1.

[0062] In one embodiment, the stretching parameter is calculated according to the first average information intensity, the second average information intensity and the number of first pixels, including: Calculate the root mean square of the first average information intensity and the second average information intensity; calculate the stretching coefficient according to the first average information intensity and the root mean square; calculate the stretching strength according to the number of first pixels and the root mean square.

[0063] In this embodiment, the stretching parameters include a stretching coefficient and a stretching strength. The stretching coefficient is determined by the first average information strength S mean , the second average information strength T mean The calculation method is the first average information intensity divided by the root mean square of the first average information intensity and the second average information intensity:

[0064]

[0065] Where β is the stretching coefficient, which controls the ratio of contrast enhancement according to the ratio of the first information in the low-frequency information. μ is the stretching strength, which controls the strength of contrast enhancement according to the number of textures in the low-frequency fusion information. The stretching strength μ is determined by the first average information strength S mean , the second average information strength T mean , the first pixel statistics number C and the stretching upper limit value thr2 are calculated, and the calculation method is the first pixel statistics number C divided by the first average information intensity S mean and the second average information strength T mean The root mean square of , the result range is limited to between 1 and the stretching upper limit thr2. is to take the maximum value among them, The minimum value is taken. It should be noted that for extreme scenes, such as images with rich textures, a larger stretching effect is desired, while for images with less rich textures, excessive stretching is not desired. In order to prevent over-stretching and under-stretching, the present embodiment sets a stretching upper limit value thr2 to ensure a basic stretching effect.

[0066] The stretching parameter in this embodiment is the first pixel statistical number C divided by the first average information intensity S mean and the second average information strength T mean The root mean square of the image is obtained by introducing the first average information intensity and the second average information intensity, and using the statistical information of the first histogram, the calculated stretching parameter can be adaptively adjusted according to the texture distribution, so that the enhancement ratio and amplitude of the low-frequency information can be automatically adjusted according to the image content. Specifically, for images of different scenes, a suitable stretching strength can be provided. For example, for the two endoscopic detection scenes of the nasal cavity and the abdominal cavity, the stretching parameter calculation method of this embodiment can be used to consider the image texture distribution, so that the enhancement ratio and amplitude of the low-frequency information can be automatically adjusted according to the image content.

[0067] In one embodiment, the stretching value is calculated according to the stretching parameter, including: The weight coefficient is determined according to the stretch coefficient; the stretch strength and the values ​​of each brightness level in the first histogram are weighted and summed according to the weight coefficient to obtain the stretch value.

[0068] The stretch value can be obtained by weighting the tensile strength and the first histogram according to the stretch coefficient:

[0069] Among them, he(n) is the stretching value corresponding to brightness level n, and hist(n) is the value corresponding to the brightness level n of the first histogram.

[0070] In one embodiment, obtaining low-frequency information and high-frequency information of a target image includes: The brightness information of the target image is obtained; on the basis of maintaining the original resolution of the target image, the brightness information is decomposed in the frequency domain to obtain visible low-frequency information and visible high-frequency information, as well as invisible low-frequency information and invisible high-frequency information.

[0071] Figure 5 It is a schematic diagram of Laplace pyramid decomposition of related technologies. Figure 6Schematic diagram of frequency domain decomposition of this embodiment. The frequency domain decomposition of this embodiment is different from the Laplace pyramid decomposition. In each decomposition process, no downsampling is performed, but the low-frequency information and high-frequency information of the original resolution are maintained, and the generation of halo in the reconstructed image due to the low-frequency information enhancement processing is suppressed. Figure 6 As shown in FIG. 1 , a frequency domain decomposition obtains low-frequency information and high-frequency information of the same size as the input image resolution. After three frequency domain decompositions, one low-frequency information and three high-frequency information are obtained.

[0072] In one embodiment, the low-frequency information fusion and the high-frequency information fusion can be implemented by the following method: (1) Low-frequency information fusion can use the brightness information in the visible light image and the non-visible light image as a guide to generate a brightness weight mapping table, and weight the low-frequency information in the visible light image and the non-visible light image to obtain low-frequency fusion information. For example, if the brightness weight mapping table is w, then the low-frequency fusion information lum Low for:

[0073] Among them, vis Low For visible low-frequency information, sp Low It is invisible low-frequency information.

[0074] (2) High-frequency information fusion: According to the visible high-frequency information and invisible high-frequency information corresponding to each layer, the high-frequency weight table of each layer is obtained to perform high-frequency information fusion. For example, the high-frequency weight table is , then high frequency fusion information for:

[0075] in, For visible high-frequency information, is the invisible high-frequency information, and n represents the information of the nth layer.

[0076] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0077] In one embodiment, Figure 7 A flowchart of another multispectral image enhancement method is provided, such as Figure 7 As shown, the process includes the following steps: Step 1: Convert the visible light image and the non-visible light image into a specified color space to obtain brightness information and color information respectively.

[0078] Step 2: Perform multi-layer frequency domain decomposition on the visible brightness information and the invisible brightness information to obtain visible low-frequency information, invisible low-frequency information, visible high-frequency information, and invisible high-frequency information.

[0079] Step 3: Through the fusion rule of brightness mapping, the visible low-frequency information and the invisible low-frequency information are fused to obtain low-frequency fusion information; through different image frequency domain weight tables, the visible high-frequency information and the invisible high-frequency information are fused to obtain high-frequency fusion information.

[0080] Step 4: Perform brightness and contrast enhancement processing on the low-frequency fusion information to obtain low-frequency enhanced information, which specifically includes histogram statistics, calculation of stretching parameters, regional stretching, and calculation of brightness mapping information.

[0081] Step 5: Reconstruct brightness based on the low-frequency enhancement information and the high-frequency fusion information to obtain brightness reconstruction information.

[0082] Step 6: Perform color adjustment processing such as color denoising and color mapping on the color information to obtain color enhancement information.

[0083] Step 7: Combine the brightness reconstruction information with the color enhancement information to obtain a multispectral fusion image.

[0084] This embodiment decomposes the visible light image and the non-visible light image into low-frequency information and high-frequency information of full resolution through multi-layer frequency domain decomposition, introduces the first histogram statistics into the low-frequency information, calculates the stretching parameters (stretching coefficient and stretching strength) according to the judgment of the first information area and the second information area, stretches the first histogram by area, and acts on the low-frequency information. This fusion enhancement process avoids the influence of noise, overexposure, and dark areas, and automatically adjusts the enhancement amplitude of the fused image. Specifically, when multi-layer frequency domain decomposition is performed, each decomposition uses the frequency domain information of full resolution to suppress the generation of halo in the reconstructed image by the low-frequency information enhancement process. When introducing the first histogram, the first information area and the second information area are distinguished according to the mean of the gradient information in different directions, avoiding the interference of noise in the low-frequency fusion information on the enhanced information statistics. When calculating the stretching coefficient and intensity, the first average information intensity and the second average information intensity are introduced, and the information of the first histogram statistics is used so that the enhancement ratio and amplitude can be automatically adjusted according to the image content. When stretching by region, the first histogram is stretched by region and intensity according to the preset low-brightness and high-brightness thresholds to ensure that the fusion enhancement is not affected by overexposed or dark areas.

[0085] In one embodiment, a multi-spectral image enhancement device is provided, which is used to implement the above-mentioned embodiments and preferred implementations, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0086] Figure 8 is a structural block diagram of the multi-spectral image enhancement device of this embodiment. Figure 8 As shown, the device comprises: An acquisition module, used to acquire low-frequency information and high-frequency information of a target image; wherein the target image includes a visible light image and a non-visible light image; A statistical module, for statistically analyzing the brightness level information of the first information area and the brightness level information of the second information area based on the low-frequency information, to obtain a first histogram and a second histogram respectively; wherein the average gradient value of the first information area is greater than the average gradient value of the second information area; A stretching module, configured to perform stretching processing on the first histogram according to the first histogram and the second histogram, and calculate brightness mapping information according to the stretched first histogram; An enhancement module, used for enhancing the low-frequency information according to the brightness mapping information to obtain low-frequency enhancement information; A reconstruction module is used to reconstruct the brightness information of the target image according to the low-frequency enhancement information and the high-frequency information to obtain brightness reconstruction information; The merging module is used to merge the brightness reconstruction information and the color information of the visible light image to obtain a multi-spectral fused image.

[0087] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment. The above modules can be functional modules or program modules, which can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0088] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a multispectral image enhancement method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0089] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0090] In addition, in combination with the multispectral image enhancement method provided in the above embodiments, a storage medium may be provided in this embodiment to implement the multispectral image enhancement method. The storage medium stores a computer program; when the computer program is executed by a processor, any multispectral image enhancement method in the above embodiments is implemented.

[0091] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0092] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0093] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multispectral image enhancement method, characterized in that: include: Acquire low-frequency information and high-frequency information of a target image; wherein the target image includes a visible light image and a non-visible light image; Based on the low-frequency information, the brightness level information of the first information area and the brightness level information of the second information area are counted to obtain a first histogram and a second histogram respectively; wherein the average gradient value of the first information area is greater than the average gradient value of the second information area; According to the first histogram and the second histogram, stretching the first histogram, and calculating brightness mapping information according to the stretched first histogram; Performing enhancement processing on the low-frequency information according to the brightness mapping information to obtain low-frequency enhancement information; Reconstructing the brightness information of the target image according to the low-frequency enhancement information and the high-frequency information to obtain brightness reconstruction information; The brightness reconstruction information and the color information of the visible light image are combined to obtain a multi-spectral fusion image.

2. The multispectral image enhancement method according to claim 1, characterized in that: According to the first histogram and the second histogram, stretching the first histogram includes: Calculate a stretch value according to the first histogram and the second histogram; Dividing the first histogram into a plurality of different brightness regions, and determining brightness ratios corresponding to the different brightness regions; In each of the different brightness regions, the first histogram is updated according to the stretching value and the corresponding brightness ratio to obtain the stretched first histogram.

3. The multispectral image enhancement method according to claim 1 or claim 2, characterized in that: The method comprises: calculating the brightness level information of the first information region and the brightness level information of the second information region based on the low-frequency information to obtain a first histogram and a second histogram respectively. The method comprises: In the low-frequency information, counting the gradient value of each pixel, and comparing the gradient value with a preset threshold; Classify the first pixel whose gradient value is greater than the preset threshold into the first information area, and classify the second pixel whose gradient value is not greater than the preset threshold into the second information area; The number of each of the first pixels at each brightness level is counted in the first information region to obtain the first histogram, and the number of each of the second pixels at each brightness level is counted in the second information region to obtain the second histogram.

4. The multispectral image enhancement method according to claim 3, characterized in that: Calculating a stretch value according to the first histogram and the second histogram includes: Calculating a first average information intensity of the first information area according to the gradient value of each of the first pixels and the total number of the first pixels; Calculating a second average information intensity of the second information area according to the gradient value of each of the second pixels and the total number of the second pixels; Calculating a stretching parameter according to the first average information intensity, the second average information intensity, and the number of the first pixels; The stretching value is obtained by calculation according to the stretching parameter.

5. The multispectral image enhancement method according to claim 4, characterized in that: Calculating a stretching parameter according to the first average information intensity, the second average information intensity, and the number of the first pixels includes: Calculating a root mean square of the first average information strength and the second average information strength; Calculate a stretching coefficient according to the first average information intensity and the root mean square; The tensile strength is calculated according to the number of the first pixels and the root mean square.

6. The multispectral image enhancement method according to claim 5, characterized in that: Calculating the stretching value according to the stretching parameter includes: Determining a weight coefficient according to the stretch coefficient; The stretching strength and the values ​​of each brightness level in the first histogram are weighted and summed according to the weight coefficient to obtain the stretching value.

7. The multispectral image enhancement method according to claim 1 or claim 2, characterized in that: Calculating brightness mapping information according to the first histogram after the stretching process includes: Determining a target histogram in the first histogram; wherein the target histogram includes a plurality of the first histograms arranged in the first n brightness levels in order of brightness levels from high to low; wherein n is a positive integer; The sum of the target histogram is divided by the sum of all the first histograms, and the obtained quotient is multiplied by the maximum brightness level value to obtain the brightness enhancement level value corresponding to each brightness input level value.

8. The multispectral image enhancement method according to claim 1, characterized in that: Reconstructing the brightness information of the target image according to the low-frequency enhancement information and the high-frequency information to obtain the brightness reconstruction information includes: Add the low-frequency enhancement information of the last layer to the high-frequency information of the same layer to obtain the low-frequency enhancement information of the previous layer; The low-frequency enhancement information of the previous layer is added to the high-frequency information of the same layer to obtain the low-frequency enhancement information of the previous layer, and so on, until the low-frequency enhancement information of the first layer is added to the high-frequency information of the same layer to obtain the brightness reconstruction information.

9. The multispectral image enhancement method according to claim 1, characterized in that: The low-frequency information includes visible low-frequency information and invisible low-frequency information; or, the low-frequency information includes low-frequency fusion information obtained by fusion of visible low-frequency information and invisible low-frequency information; The high-frequency information includes visible high-frequency information and invisible high-frequency information; or, the high-frequency information includes high-frequency fusion information obtained by fusing visible high-frequency information with invisible high-frequency information.

10. The multispectral image enhancement method according to any one of claims 1, 2, 4 to 6, 8 and 9, characterized in that: Obtain low-frequency and high-frequency information of the target image, including: Acquiring brightness information of the target image; On the basis of maintaining the original resolution of the target image, the brightness information is decomposed in the frequency domain to obtain visible low-frequency information and visible high-frequency information, as well as invisible low-frequency information and invisible high-frequency information.

11. A multispectral image enhancement device, characterized in that: The device comprises: An acquisition module, used to acquire low-frequency information and high-frequency information of a target image; wherein the target image includes a visible light image and a non-visible light image; A statistical module, configured to collect the brightness level information of the first information region and the brightness level information of the second information region based on the low-frequency information, and obtain a first histogram and a second histogram respectively; wherein the average gradient value of the first information region is greater than the average gradient value of the second information region; a stretching module, configured to perform stretching processing on the first histogram according to the first histogram and the second histogram, and calculate brightness mapping information according to the first histogram after the stretching processing; An enhancement module, configured to perform enhancement processing on the low-frequency information according to the brightness mapping information to obtain low-frequency enhancement information; A reconstruction module, used to reconstruct the brightness information of the target image according to the low-frequency enhancement information and the high-frequency information to obtain brightness reconstruction information; The merging module is used to merge the brightness reconstruction information and the color information of the visible light image to obtain a multi-spectral fused image.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images