Image processing method, device, computing equipment and medium

Abstract

An image processing method includes: acquiring a first image of a target object using a first imaging method (S110); acquiring a second image of the target object using a second imaging method (S120); constructing a first sub-image and a second sub-image based on the first image (S130); constructing a third sub-image and a fourth sub-image based on the second image (S140); and determining pixel value distribution information of a fused image based on pixel value distribution information of each of the first sub-image, the second sub-image, the third sub-image, and the fourth sub-image (S150), thereby obtaining a fused image. Also provided are an image processing apparatus, a computing device, and a computer-readable storage medium.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more particularly, to an image processing method, an image processing apparatus, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In practical applications, different images can be acquired using different imaging methods. Because different imaging methods have different imaging characteristics, the images acquired using these methods contain different information about the target object. To obtain an image with more complete information about the target object, it is necessary to fuse the images acquired using these different imaging methods to create a fused image. This fused image contains more complete information about the target object.

[0003] In the process of realizing the concept of the present disclosure, the inventors discovered that there are at least the following problems in the related art.

[0004] When image fusion is performed using image fusion methods in related technologies, there are problems such as unclear outlines of target objects in the fused image, blurred target objects, and lost details of target objects, resulting in poor image fusion quality. Summary of the Invention

[0005] In view of this, the present disclosure provides an optimized image processing method and apparatus, as well as a computing device, a medium, and a program product.

[0006] One aspect of the present disclosure provides an image processing method, including: acquiring a first image of a target object using a first imaging method, acquiring a second image of the target object using a second imaging method, constructing a first sub-image and a second sub-image based on the first image, constructing a third sub-image and a fourth sub-image based on the second image, and determining pixel value distribution information of a fused image based on respective pixel value distribution information of the first sub-image, the second sub-image, the third sub-image, and the fourth sub-image, so as to obtain a fused image.

[0007] According to an embodiment of the present disclosure, the above-mentioned construction of the first sub-image and the second sub-image based on the first image includes: determining a first segmentation threshold based on pixel value distribution information of the first image, determining the category to which each pixel point in the first image belongs based on the first segmentation threshold, the category including a first category and a second category, constructing the first sub-image based on the pixel points in the first image belonging to the first category, and constructing the second sub-image based on the pixel points in the first image belonging to the second category.

[0008] According to an embodiment of the present disclosure, the above-mentioned determination of the category to which each pixel point in the first image belongs based on the first segmentation threshold includes: for each pixel point in the first image, determining whether the pixel value of the pixel point is greater than the first segmentation threshold; if so, determining that the pixel point belongs to the first category; if not, determining that the pixel point belongs to the second category.

[0009] According to an embodiment of the present disclosure, the above-mentioned determination of the first segmentation threshold based on the pixel value distribution information of the first image includes: determining a first coefficient, calculating a first threshold based on the pixel value distribution information of the first image using a maximum inter-class variance algorithm, and determining the first segmentation threshold based on the first coefficient and the first threshold.

[0010] According to an embodiment of the present disclosure, the above-mentioned construction of the third sub-image and the fourth sub-image based on the second image includes: performing filtering processing on the pixel value distribution information in the second image to determine the pixel value change rate of each pixel point in the second image relative to the adjacent pixel points, and for each pixel point in the second image, determining whether the pixel value change rate of the pixel point is less than a preset change rate, and if so, determining that the pixel point belongs to the third category; if not, determining that the pixel point belongs to the fourth category, constructing the third sub-image based on the pixel points in the second image that belong to the third category, and constructing the fourth sub-image based on the pixel points in the second image that belong to the fourth category.

[0011] According to an embodiment of the present disclosure, the filtering process for the pixel value distribution information in the second image includes: using a weighted least squares filter to filter the pixel value distribution information in the second image.

[0012] According to an embodiment of the present disclosure, the method further includes: determining a second segmentation threshold based on the pixel value distribution information of the second image. For any pixel point among the multiple pixel points of the fourth sub-image: determining a first enhancement coefficient based on the pixel value of the pixel point in the second image that matches the position information of the any pixel point, the second segmentation threshold, and a first predetermined function, obtaining the pixel value of the pixel point in the first image that matches the position information of the any pixel point as a second enhancement coefficient, and determining an enhanced pixel value of the any pixel value based on the pixel value of the any pixel point, the first enhancement coefficient, and the second enhancement coefficient. Constructing a fifth sub-image based on the enhanced pixel values ​​of each of the multiple pixel points of the fourth sub-image.

[0013] According to an embodiment of the present disclosure, the first predetermined function includes a monotonically decreasing function.

[0014] According to an embodiment of the present disclosure, the above-mentioned determination of the second segmentation threshold based on the pixel value distribution information of the second image includes: determining a second coefficient, calculating the second threshold based on the pixel value distribution information of the second image using the maximum inter-class variance algorithm, and determining the second segmentation threshold based on the second coefficient and the second threshold.

[0015] According to an embodiment of the present disclosure, the method further includes: respectively determining a first weight matrix for the first sub-image, a second weight matrix for the second sub-image, a third weight matrix for the third sub-image, a fourth weight matrix for the fourth sub-image, and a fifth weight matrix for the fifth sub-image. Determining the pixel value distribution information of the fused image includes: using the weight values ​​for any position in the first weight matrix, the second weight matrix, the third weight matrix, the fourth weight matrix, and the fifth weight matrix, performing a weighted summation on the pixel values ​​for the pixel points at any position in the first sub-image, the second sub-image, the third sub-image, the fourth sub-image, and the fifth sub-image, so as to determine the pixel value for the pixel point at any position in the fused image.

[0016] According to an embodiment of the present disclosure, determining the first weight matrix of the first sub-image includes: setting the first weight value in the first sub-image that matches the pixel points belonging to the first category to 1, setting the first weight value in the first sub-image that matches the pixel points other than the pixel points belonging to the first category to 0, and constructing the first weight matrix based on the first weight value.

[0017] According to an embodiment of the present disclosure, determining the second weight matrix of the second sub-image includes: for any pixel point in the second sub-image belonging to the second category, determining a first specific value based on the pixel value of the any pixel point, the first segmentation threshold and the second predetermined function, setting the second weight value in the second sub-image that matches the any pixel point to the first specific value, setting the second weight value in the second sub-image that matches the pixel points other than the pixel points belonging to the second category to 0, and constructing the second weight matrix based on the second weight value.

[0018] According to an embodiment of the present disclosure, determining the third weight matrix of the third sub-image includes: performing matrix addition on the first weight matrix and the second weight matrix to obtain a first specific matrix, performing matrix subtraction on the second specific matrix and the first specific matrix to obtain the third weight matrix, and each element in the second specific matrix is ​​1.

[0019] According to an embodiment of the present disclosure, determining the fourth weight matrix of the fourth sub-image includes: setting the third weight value in the fourth sub-image that matches the pixel points belonging to the fourth category to 1, and constructing the fourth weight matrix based on the third weight value.

[0020] According to an embodiment of the present disclosure, determining the fifth weight matrix of the fifth sub-image includes: determining the brightness value corresponding to the pixel value of each pixel point in the first image, determining the average of the brightness values ​​of multiple pixel points in the first image, determining a second specific value based on the average, setting the fourth weight value matching each pixel point in the fifth sub-image to the second specific value, and constructing the fifth weight matrix based on the fourth weight value.

[0021] According to an embodiment of the present disclosure, the first imaging method includes a positron emission tomography (PET) imaging method or a single photon emission PET imaging method, and the second imaging method includes a magnetic resonance imaging (MRI) method.

[0022] Another aspect of the present disclosure provides an image processing device, comprising: a first acquisition module, a second acquisition module, a first construction module, a second construction module, and a first determination module. The first acquisition module acquires a first image of a target object using a first imaging method. The second acquisition module acquires a second image of the target object using a second imaging method. The first construction module constructs a first sub-image and a second sub-image based on the first image. The second construction module constructs a third sub-image and a fourth sub-image based on the second image. The first determination module determines pixel value distribution information of a fused image based on pixel value distribution information of each of the first sub-image, the second sub-image, the third sub-image, and the fourth sub-image, so as to obtain a fused image.

[0023] According to an embodiment of the present disclosure, the above-mentioned construction of the first sub-image and the second sub-image based on the first image includes: determining a first segmentation threshold based on pixel value distribution information of the first image, determining the category to which each pixel point in the first image belongs based on the first segmentation threshold, the category including a first category and a second category, constructing the first sub-image based on the pixel points in the first image belonging to the first category, and constructing the second sub-image based on the pixel points in the first image belonging to the second category.

[0024] According to an embodiment of the present disclosure, the above-mentioned determination of the category to which each pixel point in the first image belongs based on the first segmentation threshold includes: for each pixel point in the first image, determining whether the pixel value of the pixel point is greater than the first segmentation threshold; if so, determining that the pixel point belongs to the first category; if not, determining that the pixel point belongs to the second category.

[0025] According to an embodiment of the present disclosure, the above-mentioned determination of the first segmentation threshold based on the pixel value distribution information of the first image includes: determining a first coefficient, calculating a first threshold based on the pixel value distribution information of the first image using a maximum inter-class variance algorithm, and determining the first segmentation threshold based on the first coefficient and the first threshold.

[0026] According to an embodiment of the present disclosure, the above-mentioned construction of the third sub-image and the fourth sub-image based on the second image includes: performing filtering processing on the pixel value distribution information in the second image to determine the pixel value change rate of each pixel point in the second image relative to the adjacent pixel points, and for each pixel point in the second image, determining whether the pixel value change rate of the pixel point is less than a preset change rate; if so, determining that the pixel point belongs to the third category; if not, determining that the pixel point belongs to the fourth category; constructing the third sub-image based on the pixel points in the second image that belong to the third category, and constructing the fourth sub-image based on the pixel points in the second image that belong to the fourth category.

[0027] According to an embodiment of the present disclosure, the filtering process for the pixel value distribution information in the second image includes: using a weighted least squares filter to filter the pixel value distribution information in the second image.

[0028] According to an embodiment of the present disclosure, the above-mentioned device also includes: a second determination module, a third determination module, a third acquisition module, a fourth determination module and a third construction module. Among them, the second determination module determines the second segmentation threshold based on the pixel value distribution information of the second image. For any pixel point among the multiple pixel points of the fourth sub-image: the third determination module determines the first enhancement coefficient based on the pixel value of the pixel point in the second image that matches the position information of the any pixel point, the second segmentation threshold and the first predetermined function. The third acquisition module obtains the pixel value of the pixel point in the first image that matches the position information of the any pixel point as the second enhancement coefficient. The fourth determination module determines the enhanced pixel value of the any pixel value based on the pixel value of the any pixel point, the first enhancement coefficient and the second enhancement coefficient. The third construction module constructs the fifth sub-image based on the enhanced pixel values ​​of each of the multiple pixel points of the fourth sub-image.

[0029] According to an embodiment of the present disclosure, the first predetermined function includes a monotonically decreasing function.

[0030] According to an embodiment of the present disclosure, the above-mentioned determination of the second segmentation threshold based on the pixel value distribution information of the second image includes: determining a second coefficient, calculating the second threshold based on the pixel value distribution information of the second image using the maximum inter-class variance algorithm, and determining the second segmentation threshold based on the second coefficient and the second threshold.

[0031] According to an embodiment of the present disclosure, the apparatus further includes: a fifth determination module for respectively determining a first weight matrix for the first sub-image, a second weight matrix for the second sub-image, a third weight matrix for the third sub-image, a fourth weight matrix for the fourth sub-image, and a fifth weight matrix for the fifth sub-image. Determining the pixel value distribution information of the fused image includes: using the weight values ​​for any position in the first weight matrix, the second weight matrix, the third weight matrix, the fourth weight matrix, and the fifth weight matrix, performing a weighted summation on the pixel values ​​for the pixel points at any position in the first sub-image, the second sub-image, the third sub-image, the fourth sub-image, and the fifth sub-image, so as to determine the pixel value for the pixel point at any position in the fused image.

[0032] According to an embodiment of the present disclosure, determining the first weight matrix of the first sub-image includes: setting the first weight value in the first sub-image that matches the pixel points belonging to the first category to 1, setting the first weight value in the first sub-image that matches the pixel points other than the pixel points belonging to the first category to 0, and constructing the first weight matrix based on the first weight value.

[0033] According to an embodiment of the present disclosure, determining the second weight matrix of the second sub-image includes: for any pixel point in the second sub-image belonging to the second category, determining a first specific value based on the pixel value of the any pixel point, the first segmentation threshold and the second predetermined function, setting the second weight value in the second sub-image that matches the any pixel point to the first specific value, setting the second weight value in the second sub-image that matches the pixel points other than the pixel points belonging to the second category to 0, and constructing the second weight matrix based on the second weight value.

[0034] According to an embodiment of the present disclosure, determining the third weight matrix of the third sub-image includes: performing matrix addition on the first weight matrix and the second weight matrix to obtain a first specific matrix, performing matrix subtraction on the second specific matrix and the first specific matrix to obtain the third weight matrix, and each element in the second specific matrix is ​​1.

[0035] According to an embodiment of the present disclosure, determining the fourth weight matrix of the fourth sub-image includes: setting the third weight value in the fourth sub-image that matches the pixel points belonging to the fourth category to 1, and constructing the fourth weight matrix based on the third weight value.

[0036] According to an embodiment of the present disclosure, determining the fifth weight matrix of the fifth sub-image includes: determining the brightness value corresponding to the pixel value of each pixel point in the first image, determining the average of the brightness values ​​of multiple pixel points in the first image, determining a second specific value based on the average, setting the fourth weight value matching each pixel point in the fifth sub-image to the second specific value, and constructing the fifth weight matrix based on the fourth weight value.

[0037] According to an embodiment of the present disclosure, the first imaging modality includes a positron emission tomography (PET) imaging modality or a single photon emission PET imaging modality, and the second imaging modality includes a magnetic resonance imaging (MRI) modality.

[0038] Another aspect of the present disclosure provides a computing device, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to perform the method described above.

[0039] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the method described above when executed.

[0040] Another aspect of the present disclosure provides a computer program product comprising computer executable instructions, which are used to implement the method described above when the instructions are executed.

[0041] According to the embodiments of the present disclosure, the image processing method of the embodiments of the present disclosure can at least partially solve the problem of image fusion using the image fusion method in the related art, such as unclear target object contours, blurred target objects, loss of target object details, etc., which leads to poor image fusion quality, and thus can achieve the technical effect of improving the quality of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0043] Figure 1 The following schematically shows a flow chart of an image processing method according to an embodiment of the present disclosure;

[0044] Figure 2 The overall framework diagram of the image processing method according to the embodiment of the present disclosure is schematically shown;

[0045] Figure 3 Schematically illustrates a schematic diagram of constructing a first sub-image and a second sub-image according to an embodiment of the present disclosure;

[0046] Figure 4 Schematic diagram of constructing a third sub-image and a fourth sub-image according to an embodiment of the present disclosure is schematically shown;

[0047] Figure 5-Figure 9 A schematic diagram of determining a weight matrix according to an embodiment of the present disclosure is schematically shown;

[0048] Figure 10-Figure 45 A schematic diagram schematically illustrates experimental results according to an embodiment of the present disclosure;

[0049] Figure 46 A block diagram schematically shows an image processing apparatus according to an embodiment of the present disclosure; and

[0050] Figure 47 The block diagram schematically shows a computer system suitable for image processing according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0051] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0052] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0053] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0054] When expressions such as "at least one of A, B and C, etc." are used, they should generally be interpreted in accordance with the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0055] Image fusion is typically performed based on multi-scale transformations. For example, a source image is decomposed into multiple sub-band coefficients at different scales and directions. These sub-band coefficients are then fused using specific fusion rules to produce a fused image. Multi-scale transformation theory is applied to image fusion because it aligns with human visual characteristics and can express essential image information.

[0056] Traditional multi-scale transforms include Laplacian Pyramid Blending and Discrete Wavelet Transform (DWT). Taking the DWT as an example, the DWT decomposes the source image into high-frequency and low-frequency sub-images using DWT. The high-frequency sub-images are fused using the maximum absolute value method to effectively preserve detailed texture information; the low-frequency sub-images are fused using the regional energy ratio rule to preserve the majority of the image information. The sub-images are then reconstructed into a fused image using the inverse DWT. Multi-scale geometric analysis (MGA), as a two-dimensional transform tool, can effectively represent image information such as edges, curves, lines, and textures. Typical examples of MGA include the Contourlet Transform (CMT) and the Shearlet Transform (SWT). Representative image fusion algorithms also include the Non-Subsampled Contourlet Transform (NSCT) and the Adaptive Sparse Representation (ASR) algorithm. The non-subsampled contourlet transform is an improved form of the contourlet transform. The non-subsampled contourlet transform has translation invariance and does not produce Gibbs effect.

[0057] However, image fusion algorithms in related technologies suffer from poor fusion quality. For example, due to the limited number of directions in the discrete wavelet transform, it is not effective in extracting information such as image texture edges. During wavelet decomposition and reconstruction, filter oscillations can create pseudo-contours at image edges, affecting the fusion effect. While the non-subsampled contourlet transform algorithm helps better preserve image edge information and contour structure, thereby enhancing translation invariance, the resulting fused image is blurry and some texture details are lost.

[0058] Given the poor fusion quality of image fusion algorithms in related technologies, the present disclosure proposes an image processing method. This method fuses images, improving the quality of the fused image. The specific implementation of the image processing method in the present disclosure is described below.

[0059] An embodiment of the present disclosure provides an image processing method, comprising: acquiring a first image of a target object using a first imaging method, and acquiring a second image of the target object using a second imaging method. Then, constructing a first sub-image and a second sub-image based on the first image, and constructing a third sub-image and a fourth sub-image based on the second image. Next, determining pixel value distribution information of a fused image based on pixel value distribution information of each of the first sub-image, the second sub-image, the third sub-image, and the fourth sub-image, to obtain the fused image.

[0060] Figure 1 The flowchart of the image processing method according to the embodiment of the present disclosure is schematically shown. Figure 2 The overall framework diagram of the image processing method according to the embodiment of the present disclosure is schematically shown. Figure 1 and Figure 2 An image processing method according to an embodiment of the present disclosure is described.

[0061] like Figure 1 As shown, the data processing method of the embodiment of the present disclosure includes, for example, the following operations S110 to S150.

[0062] In operation S110 , a first image of a target object is acquired using a first imaging method.

[0063] In operation S120 , a second image of the target object is acquired using a second imaging method.

[0064] According to an embodiment of the present disclosure, the first imaging modality may include nuclear imaging, which may include positron emission computed tomography (PET) or single-photon emission computed tomography (SPECT). The second imaging modality may include magnetic resonance imaging (MRI).

[0065] The first imaging modality has a high sensitivity, reaching the nanogram level, and can better capture functional and metabolic information about the target subject. However, the spatial resolution of the first imaging modality is relatively low. The imaging process of the first imaging modality may include, for example, labeling a compound or metabolic substrate with a radionuclide and introducing it into the target subject. The target subject may then undergo physiological metabolic activity based on the radionuclide-labeled compound or metabolic substrate. A first image of the target subject undergoing physiological metabolic activity is then acquired using the first imaging modality. This first image may represent the functional and metabolic information of the target subject.

[0066] The second imaging method has the advantage of high spatial resolution, which is advantageous for obtaining structural information about the target object. The second imaging method has a lower sensitivity, reaching the micromolecular level. The second image of the target object obtained using the second imaging method contains structural information about the target object.

[0067] In the disclosed embodiments, a first image of a target object is acquired using a first imaging modality, the first image including functional and metabolic information of the target object. A second image of the target object is acquired using a second imaging modality, the second image including structural information of the target object. The first and second images can be fused to produce a fused image that combines the functional and metabolic information of the first image with the structural information of the second image.

[0068] In operation S130 , a first sub-image and a second sub-image are constructed based on the first image.

[0069] According to an embodiment of the present disclosure, the image acquired by the first imaging method is, for example, a grayscale image. The first image can be obtained by performing pseudo-color processing on the grayscale image. Therefore, the first image, for example, is composed of a plurality of pixels, each of which may have a corresponding pixel value and a corresponding grayscale value, which is the grayscale value of the corresponding pixel in the grayscale image before the pseudo-color processing.

[0070] According to an embodiment of the present disclosure, a first segmentation threshold may be determined based on pixel value distribution information of the first image. The first segmentation threshold may be, for example, a value greater than or equal to 0 and less than or equal to 255. Then, based on the first segmentation threshold, the category to which each pixel in the first image belongs is determined. The two categories may include a first category and a second category.

[0071] For example, for each pixel in the first image, determine whether the pixel value of the pixel is greater than the first segmentation threshold. If so, determine that the pixel belongs to the first category. If not, determine that the pixel belongs to the second category.

[0072] Then, a first sub-image is constructed based on the pixels in the first image that belong to the first category, and a second sub-image is constructed based on the pixels in the first image that belong to the second category.

[0073] According to embodiments of the present disclosure, the constructed first sub-image, for example, includes important functional and metabolic information from the first image. The functional and metabolic information contained in the constructed second sub-image can be ignored. That is, the amount of functional and metabolic information contained in the first sub-image is far greater than that contained in the second sub-image. If the first image is a color image obtained through pseudo-color processing, the first sub-image includes most of the color information of the first image, while the second sub-image includes less color information.

[0074] According to an embodiment of the present disclosure, the image size of the first image may be, for example, 256*256, 512*512, etc. When the image size of the first image is 256*256, the number of pixels that can represent the first image is 256*256=65536. When the image size of the first image is 512*512, the number of pixels that can represent the first image is 512*512=262144.

[0075] Figure 3 A schematic diagram of constructing a first sub-image and a second sub-image according to an embodiment of the present disclosure is schematically shown.

[0076] The following will be combined Figure 3 The specific implementation process of operation S130 is described below.

[0077] To facilitate understanding of the process of constructing the first sub-image and the second sub-image based on the first image in the embodiment of the present disclosure, the embodiment of the present disclosure takes the image size of the first image as 3*3 as an example.

[0078] like Figure 3 As shown, the first image includes, for example, 9 pixels, which can be represented as a 11 ~a 33 . With pixel a 11 ~a 33 The one-to-one corresponding pixel value is P 11 ~P 33 The first segmentation threshold is, for example, represented by T Hpet When compared with pixel a 12 、a 21 、a 22 、a 32 One-to-one corresponding pixel value P 12 、P 21 、P 22 、P 32 are both greater than the first segmentation threshold T Hpet When determining pixel a 12 、a 21 、a 22 、a 32 Belongs to the first category.

[0079] Then, a first sub-image is constructed based on the pixels belonging to the first category. For example, the first sub-image is constructed based on the pixel value and position information corresponding to each pixel in the first category. The first sub-image includes, for example, pixel c 11 ~c 33 . The first sub-image has the same pixel a 12 、a 21 、a 22 、a 32 One-to-one corresponding pixel point is, for example, c 12 、c 21 、c 22 、c 32 . Pixel c in the first sub-image 12 、c 21 、c 22 、c 32 The pixel values ​​of the first image are respectively 12 、a 21 、a 22 、a 32 The pixel values ​​of the first sub-image are the same. 12 、c 21 、c 22 、c 32 For example, the pixel values ​​of all other pixel points except are set to 0.

[0080] Similarly, the construction process of the second sub-image is similar to that of the first sub-image.

[0081] For example, in the first image, the pixel a 11 、a 13 、a 23 、a 31 、a 33 One-to-one corresponding pixel value P 11 、P 13 、P 23 、P 31 、P 33 are all less than or equal to the first segmentation threshold T Hpet , determine pixel a 11 、a 13 、a 23 、a 31 、a 33 Belongs to the second category.

[0082] Then, a second sub-image is constructed based on the pixels belonging to the second category. For example, the second sub-image is constructed based on the pixel value and position information corresponding to each pixel in the second category. The second sub-image includes, for example, pixel d 11 ~d 33 . The second sub-image has the same pixel a 11、a 13 、a 23 、a 31 、a 33 A corresponding pixel point is, for example, d 11 d 13 d 23 d 31 d 33 Pixel d in the second sub-image 11 d 13 d 23 d 31 d 33 The pixel values ​​of the first image are respectively 11 、a 13 、a 23 、a 31 、a 33 The pixel values ​​of the second sub-image are the same. 11 d 13 d 23 d 31 d 33 For example, the pixel values ​​of all other pixel points except are set to 0.

[0083] Among them, the pixel value of the pixel point is compared with the first segmentation threshold T Hpet When comparing, the gray value of the pixel can be compared with the first segmentation threshold T Hpet Make a comparison.

[0084] According to an embodiment of the present disclosure, the first segmentation threshold T is determined based on the pixel value distribution information of the first image. Hpet The method may include: calculating a first segmentation threshold value based on pixel value distribution information of the first image using a maximum inter-class variance algorithm. Hpet For example, the first segmentation threshold T can be calculated based on the pixel value of each pixel point of the first image. Hpet , pixel values ​​may include grayscale values.

[0085] The principle of the maximum inter-class variance algorithm is, for example, to divide all pixels in the first image into a first category and a second category so that the variance between the grayscale values ​​corresponding to the pixels in the first category and the grayscale values ​​corresponding to the pixels in the second category is maximized. Hpet The calculation formula of is, for example, formula (1).

[0086] T Hpet =W Tpet ×T pet if T Hpet >255, T Hpet =255(1)

[0087] Among them, W Tpet is the first coefficient, and the value of the first coefficient is an empirical value. Tpet ∈[1,2];T pet is the first threshold value calculated based on the maximum inter-class variance method. Tpet and the first threshold T pet The first segmentation threshold T can be determined Hpet .

[0088] In operation S140, a third sub-image and a fourth sub-image are constructed based on the second image.

[0089] According to an embodiment of the present disclosure, for example, filtering processing is performed on the pixel value distribution information in the second image to determine the rate of change of the pixel value of each pixel point relative to adjacent pixels in the second image. For example, filtering processing is performed on the pixel value distribution information in the second image using a weighted least squares filter.

[0090] Then, for each pixel in the second image, a determination is made as to whether the rate of change of the pixel value of the pixel is less than a preset rate of change. If so, the pixel is determined to belong to the third category. If not, the pixel is determined to belong to the fourth category. The preset rate of change can be specifically set based on actual application circumstances.

[0091] Among them, determining the pixel value change rate of each pixel point in the second image relative to the adjacent pixel points, for example, includes: subtracting the pixel value of any pixel point in the second image from the pixel values ​​of multiple adjacent pixel points of the any pixel point to obtain multiple pixel difference values ​​corresponding one-to-one to the multiple adjacent pixel points, and performing weighted summation on the multiple pixel differences to obtain the pixel value change rate of the any pixel point.

[0092] Next, a third sub-image is constructed based on the pixels in the second image that belong to the third category, and a fourth sub-image is constructed based on the pixels in the second image that belong to the fourth category.

[0093] It can be understood that by filtering the pixel value distribution information in the second image, the pixels in the second image are classified according to the rate of change of each pixel value. Pixels belonging to the third category have a smaller rate of change of pixel values ​​relative to adjacent pixels, so that the constructed third sub-image includes smoothing information from the second image. Pixels belonging to the fourth category have a larger rate of change of pixel values ​​relative to adjacent pixels, so that the constructed fourth sub-image includes texture information from the second image.

[0094] According to an embodiment of the present disclosure, the image size of the second image may be, for example, 256*256, 512*512, etc. When the image size of the second image is 256*256, the number of pixels that can represent the second image is 256*256=65536. When the image size of the second image is 512*512, the number of pixels that can represent the second image is 512*512=262144.

[0095] Figure 4 A schematic diagram of constructing a third sub-image and a fourth sub-image according to an embodiment of the present disclosure is schematically shown.

[0096] The following will be combined Figure 4 The specific implementation process of operation S140 is described below.

[0097] To facilitate understanding of the process of constructing the third sub-image and the fourth sub-image based on the second image in the embodiment of the present disclosure, the embodiment of the present disclosure takes the image size of the second image as 3*3 as an example.

[0098] like Figure 4 As shown, the first image includes, for example, 9 pixels, which can be represented as b 11 ~b 33 . With pixel b 11 ~b 33 The one-to-one corresponding pixel value is Q 11 ~Q 33 When compared with pixel b 11 、b 13 、b 22 、b 31 、b 33 One-to-one corresponding pixel value Q 11 , Q 13 , Q 22 , Q 31 , Q 33 When the change rate of is less than the preset change rate, determine the pixel point b 11 、b 13 、b 22 、b 31 、b 33 Belongs to the third category. The pixel value change rate may be, for example, a grayscale value change rate.

[0099] Then, a third sub-image is constructed based on the pixels belonging to the third category. For example, a third sub-image is constructed based on the pixel value and position information corresponding to each pixel in the third category. The third sub-image includes, for example, pixel e 11 ~e 33 . The third sub-image has the same pixel b 11 、b 13 、b 22 、b31 、b 33 One-to-one corresponding pixel point is, for example, e 11 、e 13 、e 22 、e 31 、e 33 . Pixel e in the third sub-image 11 、e 13 、e 22 、e 31 、e 33 The pixel values ​​of the second image are respectively 11 、b 13 、b 22 、b 31 、b 33 The pixel values ​​of the third sub-image are the same. 11 、e 13 、e 22 、e 31 、e 33 For example, the pixel values ​​of all other pixel points except are set to 0.

[0100] Similarly, the construction process of the fourth sub-image is similar to that of the third sub-image.

[0101] For example, when the pixel b in the second image 12 、b 21 、b 23 、b 32 One-to-one corresponding pixel value Q 12 , Q 21 , Q 23 , Q 32 When the rate of change of is greater than or equal to the preset rate of change, determine the pixel point b 12 、b 21 、b 23 、b 32 Belongs to the fourth category.

[0102] Then, a fourth sub-image is constructed based on the pixels belonging to the fourth category. For example, the fourth sub-image is constructed based on the pixel value and position information corresponding to each pixel in the fourth category. The fourth sub-image includes, for example, pixel f 11 ~f 33 . The fourth sub-image is related to pixel b 12 、b 21 、b 23 、b 32 One-to-one corresponding pixel point is, for example, f 12 、f 21 、f 23 、f 32 . The pixels f12 and f 21 、f23 、f 32 The pixel values ​​of the second image are respectively 12 、b 21 、b 23 、b 32 The pixel values ​​of the fourth sub-image are the same. 12 、f 21 、f 23 、f 32 For example, the pixel values ​​of all other pixel points except are set to 0.

[0103] Next, in operation S150 , pixel value distribution information of the fused image is determined based on the respective pixel value distribution information of the first sub-image, the second sub-image, the third sub-image, and the fourth sub-image, so as to obtain the fused image.

[0104] Specifically, if Figure 2 As shown, the fourth sub-image can be enhanced to obtain a fifth sub-image. Then, the first sub-image, the second sub-image, the third sub-image, the fourth sub-image and the fifth sub-image are fused to obtain a fused image.

[0105] The following describes a process of performing enhancement processing on the fourth sub-image to obtain the fifth sub-image.

[0106] First, based on the pixel value distribution information of the second image, a second segmentation threshold is determined. The calculation process of the second segmentation threshold is similar to the calculation process of the first segmentation threshold.

[0107] The second segmentation threshold can be expressed as T Hmri , the maximum inter-class variance algorithm can be used to calculate the second segmentation threshold T based on the pixel value distribution information of the second image. Hmri The principle of the maximum inter-class variance algorithm is, for example, to divide all pixels in the second image into the third category and the fourth category so that the variance between the grayscale values ​​corresponding to the pixels in the third category and the grayscale values ​​corresponding to the pixels in the fourth category is maximized. Hmri The calculation formula of is, for example, formula (2).

[0108] T Hmri =W Tmri ×T mri , if T Hmri >255, T Hmri =255(2)

[0109] Among them, W Tmri is the second coefficient, and the value of the second coefficient is an empirical value. Tmri ∈[1,2];T mriis the second threshold value calculated based on the maximum inter-class variance method. Tmri and the second threshold T mri The second segmentation threshold T can be determined Hmri .

[0110] According to the embodiment of the present disclosure, i represents the i-th row in the fifth sub-image, and j represents the j-th column in the fifth sub-image. The pixel value of the pixel point in the i-th row and j-th column in the fifth sub-image is expressed as The process of constructing the fifth sub-image can be expressed as formula (3).

[0111]

[0112] in, For example, it represents the pixel value of the pixel in the i-th row and j-th column in the fourth sub-image. For example, it represents the pixel value of the pixel point in the i-th row and j-th column in the second image. For example, it represents the pixel value of the pixel at the i-th row and j-th column in the first image. Specifically, For example, the pixel value of the pixel at the i-th row and j-th column in the first image is normalized to the interval [0, 1]. In the embodiment of the present disclosure, the image sizes of the first image, the second image, the first sub-image, the second sub-image, the third sub-image, the fourth sub-image, and the fifth sub-image can all be the same.

[0113] The process of constructing the fifth sub-image will be described below in conjunction with formula (3).

[0114] After determining the second segmentation threshold T Hmri Then, for any pixel point among the plurality of pixels of the fourth sub-image, for example, the pixel point at the i-th row and j-th column in the fourth sub-image:

[0115] Based on the pixel value of the pixel point in the second image that matches the position information of any pixel point The second segmentation threshold T Hmri and a first predetermined function (cosine function), determine the first enhancement coefficient The first predetermined function may be a monotonically decreasing function, such as a cosine function whose function input value is in the interval [0, π / 2]. The pixel point in the second image that matches the position information of the any pixel point is reflected when: the position information of the any pixel point is the i-th row and j-th column in the fourth sub-image, and the position information of the pixel point in the second image that matches the position information of the any pixel point is also the i-th row and j-th column in the second image.

[0116] Then, the pixel value of the pixel point in the first image that matches the position information of any pixel point is obtained. as the second enhancement factor.

[0117] Based on the pixel value of any pixel The first enhancement coefficient and the second enhancement coefficient are used to determine the enhanced pixel value of any pixel value, and the enhanced pixel value can be used as the pixel value of the pixel point corresponding to the any pixel point in the fifth sub-image.

[0118] After obtaining the enhanced pixel value of each pixel in the fourth sub-image, the enhanced pixel value of each pixel can be used as the pixel value of the corresponding pixel in the fifth sub-image. That is, the pixel value of the pixel in the i-th row and j-th column in the constructed fifth sub-image is expressed as

[0119] It can be seen from formula (3) that the fourth sub-image can be enhanced by using formula (3). For each pixel in the partial pixel, the pixel satisfies the following two conditions at the same time:

[0120] (1) The pixel value of the pixel point is less than or equal to the second segmentation threshold.

[0121] (2) The pixel point corresponds to a pixel point in the first image whose pixel value is greater than the first segmentation threshold.

[0122] Among them, the enhancement principle of the first enhancement coefficient is reflected in that: for pixels in the second image whose pixel values ​​are less than or equal to the second segmentation threshold (classified as class A pixels), the smaller the pixel value, the greater the degree of enhancement of the fourth sub-image.

[0123] Among them, the enhancement principle of the second enhancement coefficient is reflected in that: among the pixels in the first image whose pixel values ​​are greater than the first segmentation threshold and whose position information matches the pixel points of category a, the larger the pixel value corresponding to the pixel point in the first sub-image, the greater the degree of enhancement of the fourth sub-image.

[0124] It can be understood that, according to formula (3), no enhancement processing is performed on the pixels other than the aforementioned pixels in the fourth sub-image. For each of the other pixels, the pixel satisfies either or both of the following two conditions:

[0125] (1) Corresponding to the pixel points in the second image whose pixel values ​​are greater than the second segmentation threshold.

[0126] (2) A pixel point corresponding to a pixel value in the second image that is less than or equal to the second segmentation threshold, and a pixel point corresponding to the pixel point position in the first image that has a pixel value that is less than or equal to the first segmentation threshold.

[0127] In the embodiment of the present disclosure, during the enhancement processing of the fourth sub-image, the pixel information of the second image and the first image (including pixel position and pixel value information) is comprehensively referenced, so that the fused image obtained by fusion using the fifth sub-image retains the texture information of the second image to a large extent.

[0128] In the embodiment of the present disclosure, since the fourth sub-image contains the texture information of the second image, the fifth sub-image is obtained by enhancing the fourth sub-image, so that the fused image obtained by fusion using the fifth sub-image retains the texture information of the second image to a large extent.

[0129] According to an embodiment of the present disclosure, after obtaining the first, second, third, fourth, and fifth sub-images, a first weight matrix for the first sub-image, a second weight matrix for the second sub-image, a third weight matrix for the third sub-image, a fourth weight matrix for the fourth sub-image, and a fifth weight matrix for the fifth sub-image can be determined, respectively, so that the sub-images can be fused using the respective weight matrices to obtain a fused image.

[0130] Figure 5-Figure 9 The figure schematically shows a diagram of determining a weight matrix according to an embodiment of the present disclosure.

[0131] The following will be combined Figure 3 、 Figure 4 and Figure 5-Figure 9 To describe how each weight matrix is ​​determined.

[0132] like Figure 5 As shown, determining a first weight matrix for a first sub-image may include:

[0133] A first weight value matching pixels belonging to the first category in the first sub-image is set to 1. A first weight value matching pixels other than pixels belonging to the first category in the first sub-image is set to 0. A first weight matrix is ​​then constructed based on the first weight values. The matrix size of the first weight matrix corresponds to the image size of the first sub-image. For example, if the image size of the first sub-image is 3*3, the matrix size of the first weight matrix is ​​also 3*3.

[0134] For example, Figure 3 and Figure 5 As shown, the pixels belonging to the first category include a 12 、a 21 、a 22 、a 32 .like Figure 5 As shown, the first weight matrix is ​​represented as W pet_1 The first weight matrix W pet_1 and pixel a 12 、a21 、a 22 、a 32 The corresponding first weight value is 1, and the first weight matrix W pet_1 The other first weight values ​​are 0.

[0135] like Figure 6 As shown, determining the second weight matrix for the second sub-image may include:

[0136] like Figure 3 As shown, the pixel point belonging to the second category in the second sub-image is a 11 、a 13 、a 23 、a 31 、a 33 Pixel a 11 、a 13 、a 23 、a 31 、a 33 The pixel value of any pixel in is expressed as The pixel value The pixel value is equal to the pixel value of the corresponding pixel in the first image.

[0137] For any pixel point belonging to the second category in the second sub-image, based on the pixel value of any pixel point The first segmentation threshold T Hpet and a second predetermined function (sine function) to determine a first specific value, which is expressed as Set the second weight value matching any pixel in the second sub-image to the first specific value First specific value For example, as shown in formula (4):

[0138]

[0139] The second weight values ​​that match the pixels in the second sub-image other than the pixels belonging to the second category are set to 0. Then, a second weight matrix is ​​constructed based on the second weight values.

[0140] For example, Figure 3 and Figure 6 As shown, the pixels belonging to the second category include a 11 、a 13 、a 23 、a 31 、a 33 .like Figure 6 As shown, the second weight matrix is ​​represented as W pet_2 The second weight matrix W pet_2 and pixel a 11 、a 13 、a23 、a 31 、a 33 The corresponding second weight values ​​are The second weight matrix W pet_2 The other second weight values ​​in are 0. From formula (4), we know that the second weight matrix W pet_2 Each element in is less than or equal to 1, that is, the second weight matrix W pet_2 Each element in is less than or equal to the first weight matrix W pet_1 Any element in .

[0141] In the disclosed embodiment, a larger first weight matrix is ​​set for the first sub-image, so that when performing image fusion, the color information in the first sub-image can be largely included, thereby preserving the functional and metabolic information of the first image to a large extent. A smaller second weight matrix is ​​set for the second sub-image, so that when performing image fusion, the color information of the second sub-image is weakened, thereby improving the overall contrast of the fused image and enhancing the visual effect.

[0142] like Figure 7 As shown, determining the third weight matrix of the third sub-image includes:

[0143] First, the first weight matrix W pet_1 and the second weight matrix W pet_2 Perform matrix addition to obtain the first specific matrix W pet .

[0144] Then, the second specific matrix W1 and the first specific matrix W pet Perform matrix subtraction to obtain the third weight matrix W mri_1 , each element in the second specific matrix W1 is 1.

[0145] like Figure 8 As shown, determining the fourth weight matrix of the fourth sub-image includes:

[0146] The third weight value for matching pixels belonging to the fourth category in the fourth sub-image is set to 1. The first weight value for matching pixels other than pixels belonging to the fourth category in the fourth sub-image can be set to either 0 or 1. In the disclosed embodiment, the first weight value is set to 0 as an example. Then, a fourth weight matrix is ​​constructed based on the first weight value. The matrix size of the fourth weight matrix corresponds to the image size of the fourth sub-image. For example, if the image size of the fourth sub-image is 3*3, the matrix size of the fourth weight matrix is ​​also 3*3.

[0147] For example, Figure 4 and Figure 8 As shown, the pixels belonging to the fourth category include b 12 、b 21、b 23 、b 32 .like Figure 8 As shown, the fourth weight matrix is ​​represented as W mri_2 The fourth weight matrix W mri_2 and pixel b 12 、b 21 、b 23 、b 32 The corresponding third weight value is 1, and the third weight matrix W mri_2 The other third weight values ​​can be either 0 or 1. In the embodiment of the present disclosure, for example, the other third weight value is 0.

[0148] like Figure 9 As shown, determining the fifth weight matrix of the fifth sub-image includes:

[0149] First, the brightness value corresponding to the pixel value of each pixel in the first image is determined. Then, the average brightness value of multiple pixels in the first image is determined. The average is, for example, I meanPET express.

[0150] Then, the second specific value W is determined based on the mean h_mri Among them, the second specific value W h_mri and mean I meanPET The relationship is shown in formula (5).

[0151]

[0152] Next, the fourth weight value matching each pixel in the fifth sub-image is set to the second specific value, and based on the fourth weight value, a fifth weight matrix W is constructed. H_mri , the fifth weight matrix W H_mri For example, Figure 9 shown.

[0153] In the embodiment of the present disclosure, after determining each weight, the first weight matrix W can be used to pet_1 , the second weight matrix W pet_2 , the third weight matrix W mri_1 , the fourth weight matrix W mri_2 And the fifth weight matrix W H_mri The weight value for any position in the first sub-image, the second sub-image, the third sub-image, the fourth sub-image and the fifth sub-image are weightedly summed to determine the pixel value for any position in the fused image.

[0154] For example, the image size of the fused image obtained by fusing the first sub-image, the second sub-image, the third sub-image, the fourth sub-image, and the fifth sub-image is the same as the image size of each sub-image. The pixel value of the pixel point in the i-th row and j-th column in the fused image is expressed as in, The calculation process is shown in formula (6) or formula (7).

[0155]

[0156] in, Respectively represent the pixel values ​​of the pixel points in the i-th row and j-th column in the first sub-image, the second sub-image, the third sub-image, the fourth sub-image, and the fifth sub-image.

[0157]

[0158] in, Represents the first specific matrix W pet The element in row i and column j of the first specific matrix W pet The first weight matrix W pet_1 and the second weight matrix W pet_2 The first specific matrix W is obtained by matrix addition. pet It can be used as the weight matrix of the first image as a whole. represents the pixel value of the pixel at row i and column j in the first image.

[0159] According to the embodiment of the present disclosure, the above formula (7) represents the pixel value of the pixel at row i and column j in the first image, represents the pixel value of the pixel at the i-th row and j-th column in the third sub-image. Examples of the meanings include:

[0160] When the third sub-image is fused with the first image, a better effect is obtained between color fidelity and detail texture preservation. Figure 7 As shown, due to the third weight matrix W mri_1 is obtained by combining the second specific matrix W1 and the first specific matrix W pet It can be seen that for the area with better color fidelity in the first image (the pixel points corresponding to the area are in the first specific matrix W pet The larger the weight value is), the less detail texture is retained in the third sub-image corresponding to the area (the pixel points corresponding to the area are in the third weight matrix W mri_1On the contrary, for an area with worse color fidelity in the first image, more detail texture is retained in the third sub-image corresponding to the area.

[0161] From the above formula (7), we can see that The visual effect of the enhanced fusion image is achieved. Specifically, due to the fifth weight matrix W H_mri is obtained based on the brightness value of each pixel in the first image. Therefore, by the fifth weight matrix W H_mri The pixel value of each pixel in the fifth sub-image During enhancement, the brightness characteristics of the first image are taken into account. For example, when the overall brightness of the first image is high, the elements in the fifth weight matrix have larger values; when the overall brightness of the first image is low, the elements in the fifth weight matrix have smaller values. Furthermore, because the fifth sub-image is obtained by enhancing the fourth sub-image using the second enhancement coefficient, after image fusion using the fifth sub-image, the pixels in the resulting fused image corresponding to pixels belonging to the first category in the first sub-image exhibit a clearer texture structure due to the second enhancement coefficient.

[0162] Figure 10-Figure 45 The figure schematically shows the experimental results according to the embodiment of the present disclosure.

[0163] In order to illustrate the fusion effect of image fusion using the method of the embodiment of the present disclosure, 36 groups of comparative experiments were conducted in the embodiment of the present disclosure. Figure 10-Figure 45 The experimental results of 36 groups of comparative experiments are shown in sequence.

[0164] Among them, Figure 10-Figure 45 In any of the figures, the images are from left to right: the first image, the second image, the fused image obtained by using the Laplace fusion algorithm, the fused image obtained by using the discrete wavelet fusion algorithm, the fused image obtained by using the non-subsampled contourlet transform algorithm, the fused image obtained by using the adaptive sparse representation algorithm, and the fused image obtained by using the fusion method of the embodiment of the present disclosure.

[0165] Table 1 shows, for example, evaluation results of fused images obtained by various methods using multiple evaluation indices.

[0166] Table 1

[0167] Fused images Standard deviation Spatial frequency Average gradient Mutual Information Cross Entropy bias index Image A 61.95 8.41 24.87 1.61 1.59 0.64 Image B 58.11 9.01 25.92 1.54 1.65 0.63 Image C 57.19 8.27 24.24 1.53 1.91 0.59 Image D 53.11 7.42 22.22 1.61 1.55 0.60 Image E 67.98 9.96 28.72 2.03 1.25 0.54

[0168] Among them, image A represents the fused image obtained by using the Laplace fusion algorithm, image B represents the fused image obtained by using the discrete wavelet fusion algorithm, image C represents the fused image obtained by using the non-subsampled contourlet transform algorithm, image D represents the fused image obtained by using the adaptive sparse representation algorithm, and image E represents the fused image obtained by using the fusion method of the embodiment of the present disclosure.

[0169] The multiple evaluation indicators include standard deviation, spatial frequency, average gradient, mutual information, cross entropy, deviation index, etc. These multiple evaluation indicators can usually be used to evaluate the quality of image fusion.

[0170] The evaluation index values ​​for each fused image in Table 1 are obtained by averaging the corresponding evaluation index values ​​for each of the 36 experimental data sets. For example, taking the standard deviation of image a as an example, by calculating the standard deviation of image a for each of the 36 experimental sets, we obtain 36 standard deviations corresponding to the 36 sets of experiments. The final standard deviation of image a is 61.95, which is then averaged.

[0171] The meaning of each evaluation indicator will be explained below.

[0172] The standard deviation is the square root of the root mean square error, which is the difference between the pixel values ​​for each pixel between the source image and the fused image.

[0173] Spatial frequency reflects the rate of change of image grayscale and can be used to reflect the clarity of an image. The clearer the image, the higher the spatial frequency.

[0174] The average gradient reflects the clarity of the fused image and can also measure the spatial resolution of the fused image. The larger the average gradient, the higher the spatial resolution.

[0175] Mutual information is an information metric that refers to the internal connection between two events. The larger the mutual information value, the more source image information is retained in the fused image, and the better the fusion effect.

[0176] Cross entropy reflects the difference between the grayscale values ​​of corresponding pixels in the source image and the fused image. The smaller the cross entropy value, the smaller the difference and the better the fusion effect.

[0177] The deviation index reflects the difference in spectral information between the source image and the fused image. The closer the deviation index is to 0, the smaller the difference is, and the closer the source image and the fused image are.

[0178] according to Figure 10-Figure 45It can be seen from the presented visual effects and the values ​​corresponding to the evaluation indicators in Table 1 that the quality of the fused image obtained by the image fusion method of the embodiment of the present disclosure is significantly better than the quality of the fused image obtained by other fusion methods.

[0179] Figure 46 The following schematically shows a block diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0180] like Figure 46 As shown, the image processing device 4600 may include a first acquisition module 4610 , a second acquisition module 4620 , a first construction module 4630 , a second construction module 4640 and a first determination module 4650 .

[0181] The first acquisition module 4610 can be used to acquire a first image of the target object using a first imaging method. According to an embodiment of the present disclosure, the first acquisition module 4610 can, for example, execute the above reference Figure 1 The operation S110 described above will not be repeated here.

[0182] The second acquisition module 4620 can be used to acquire a second image of the target object using a second imaging method. Figure 1 The operation S120 described above will not be repeated here.

[0183] The first construction module 4630 may be used to construct a first sub-image and a second sub-image based on the first image. Figure 1 The operation S130 described above will not be repeated here.

[0184] The second construction module 4640 may be used to construct a third sub-image and a fourth sub-image based on the second image. Figure 1 The operation S140 described above will not be repeated here.

[0185] The first determining module 4650 can be used to determine the pixel value distribution information of the fused image based on the pixel value distribution information of each of the first sub-image, the second sub-image, the third sub-image, and the fourth sub-image, so as to obtain the fused image. According to an embodiment of the present disclosure, the first determining module 4650 can, for example, execute the above reference Figure 1 The operation S150 described above will not be repeated here.

[0186] According to the modules, submodules, units, and subunits of the embodiments of the present invention, any multiple or at least part of the functions of any multiple thereof can be implemented in one module. According to the modules, submodules, units, and subunits of the embodiments of the present invention, any one or more thereof can be split into multiple modules for implementation. According to the modules, submodules, units, and subunits of the embodiments of the present invention, any one or more thereof can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware of any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware, and firmware or in an appropriate combination of any of them. Alternatively, according to the modules, submodules, units, and subunits of the embodiments of the present invention, one or more thereof can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is run.

[0187] For example, any multiple of the first acquisition module 4610, the second acquisition module 4620, the first construction module 4630, the second construction module 4640, and the first determination module 4650 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first acquisition module 4610, the second acquisition module 4620, the first construction module 4630, the second construction module 4640, and the first determination module 4650 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the first acquisition module 4610, the second acquisition module 4620, the first construction module 4630, the second construction module 4640 and the first determination module 4650 can be at least partially implemented as a computer program module, which can perform corresponding functions when executed.

[0188] Figure 47 The block diagram schematically shows a computer system suitable for image processing according to an embodiment of the present disclosure. Figure 47 The computer system shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0189] like Figure 47 As shown, the computer system 4700 according to an embodiment of the present disclosure includes a processor 4701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 4702 or a program loaded from a storage portion 4708 into a random access memory (RAM) 4703. The processor 4701 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 4701 may also include onboard memory for caching purposes. The processor 4701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0190] Various programs and data required for the operation of the system 4700 are stored in RAM 4703. The processor 4701, ROM 4702, and RAM 4703 are connected to each other via a bus 4704. The processor 4701 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in ROM 4702 and / or RAM 4703. It should be noted that the programs may also be stored in one or more memories other than ROM 4702 and RAM 4703. The processor 4701 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0191] According to an embodiment of the present disclosure, system 4700 may further include an input / output (I / O) interface 4705, which is also connected to bus 4704. System 4700 may also include one or more of the following components connected to I / O interface 4705: an input section 4706 including a keyboard, mouse, etc.; an output section 4707 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 4708 including a hard disk; and a communication section 4709 including a network interface card such as a LAN card or a modem. Communication section 4709 performs communication processing via a network such as the Internet. A drive 4710 is also connected to I / O interface 4705 as needed. Removable media 4711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in drive 4710 as needed, so that computer programs read from the removable media can be installed into storage section 4708 as needed.

[0192] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 4709, and / or installed from the removable medium 4711. When the computer program is executed by the processor 4701, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0193] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0194] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0195] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 4702 and / or RAM 4703 described above and / or one or more memories other than the ROM 4702 and RAM 4703 .

[0196] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0197] Those skilled in the art will appreciate that various combinations and / or combinations of features described in the various embodiments and / or claims of this disclosure may be made, even if such combinations or combinations are not explicitly described in this disclosure. In particular, various combinations and / or combinations of features described in the various embodiments and / or claims of this disclosure may be made, without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0198] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. An image processing method, comprising: Acquire a first image of the target object using a first imaging method; Acquiring a second image of the target object using a second imaging method; constructing a first sub-image and a second sub-image based on the first image, wherein the first sub-image and the second sub-image are constructed based on different segmentation categories to which pixels of the first image belong; constructing a third sub-image and a fourth sub-image based on the second image, wherein the third sub-image and the fourth sub-image are constructed based on different pixel value change rates of the second image; and determining pixel value distribution information of a fused image based on respective pixel value distribution information of the first sub-image, the second sub-image, the third sub-image, the fourth sub-image, and the fifth sub-image, to obtain a fused image; The method for constructing the fifth sub-image includes: determining a second segmentation threshold based on pixel value distribution information of the second image; For any pixel point among the multiple pixels of the fourth sub-image: determining a first enhancement coefficient based on a pixel value of a pixel point in the second image that matches the position information of the any pixel point, the second segmentation threshold, and a first predetermined function, wherein the first predetermined function includes a monotonically decreasing function; Obtaining a pixel value of a pixel point in the first image that matches the position information of any pixel point as a second enhancement coefficient; Determining an enhanced pixel value of the any pixel value based on the pixel value of the any pixel point, the first enhancement coefficient, and the second enhancement coefficient; A fifth sub-image is constructed based on the enhanced pixel values ​​of each of the plurality of pixel points of the fourth sub-image.

2. The method according to claim 1, wherein The constructing of the first sub-image and the second sub-image based on the first image includes: determining a first segmentation threshold based on pixel value distribution information of the first image; Determining, based on the first segmentation threshold, a category to which each pixel in the first image belongs, the categories including a first category and a second category; constructing the first sub-image based on the pixels in the first image belonging to the first category; and The second sub-image is constructed based on the pixels in the first image that belong to the second category.

3. The method according to claim 2, wherein: Determining the category to which each pixel in the first image belongs based on the first segmentation threshold includes: For each pixel in the first image, determining whether a pixel value of the pixel is greater than the first segmentation threshold; If yes, determining that the pixel point belongs to the first category; and If not, it is determined that the pixel point belongs to the second category.

4. The method according to claim 2 or 3, wherein: The determining of the first segmentation threshold based on the pixel value distribution information of the first image includes: determining a first coefficient; Calculating a first threshold value based on pixel value distribution information of the first image using a maximum inter-class variance algorithm; and The first segmentation threshold is determined based on the first coefficient and the first threshold.

5. The method according to any one of claims 1 to 3, wherein: The constructing of a third sub-image and a fourth sub-image based on the second image includes: performing filtering processing on pixel value distribution information in the second image to determine a pixel value change rate of each pixel point relative to adjacent pixels in the second image; For each pixel in the second image, determining whether a rate of change of a pixel value of the pixel is less than a preset rate of change; if so, determining that the pixel belongs to the third category; if not, determining that the pixel belongs to the fourth category; constructing the third sub-image according to the pixels belonging to the third category in the second image; and The fourth sub-image is constructed according to the pixels in the second image that belong to the fourth category.

6. The method according to claim 5, wherein: The filtering process for the pixel value distribution information in the second image includes: A weighted least squares filter is used to perform filtering processing on the pixel value distribution information in the second image.

7. The method according to claim 1, wherein The determining of the second segmentation threshold based on the pixel value distribution information of the second image includes: determining a second coefficient; Calculating a second threshold based on pixel value distribution information of the second image using a maximum inter-class variance algorithm; and The second segmentation threshold is determined based on the second coefficient and the second threshold.

8. The method according to claim 1, further comprising: respectively determining a first weight matrix for the first sub-image, a second weight matrix for the second sub-image, a third weight matrix for the third sub-image, a fourth weight matrix for the fourth sub-image, and a fifth weight matrix for the fifth sub-image; Among them, the determining of the pixel value distribution information of the fused image includes: using the weight values ​​for any position in the first weight matrix, the second weight matrix, the third weight matrix, the fourth weight matrix and the fifth weight matrix, performing weighted summation on the pixel values ​​of the pixel points for any position in the first sub-image, the second sub-image, the third sub-image, the fourth sub-image and the fifth sub-image to determine the pixel value for any position in the fused image.

9. The method according to claim 8, wherein Determining a first weight matrix for the first sub-image includes: Setting a first weight value of the first sub-image that matches the pixels belonging to the first category to 1; Setting a first weight value matching pixels other than pixels belonging to the first category in the first sub-image to 0; and constructing the first weight matrix based on the first weight value; The constructing of the first sub-image and the second sub-image based on the first image includes: determining a first segmentation threshold based on pixel value distribution information of the first image; Based on the first segmentation threshold, a category to which each pixel in the first image belongs is determined, where the categories include a first category and a second category.

10. The method according to claim 8, wherein Determining a second weight matrix for the second sub-image includes: For any pixel point in the second sub-image belonging to the second category, determining a first specific value based on a pixel value of the any pixel point, a first segmentation threshold, and a second predetermined function; Setting a second weight value matching the any pixel point in the second sub-image to the first specific value; Setting a second weight value matching pixels other than pixels belonging to the second category in the second sub-image to 0; and constructing the second weight matrix based on the second weight value; The constructing of the first sub-image and the second sub-image based on the first image includes: determining a first segmentation threshold based on pixel value distribution information of the first image; Based on the first segmentation threshold, a category to which each pixel in the first image belongs is determined, where the categories include a first category and a second category.

11. The method according to claim 8, wherein Determining a third weight matrix of the third sub-image includes: performing matrix addition on the first weight matrix and the second weight matrix to obtain a first specific matrix; and Matrix subtraction is performed on the second specific matrix and the first specific matrix to obtain the third weight matrix, where each element in the second specific matrix is ​​1.

12. The method according to claim 8, wherein Determining a fourth weight matrix of the fourth sub-image includes: Setting a third weight value of pixels in the fourth sub-image that match pixels belonging to the fourth category to 1; and constructing the fourth weight matrix based on the third weight value; The constructing of a third sub-image and a fourth sub-image based on the second image includes: performing filtering processing on pixel value distribution information in the second image to determine a pixel value change rate of each pixel point relative to adjacent pixels in the second image; For each pixel point in the second image, determine whether the pixel value change rate of the pixel point is less than a preset change rate; if yes, determine that the pixel point belongs to the third category; if not, determine that the pixel point belongs to the fourth category.

13. The method according to claim 8, wherein Determining a fifth weight matrix of the fifth sub-image includes: Determining a brightness value corresponding to a pixel value of each pixel in the first image; Determining an average of brightness values ​​of a plurality of pixels in the first image; determining a second specific value based on the mean; Setting the fourth weight value matched with each pixel in the fifth sub-image to be the second specific value; and Based on the fourth weight value, the fifth weight matrix is ​​constructed.

14. The method according to claim 1, wherein The first imaging modality includes a positron emission tomography (PET) or a single photon emission PET imaging (SPECT) modality, and the second imaging modality includes a magnetic resonance imaging (MRI) modality.

15. An image processing apparatus, comprising: A first acquisition module, which acquires a first image of the target object using a first imaging method; A second acquisition module, which acquires a second image of the target object using a second imaging method; A first construction module is configured to construct a first sub-image and a second sub-image based on the first image, wherein the first sub-image and the second sub-image are constructed based on different segmentation categories to which pixels of the first image belong; a second constructing module, constructing a third sub-image and a fourth sub-image based on the second image, wherein the third sub-image and the fourth sub-image are constructed based on different pixel value change rates of the second image; as well as a first determining module, configured to determine pixel value distribution information of a fused image based on pixel value distribution information of each of the first sub-image, the second sub-image, the third sub-image, the fourth sub-image, and the fifth sub-image, to obtain a fused image; a second determining module, determining a second segmentation threshold based on pixel value distribution information of the second image; For any pixel point among the multiple pixels of the fourth sub-image: a third determining module, configured to determine a first enhancement coefficient based on a pixel value of a pixel point in the second image that matches the position information of the any pixel point, the second segmentation threshold, and a first predetermined function, wherein the first predetermined function includes a monotonically decreasing function; a third acquisition module, which acquires a pixel value of a pixel point in the first image that matches the position information of any pixel point as a second enhancement coefficient; a fourth determining module, which determines an enhanced pixel value of the any pixel value based on the pixel value of the any pixel point, the first enhancement coefficient, and the second enhancement coefficient; The third constructing module constructs a fifth sub-image based on the enhanced pixel values ​​of the plurality of pixel points of the fourth sub-image.

16. The device according to claim 15, wherein The constructing of the first sub-image and the second sub-image based on the first image includes: determining a first segmentation threshold based on pixel value distribution information of the first image; Determining, based on the first segmentation threshold, a category to which each pixel in the first image belongs, the categories including a first category and a second category; constructing the first sub-image based on the pixels in the first image belonging to the first category; and The second sub-image is constructed based on the pixels in the first image that belong to the second category.

17. The device according to claim 16, wherein Determining the category to which each pixel in the first image belongs based on the first segmentation threshold includes: For each pixel in the first image, determining whether a pixel value of the pixel is greater than the first segmentation threshold; If yes, determining that the pixel point belongs to the first category; and If not, it is determined that the pixel point belongs to the second category.

18. The device according to claim 16 or 17, wherein The determining of the first segmentation threshold based on the pixel value distribution information of the first image includes: determining a first coefficient; Calculating a first threshold value based on pixel value distribution information of the first image using a maximum inter-class variance algorithm; and The first segmentation threshold is determined based on the first coefficient and the first threshold.

19. The device according to any one of claims 15 to 17, wherein: The constructing of a third sub-image and a fourth sub-image based on the second image includes: performing filtering processing on pixel value distribution information in the second image to determine a pixel value change rate of each pixel point relative to adjacent pixels in the second image; For each pixel in the second image, determining whether a rate of change of a pixel value of the pixel is less than a preset rate of change; if so, determining that the pixel belongs to the third category; if not, determining that the pixel belongs to the fourth category; constructing the third sub-image according to the pixels belonging to the third category in the second image; and The fourth sub-image is constructed according to the pixels in the second image that belong to the fourth category.

20. The device according to claim 19, wherein The filtering process for the pixel value distribution information in the second image includes: A weighted least squares filter is used to perform filtering processing on the pixel value distribution information in the second image.

21. The apparatus according to claim 15, wherein The determining of the second segmentation threshold based on the pixel value distribution information of the second image includes: determining a second coefficient; Calculating a second threshold based on pixel value distribution information of the second image using a maximum inter-class variance algorithm; and The second segmentation threshold is determined based on the second coefficient and the second threshold.

22. The apparatus according to claim 15, further comprising: a fifth determining module, configured to respectively determine a first weight matrix for the first sub-image, a second weight matrix for the second sub-image, a third weight matrix for the third sub-image, a fourth weight matrix for the fourth sub-image, and a fifth weight matrix for the fifth sub-image; Among them, the determining of the pixel value distribution information of the fused image includes: using the weight values ​​for any position in the first weight matrix, the second weight matrix, the third weight matrix, the fourth weight matrix and the fifth weight matrix, performing weighted summation on the pixel values ​​of the pixel points for any position in the first sub-image, the second sub-image, the third sub-image, the fourth sub-image and the fifth sub-image to determine the pixel value for any position in the fused image.

23. The device according to claim 22, wherein Determining a first weight matrix for the first sub-image includes: Setting a first weight value of the first sub-image that matches the pixels belonging to the first category to 1; Setting a first weight value matching pixels other than pixels belonging to the first category in the first sub-image to 0; and constructing the first weight matrix based on the first weight value; The constructing of the first sub-image and the second sub-image based on the first image includes: determining a first segmentation threshold based on pixel value distribution information of the first image; Based on the first segmentation threshold, a category to which each pixel in the first image belongs is determined, where the categories include a first category and a second category.

24. The device according to claim 22 or 23, wherein Determining a second weight matrix for the second sub-image includes: For any pixel point in the second sub-image belonging to the second category, determining a first specific value based on a pixel value of the any pixel point, a first segmentation threshold, and a second predetermined function; Setting a second weight value matching the any pixel point in the second sub-image to the first specific value; Setting a second weight value matching pixels other than pixels belonging to the second category in the second sub-image to 0; and constructing the second weight matrix based on the second weight value; The constructing of the first sub-image and the second sub-image based on the first image includes: determining a first segmentation threshold based on pixel value distribution information of the first image; Based on the first segmentation threshold, a category to which each pixel in the first image belongs is determined, where the categories include a first category and a second category.

25. The apparatus according to claim 22, wherein Determining a third weight matrix of the third sub-image includes: performing matrix addition on the first weight matrix and the second weight matrix to obtain a first specific matrix; and Matrix subtraction is performed on the second specific matrix and the first specific matrix to obtain the third weight matrix, where each element in the second specific matrix is ​​1.

26. The apparatus according to claim 22, wherein Determining a fourth weight matrix of the fourth sub-image includes: Setting a third weight value of pixels in the fourth sub-image that match pixels belonging to the fourth category to 1; and constructing the fourth weight matrix based on the third weight value; The constructing of a third sub-image and a fourth sub-image based on the second image includes: performing filtering processing on pixel value distribution information in the second image to determine a pixel value change rate of each pixel point relative to adjacent pixels in the second image; For each pixel point in the second image, determine whether the pixel value change rate of the pixel point is less than a preset change rate; if yes, determine that the pixel point belongs to the third category; if not, determine that the pixel point belongs to the fourth category.

27. The apparatus according to claim 22, wherein Determining a fifth weight matrix of the fifth sub-image includes: Determining a brightness value corresponding to a pixel value of each pixel in the first image; Determining an average of brightness values ​​of a plurality of pixels in the first image; determining a second specific value based on the mean; Setting the fourth weight value matched with each pixel in the fifth sub-image to be the second specific value; and Based on the fourth weight value, the fifth weight matrix is ​​constructed.

28. The apparatus according to claim 15, wherein The first imaging modality includes a positron emission tomography (PET) or a single photon emission PET imaging (SPECT) modality, and the second imaging modality includes a magnetic resonance imaging (MRI) modality.

29. A computing device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to perform the method according to any one of claims 1 to 14.

30. A computer-readable storage medium having executable instructions stored thereon, wherein when the instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 14.

31. A computer program product comprising computer executable instructions, wherein the instructions are configured to implement the method according to any one of claims 1 to 14 when executed.

Images