Medical image processing method, related method and electronic equipment

Abstract

The invention provides a medical image processing method, a related method and electronic equipment, and the method comprises the steps: extracting a first degradation kernel of a first image, the first image being a medical image with a first resolution; the first degradation kernel is processed based on a mapping relation to obtain a second degradation kernel, the mapping relation is used for reflecting the relation between a first value of the first degradation kernel and a second value of the second degradation kernel, the second value is in positive correlation with the first value, and the interval where the first value is located comprises a first interval and a second interval larger than the first interval; the increasing rate of the second value in the first interval is smaller than that in the second interval; the second degradation kernel acts on a second image to obtain a third image, the second image is a medical image with a second resolution, and the second resolution is greater than the first resolution. Therefore, the third image can be closer to a real low-resolution image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to a medical image processing method and related methods and electronic equipment. Background Art

[0002] In the medical field, for the super-resolution problem of unpaired images (i.e., for low-resolution images (such as capsule gastroscopy images) that do not have paired high-resolution images), it is usually necessary to use degradation kernels to construct "paired data" and then train the super-resolution model through supervised learning. The key to the success of this type of solution is to make the constructed degradation kernel close to the real degradation kernel during the process of constructing "paired data".

[0003] Currently, there are two main approaches to constructing degradation kernels. The first approach is to manually design the kernel function based on the physical process of image degradation (i.e., manual design). The second approach is to use a machine learning model to automatically learn the degradation kernel by analyzing low-resolution images.

[0004] However, the manually designed degradation kernel in the first solution has poor adaptability and limited generalization ability. For the second solution, it was found in practice that the degradation kernel extracted when processing unpaired images is not accurate enough, resulting in inaccurate high-resolution medical images. For example, in capsule gastroscopy image super-resolution, there is a situation of false interference in the high-resolution medical image obtained by the degradation kernel extracted from the capsule gastroscopy image. Summary of the invention

[0005] The present disclosure is proposed in view of the above situation, and its purpose is to provide a medical image processing method and related methods and electronic devices that can make the third image closer to the real low-resolution image.

[0006] To this end, a first aspect of the present disclosure provides a medical image processing method, including: extracting a first degradation kernel of a first image, wherein the first image is a medical image of a first resolution; processing the first degradation kernel based on a mapping relationship to obtain a second degradation kernel, wherein the mapping relationship is used to reflect the relationship between a first value of the first degradation kernel and a second value of the second degradation kernel, wherein the second value is positively correlated with the first value, an interval in which the first value is located includes a first interval and a second interval greater than the first interval, and an increasing rate of the second value in the first interval is less than an increasing rate in the second interval; and applying the second degradation kernel to the second image to obtain a third image, wherein the second image is a medical image of a second resolution, and the second resolution is greater than the first resolution.

[0007] In the first aspect of the present disclosure, the first degradation kernel can be corrected by a mapping relationship to obtain a more accurate degradation kernel, thereby making the third image closer to the real low-resolution image. In addition, the first degradation kernel is directly extracted from the first image, and when the first degradation kernel is applied to super-resolution, the clarity of the super-resolution image can be improved. In addition, the second value of the second degradation kernel is positively correlated with the first value of the first degradation kernel, so that the second degradation kernel can retain the main features of the first degradation kernel. In addition, the increasing rate of the second value in the first interval is less than the increasing rate in the second interval, which can make the larger values ​​in the degradation kernel change more, thereby reducing the flatness of the values ​​in the degradation kernel (that is, the change of the values ​​in the degradation kernel can become sharper), and improving the accuracy of the degradation kernel (that is, the second degradation kernel can be closer to the real degradation kernel).

[0008] In addition, in the medical image processing method involved in the first aspect of the present disclosure, optionally, KernelGAN is used to obtain the first degradation kernel, thereby facilitating direct input of the first image to directly obtain the degradation kernel.

[0009] In addition, in the medical image processing method involved in the first aspect of the present disclosure, optionally, the mapping relationship is based on a monotonically increasing function; the second degenerate kernel is obtained based on the output value of the monotonically increasing function, and the increasing rate of the monotonically increasing function in the first interval is less than the increasing rate in the second interval, wherein the mapping relationship satisfies the first formula: t 1 (m, n)=f(v(m, n)), wherein f represents the monotonically increasing function, v(m, n) represents a value related to the first value at the position (m, n) of the first degenerate kernel, and t 1 (m, n) represents the output value of the monotonically increasing function. Thus, the smoothness of converting the first value to the second value can be adjusted, and the increasing rate can be finely adjusted and the range of the output value can be controlled.

[0010] In addition, in the medical image processing method involved in the first aspect of the present disclosure, optionally, the numerical value related to the first value is the first value after preprocessing, and the preprocessing is to convert the first value into a first preset interval. In this case, the monotonically increasing function can be made fairer when processing different first values. In addition, the fixed first preset interval can also facilitate the determination of the variation interval of the first value to better analyze the influence of the increasing rate of different intervals on the flatness of the degradation kernel (for example, it is convenient to consider the setting of the increasing rate based on the maximum value and the minimum value). In addition, it can also help to control the interval of the output value of the monotonically increasing function to be consistent with the first preset interval, thereby reducing the risk of the second degradation kernel deviating too much from the first degradation kernel.

[0011] In addition, in the medical image processing method involved in the first aspect of the present disclosure, optionally, the conversion process satisfies the formula:

[0012]

[0013] Among them, u(m, n) represents the first value after transformation when the position of the first degenerate core is (m, n), x(m, n) represents the first value when the position of the first degenerate core is (m, n), M and N represent the size of the first degenerate core, and max represents the maximum value function.

[0014] In addition, in the medical image processing method involved in the first aspect of the present disclosure, optionally, the first value is processed based on the mapping relationship to output an intermediate value, and the intermediate value is divided by the sum of all the intermediate values ​​to obtain the second value. In this case, the sum of all the second values ​​can be made 1, which is suitable for a degenerate kernel characterized by requiring the sum of all values ​​to be 1.

[0015] In addition, in the medical image processing method involved in the first aspect of the present disclosure, optionally, the first image is a capsule gastroscope image, and the second image is a gastroscope image other than the capsule gastroscope image. Thus, the accuracy of the degradation core of the capsule gastroscope image can be improved, and the super-resolution of the unpaired capsule gastroscope image can be achieved, and the clarity of the super-resolution image corresponding to the capsule gastroscope image can be improved.

[0016] A second aspect of the present disclosure provides a method for generating a training data set, comprising: acquiring multiple second images; executing the medical image processing method involved in the first aspect to acquire a third image through each of the multiple second images; taking each of the multiple second images and the third image corresponding to each of the multiple second images as a pair of paired images to acquire multiple pairs of the paired images; and taking the multiple pairs of the paired images as the training data set.

[0017] The third aspect of the present disclosure provides a medical image super-resolution method, comprising: inputting a first image to be super-resolved into a super-resolution model trained by a training data set generated by the method for generating a training data set involved in the second aspect of the present disclosure to obtain a super-resolution image. In this case, the training data set is generated using the corrected degenerate kernel, and the super-resolution model is trained using the training data set, which can improve the problem of false interference while improving the clarity of the super-resolution image.

[0018] The fourth aspect of the present disclosure provides an electronic device, comprising a processor and a memory, wherein the processor executes a program stored in the memory to implement the medical image processing method involved in the first aspect of the present disclosure, the method for generating a training data set involved in the second aspect of the present disclosure, or the super-resolution method of medical images involved in the third aspect of the present disclosure.

[0019] According to the present disclosure, a medical image processing method and related methods and electronic devices are provided, which can make a third image closer to a real low-resolution image. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present disclosure will now be explained in further detail, by way of example only, with reference to the accompanying drawings.

[0021] Figure 1 is a schematic diagram showing a super-resolution environment of a capsule gastroscopy image involved in the example of the present disclosure.

[0022] Figure 2 is an exemplary flow chart showing a processing method involved in the examples of the present disclosure.

[0023] Figure 3A is a schematic diagram showing a first image which is a capsule gastroscopy image, Figure 3B It shows Figure 3A is a schematic diagram of a partial enlargement of the area 801, Figure 3C is the result when the first degenerate kernel is not corrected. Figure 3A Schematic diagram of a super-resolution image corresponding to region 801.

[0024] Figure 4 is a schematic diagram showing a curve corresponding to a monotonically increasing function involved in an example of the present disclosure.

[0025] Figure 5A is a three-dimensional schematic diagram showing the degenerate kernel before correction, Figure 5B is a three-dimensional schematic diagram showing the corrected degenerate kernel.

[0026] Fig. 6A is a two-dimensional schematic diagram showing the degenerate kernel before correction, Figure 6B is a two-dimensional schematic diagram showing the corrected degenerate kernel.

[0027] Fig. 7A is the result when the first degenerate kernel is not corrected. Figure 3A A schematic diagram of a super-resolution image corresponding to region 801, Figure 7B is shown after correction of the first degenerate kernel Figure 3A Schematic diagram of a super-resolution image corresponding to region 801.

[0028] Figure 8is an exemplary flow chart showing a generation method involved in the examples of the present disclosure. DETAILED DESCRIPTION

[0029] Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present disclosure are described in detail. In the following description, the same symbols are given to the same components, and repeated descriptions are omitted. In addition, the accompanying drawings are only schematic diagrams, and the ratio of the dimensions of the components to each other or the shapes of the components, etc. may be different from the actual ones. It should be noted that the terms "including" and "having" in the present disclosure and any variations thereof, such as a process, method, system, product or device including or having a series of steps or units, are not necessarily limited to those steps or units clearly listed, but may include or have other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] First, relevant terms involved in the present disclosure are introduced.

[0031] A "degradation kernel" may be a representation of an image blurring process. The degradation kernel may be used to simulate how a low-resolution image is degraded from a high-resolution image. In some examples, the degradation kernel may be convolved with the image to obtain a degraded image.

[0032] "Super-resolution" can refer to the process of obtaining a high-resolution image by using a low-resolution image.

[0033] In response to the above-mentioned problems, the inventors have proposed a medical image processing method (also referred to as a processing method, or a low-resolution image acquisition method, etc.) after research, which can improve the accuracy of the degradation kernel (that is, it can make the degradation kernel closer to the real degradation kernel), and thus can make the third image closer to the real low-resolution image. In addition, the processing method involved in the example of the present disclosure, when the medical image is a gastroscopic image, for example, the degradation kernel of the capsule gastroscopic image is applied to other high-resolution gastroscopic images, can also be called a gastroscopic image processing method. In this way, the accuracy of the degradation kernel of the capsule gastroscopic image can be improved. In some examples, the processing method involved in the example of the present disclosure may not be limited to medical images. For example, it can be applied to any image with similar problems as the present disclosure. In this case, the processing method involved in the example of the present disclosure may also be called an image processing method.

[0034] Taking capsule gastroscopy images as an example, in order to improve the clarity of capsule gastroscopy images, the inventors found that there was a deviation between the extracted degraded core and the real degraded core when extracting the degraded core. Although the extracted degraded core can help improve the image clarity, it also increases the erroneous interference (i.e., false interference). For example, the saliva that was originally suspended in the capsule gastroscopy image is over-highlighted after processing, and intuitively looks like saliva stuck on the lens. For another example, a small dot in the white mucosa on the stomach wall is over-highlighted after processing, and it is easy to be misjudged as the presence of white particles. These interferences have a negative impact on the accuracy and experience of clinical applications.

[0035] In the present disclosure, in response to the problem of erroneous interference, the inventors have found through research that if the extracted degraded kernel is relatively flat, the result after using the degraded kernel (i.e., the super-resolution image) is prone to the problem of erroneous interference. The inventors use the extracted degraded kernel as a starting point and correct it to reduce the existence of erroneous interference (for example, reduce the existing white spot abnormality). In this case, while improving the clarity of the super-resolution image, the existence of erroneous interference can be reduced (i.e., on the one hand, the clarity can be improved, and on the other hand, the interference can be reduced), which can improve the accuracy and experience of super-resolution of capsule gastroscopy images in clinical applications.

[0036] In some examples, the scheme involved in the examples of the present disclosure can be applied to the super-resolution of unpaired images. For example, for a capsule gastroscopy image (also called a low-resolution image), it is generally difficult to obtain a high-resolution image paired with it. Therefore, the super-resolution of capsule gastroscopy images can belong to the super-resolution of unpaired images. In the super-resolution of unpaired images, the "paired data" is constructed by the degradation kernel of the low-resolution image, and then the super-resolution model is trained by supervised learning, so the accuracy of the constructed degradation kernel is crucial.

[0037] In some examples, the scheme involved in the examples of the present disclosure can be applied to the case where the high-resolution image obtained by the degenerate kernel has erroneously added highlight interference. For example, for the degenerate kernel of the capsule gastroscopy image, the white spots on the saliva or mucosa are overly highlighted in the result after using the degenerate kernel.

[0038] The examples of the present disclosure will be described in detail below. It should be noted that although many descriptions and examples involve capsule gastroscopy images, they do not limit the present disclosure. Unless there is a contradiction, the scheme of the present disclosure can be applied to any medical image or other image with similar problems to the present disclosure in which the high-resolution medical image obtained by extracting the degradation core in the super-resolution process has the problem of false interference. Taking medical images as an example, in some other medical scenarios, in the super-resolution process, due to various reasons (such as the existence of non-pairing), it is necessary to extract the degradation core from the medical image collected by the application scenario. Since the extracted degradation core is different from the real degradation core, there is false interference in the super-resolution image finally obtained. For this case, the scheme of the present disclosure is also applicable. For example, the medical image can be a bronchoscope image, an intravascular ultrasound image, or an X-ray image.

[0039] In addition, for the sake of convenience of description, the degradation kernel extracted from the medical image is referred to as the first degradation kernel, the degradation kernel after correction is referred to as the second degradation kernel, and the numerical value (i.e., weight value) in the first degradation kernel is referred to as the first value of the first degradation kernel, and the numerical value in the second degradation kernel is referred to as the second value of the second degradation kernel.

[0040] Figure 1 is a schematic diagram showing a super-resolution environment of a capsule gastroscopy image involved in the example of the present disclosure. Figure 1 A portion of a capsule endoscopy system 900 is shown in FIG.

[0041] In some examples, for capsule gastroscopy images, the super-resolution method (described later) involved in the present disclosure can be applied to Figure 1 In the super-resolution environment shown. Figure 1 In some examples, a super-resolution device 100 and a capsule endoscope system 900 may be included in a super-resolution environment. The capsule endoscope system 900 may be connected to the super-resolution device 100 via a communication link or may be integrated with the super-resolution device 100. The capsule endoscope system 900 may collect image data in the gastric cavity, and the image data obtained by the capsule endoscope system 900 may be received and processed by the super-resolution device 100 to obtain a super-resolution image. That is, the super-resolution device 100 may be used to implement one or more steps of the super-resolution method involved in the examples of the present disclosure to obtain a super-resolution image. In some examples, the super-resolution device 100 may also receive image data from other acquisition devices other than the capsule endoscope system 900 to obtain a super-resolution image.

[0042] In some examples, the capsule endoscope system 900 may include a capsule endoscope 910. The capsule endoscope 910 may move in the digestive tract of the subject. In some examples, the capsule endoscope 910 may reach the stomach cavity of the subject via the digestive tract to obtain image data in the stomach cavity. In some examples, the image data collected by the capsule endoscope 910 may be sent to the outside via a wireless signal (or feedback signal).

[0043] Figure 2 is an exemplary flow chart showing a processing method involved in the examples of the present disclosure.

[0044] Before describing the super-resolution method involved in the examples of the present disclosure, the processing method involved in the super-resolution method is described first. Figure 2 , the processing method may include extracting a first degradation kernel of the first image (step S101), processing the first degradation kernel based on the mapping relationship to obtain a second degradation kernel (step S102), and applying the second degradation kernel to the second image to obtain a third image (step S103). In this case, the first degradation kernel can be corrected through the mapping relationship to obtain a more accurate degradation kernel, thereby making the third image closer to the real low-resolution image.

[0045] In some examples, reference Figure 2 In step S101, the first image may be a medical image requiring super-resolution. That is, the first image may be a medical image with a lower resolution. In some examples, the first image may be a medical image with a first resolution.

[0046] In some examples, the first image may be a medical image acquired in an application scenario. Thus, it is convenient to make the first image and the medical image that ultimately requires super-resolution be images acquired in the same scenario. For example, the degradation kernel extracted from a capsule gastroscopy image may be used for super-resolution of a newly acquired capsule gastroscopy image. In some examples, the medical image may be a gastroscopy image, and the first image may be a capsule gastroscopy image.

[0047] In addition, the first degradation kernel may be a degradation kernel of the first image. In this case, the first degradation kernel is directly extracted from the first image, and can improve the clarity of the super-resolution image. In some examples, the degradation kernel may be two-dimensional. That is, the size of the degradation kernel may be expressed as M×N, where M and N may represent rows and columns, respectively, or columns and rows, respectively.

[0048] In some examples, the first image may be downsampled to obtain the first degradation kernel. For example, the first image may be downsampled to obtain a downsampled image having a lower resolution than the first image, and the first degradation kernel may be obtained based on the first image and the downsampled image.

[0049] In some examples, the degradation kernel can be learned from the first image based on a deep learning model. In this case, learning the degradation kernel from the low-resolution image based on the deep learning model can improve the accuracy of the degradation kernel.

[0050] In some examples, KernelGAN (Blind Super-Resolution KernelEstimation using an Internal-GAN) can be used to extract the first degradation kernel of the first image. Thus, it is convenient to directly input the first image to directly obtain the degradation kernel. It should be noted that this does not represent a limitation on the present disclosure.

[0051] For KernelGAN, its basic principle is that there are a large number of repeated small patches between cross-scale images, so that when the patch similarity between cross-scale images is the largest, it is the desired degradation kernel. KernelGAN is based on the principle of GAN (Generative Adversarial Network) to learn the degradation kernel corresponding to each image. Its training process consists of a generator and a discriminator. During the training process, the generator is reduced by a specific multiple and continuously iterated, so that the generated result is more similar to the original image in distribution, and the discriminator is also continuously iterated to further improve its ability to distinguish the generated result from the original image. In the constant confrontation between the generator and the discriminator, the two are constantly iterating, and the result of the convolution of each convolution kernel in the final generator is the final extracted degradation kernel.

[0052] Figure 3A is a schematic diagram showing a first image which is a capsule gastroscopy image. Figure 3B It shows Figure 3A A schematic diagram of a partial enlargement of area 801. Figure 3C is the result when the first degenerate kernel is not corrected. Figure 3A Schematic diagram of a super-resolution image corresponding to region 801.

[0053] As mentioned above, the high-resolution medical images obtained by the degenerate kernel have the problem of adding interference by mistake. To this end, the present disclosure also provides an example of adding interference of highlight white points by mistake by taking capsule gastroscopy images and degenerate kernels extracted by KernelGAN as examples. Figure 3A , Figure 3B and Figure 3C , it can be seen that although Figure 3A The super-resolution image of region 801 (see Figure 3C) has better clarity, but some of the points are over-highlighted in the super-resolution image (see area 811). That is, when not corrected, the degraded kernel extracted by KernelGAN causes some points of the super-resolution capsule gastroscopy image to be over-highlighted. In other words, the degraded kernel extracted by KernelGAN significantly improves the clarity, but also brings additional interference.

[0054] In some examples, reference Figure 2 In step S102, the first value of the first degenerate core is processed based on the mapping relationship to obtain the second value of the second degenerate core, thereby obtaining the second degenerate core. In the following, for the convenience of description, the process of processing the first degenerate core to obtain the second degenerate core is referred to as correction.

[0055] In addition, the mapping relationship can be used to reflect the relationship between the first value of the first degenerate core and the second value of the second degenerate core. That is, based on the mapping relationship, the first value of the corresponding position of the first degenerate core can be converted into the second value of the corresponding position of the second degenerate core.

[0056] In some examples, in the mapping relationship, the second value of the second degenerate core may be positively correlated with the first value of the first degenerate core. That is, the larger the first value, the larger the second value at the position corresponding to the first value. In this case, the second degenerate core can retain the main features of the first degenerate core.

[0057] In some examples, the mapping relationship can be used to make the change of the first value closer to the maximum value (for example, 1) more obvious. In some examples, when the first value is small, the increment rate of the second value can be slower; when the first value is large, the increment rate of the second value can be faster. Specifically, the interval in which the multiple first values ​​of the first degenerate kernel are located may include a first interval and a second interval larger than the first interval, and the increment rate of the second value in the first interval may be smaller than the increment rate in the second interval. In this case, the larger values ​​in the degenerate kernel can be changed more, thereby reducing the flatness of the values ​​in the degenerate kernel (that is, the change of the values ​​in the degenerate kernel can be made sharper), and the accuracy of the degenerate kernel can be improved (that is, the second degenerate kernel can be closer to the real degenerate kernel). The flatter the degenerate kernel, the more blurred the image will be when it acts on the image. When the image pair generated by this degenerate kernel is used to train the super-resolution model, the super-resolution model learns from a very blurred state to a very clear state. When the real picture (that is, the first image to be super-resolved) is not so blurred, the image after super-resolution is easy to be too bright, resulting in false interference. After the flatness of the values ​​in the degenerate core is reduced by the solution of the present disclosure, the risk of falsely adding interference can be reduced.

[0058] For example, the degradation kernel extracted by KernelGAN is generally flat, which causes the subsequent trained super-resolution model to excessively improve the corresponding clarity, so that the white spots in the original capsule gastroscopy image are overly highlighted.

[0059] In addition, the present invention does not specifically limit the first interval and the second interval, as well as the increment rate of the first interval and the increment rate of the second interval. The basic idea is that when the first value is small, the increment rate of the second value is slow; when the first value is large, the increment rate of the second value is fast. Those skilled in the art can make adjustments through experiments or experience. For example, the first interval and the second interval, as well as the increment rate of the first interval and the increment rate of the second interval can be hyperparameters, which can be determined through experimental traversal.

[0060] In some examples, a numerical value related to the first value may be used as an input of the mapping relationship. That is, the numerical value related to the first value may be processed based on the mapping relationship. In addition, the numerical value related to the first value may be the first value itself, or may be the first value after preprocessing. The specific implementation method of the mapping relationship is related.

[0061] In some examples, the mapping relationship may be based on function transformation. That is, the first degenerate kernel may be corrected by applying function transformation. Some examples are provided below.

[0062] Figure 4 is a schematic diagram showing a curve corresponding to a monotonically increasing function involved in an example of the present disclosure.

[0063] In some examples, the mapping relationship may be based on a monotonically increasing function. In this case, the smoothness of converting the first value to the second value can be adjusted, and it is convenient to finely adjust the increment rate and to control the interval of the output value. It should be noted that the present disclosure does not limit the specific form of the monotonically increasing function. For example, those skilled in the art may, inspired by the present disclosure, conduct experiments to determine the correlation coefficient of the monotonically increasing function.

[0064] In some examples, the second degenerate kernel may be obtained based on the output value of the monotonically increasing function. The increasing rate of the monotonically increasing function in the first interval may be less than the increasing rate in the second interval. That is, when the first value is small, the increasing rate of the monotonically increasing function is slow; when the first value is large, the increasing rate of the monotonically increasing function is fast. In some examples, a numerical value related to the first value may be used as an input of the monotonically increasing function.

[0065] In some examples, the mapping relationship based on the monotonically increasing function may satisfy the first formula:

[0066] t 1 (m,n)=f(v(m,n)),

[0067] Wherein, f represents a monotonically increasing function, v(m, n) represents a value related to the first value at the position (m, n) of the first degenerate kernel, and t 1 (m, n) represents the output value of the monotonically increasing function.

[0068] As an example, Figure 4 is a schematic diagram showing a curve corresponding to a monotonically increasing function. The x-coordinate represents a value related to the first value, which is the first value after conversion to [0, 1], and the y-coordinate represents the output value of the monotonically increasing function. Figure 4 It can be seen that when the first value is small, the increasing rate is slow; when the first value is large, the increasing rate is fast.

[0069] In some examples, for a mapping relationship based on a monotonically increasing function, the numerical value associated with the first value may be the first value after preprocessing. In some examples, the preprocessing may be to convert the first value to a first preset interval. That is, before the first degenerate core is processed based on the mapping relationship of the monotonically increasing function, the first value may be converted to a first preset interval, and the converted first value may be input into the monotonically increasing function. In this case, the monotonically increasing function can be made fairer when processing different first values. In addition, the fixed first preset interval can also facilitate determination of the variation interval of the first value to better analyze the effect of the increasing rate of different intervals on the flatness of the degenerate core (for example, to facilitate consideration of the setting of the increasing rate based on the maximum value and the minimum value). In addition, it can also help control the interval of the output value of the monotonically increasing function to be consistent with the first preset interval, thereby reducing the risk of the second degenerate core deviating too much from the first degenerate core.

[0070] In some examples, the first preset interval may be [0, 1]. In some examples, during the conversion process, each first value may be divided by the maximum value of the first value in the first degenerate core to convert the first value to [0, 1]. In some examples, the conversion process may satisfy the formula:

[0071]

[0072] Among them, u(m, n) represents the first value after transformation when the position of the first degenerate core is (m, n) (that is, u(m, n) may be a case of v(m, n)), x(m, n) represents the first value when the position of the first degenerate core is (m, n), M and N represent the size of the first degenerate core, and max represents the maximum value function.

[0073] In some examples, the output value of the monotonically increasing function may not be greater than the input value (i.e., a value related to the first value). In this case, it is convenient to control the consistency between the interval of the output value of the monotonically increasing function and the interval of the input value. For example, for the first preset interval of [0, 1], when the input value is less than 1, the output value may be less than the input value, and when the input value is equal to 1, the output value may be 1 (refer to Figure 4 ).

[0074] In some examples, the mapping relationship may also be based on a piecewise function. In some examples, a first value greater than a preset value may be set to a larger increment rate through a piecewise function to obtain a second value. In some examples, a second degenerate core may be obtained based on an output value of the piecewise function. In some examples, a value related to the first value may be used as an input to the piecewise function. In some examples, for a mapping relationship based on a piecewise function, a value related to the first value may be the first value itself.

[0075] In some examples, the mapping relationship based on the piecewise function may satisfy the second formula:

[0076]

[0077] Among them, t 2 (m, n) represents the output value of the piecewise function, f 1 and f 2 represents a monotonically increasing function, f 2 The increasing rate can be less than f 1 The increasing rate of , v(m, n) is a value related to the first value at the position (m, n) of the first degenerate core, and K represents a preset value. For the second formula, preferably, v(m, n) can be the first value itself.

[0078] In addition, the present invention does not specifically limit the preset value. Those skilled in the art can adjust it through experiments or experience. Taking the processing of the degraded kernel of the capsule gastroscopy image extracted by KernelGAN as an example, the preset value can be 0.03.

[0079] In some examples, the first value may be processed based on the mapping relationship to output an intermediate value, and the intermediate value may be subjected to target processing to obtain a second value. The target processing may be configured to make the second value in the second degenerate core conform to the characteristics of the degenerate core. That is, the target processing may be configured to maintain the sum of all values ​​of the degenerate core before and after correction.

[0080] In some examples, the characteristic of the degenerate core may be that the sum of all values ​​is required to be 1. In some examples, the target processing may be to divide the intermediate value by the sum of all intermediate values ​​to obtain the second value. That is, the first value processed by the mapping relationship may be further processed to obtain the second value, so that the sum of all second values ​​is 1. In this case, the sum of all second values ​​can be 1, which is suitable for the degenerate core characterized by requiring the sum of all values ​​to be 1.

[0081] In addition, the intermediate value may be a numerical value directly obtained through the mapping relationship (ie, an output value of the mapping relationship). For example, the intermediate value may be an output value of the first formula or the second formula.

[0082] In some examples, the second value processed based on the target may satisfy the third formula:

[0083]

[0084] Wherein, y(m, n) represents the second value at the position of the second degenerate core (m, n), t(m, n) represents the middle value at the position of the first degenerate core (m, n), and M and N represent the size of the first degenerate core.

[0085] In some examples, for a mapping relationship based on a monotonically increasing function (eg, the first formula), the first value can be converted to a first preset interval and then input into the monotonically increasing function, and the output value of the monotonically increasing function can be input into a third formula to obtain a second value.

[0086] In some examples, for a mapping relationship based on a piecewise function (eg, the second formula), a first value may be input into the piecewise function, and an output value of the piecewise function may be input into a third formula to obtain a second value.

[0087] Figure 5A is a three-dimensional schematic diagram showing the degenerate core before correction (ie, the first degenerate core). Figure 5B is a three-dimensional schematic diagram showing the corrected degenerate kernel (ie, the second degenerate kernel). Fig. 6A is a two-dimensional schematic diagram showing the degenerate kernel before correction (ie, the first degenerate kernel). Figure 6B is a two-dimensional schematic diagram showing the corrected degenerate kernel (ie, the second degenerate kernel). Fig. 7A is the result when the first degenerate kernel is not corrected. Figure 3A Schematic diagram of a super-resolution image corresponding to region 801. Figure 7B is shown after correction of the first degenerate kernel Figure 3A Schematic diagram of a super-resolution image corresponding to region 801.

[0088] In addition, the present disclosure also provides some comparison examples before and after correction. Figure 5A , Figure 5B, Fig. 6A and Figure 6B ,exist Figure 5A In , the X coordinate and the Y coordinate represent the position of the first value, and the Z coordinate represents the size of the first value. Figure 5B In , the X coordinate and the Y coordinate represent the position of the second value, and the Z coordinate represents the size of the second value. Fig. 6A In , the first value of each position in the first degenerate kernel and the color representation of the first value are shown. Figure 6B , the second value at each position in the second degenerate kernel and the color representation of the second value are shown.

[0089] From the above figures, we can see that the degenerate kernel before correction is relatively flat and has a certain regularity. The changes of some larger values ​​are not obvious (that is, they are relatively close). In other words, from a three-dimensional perspective, it is relatively flat. For example, Fig. 6A The values ​​in the middle position were originally around 0.07, but after correction, the distribution of values ​​in the middle position was different.

[0090] In addition, refer to Fig. 7A and Figure 7B It can be seen that after correcting the degradation kernel, the problem of bright white spots in the super-resolution image is improved (see area 811).

[0091] In some examples, reference Figure 2 In step S103, the second degenerate kernel may be convolved with the second image to obtain a third image. In some examples, noise may be added after the convolution to obtain the third image.

[0092] In some examples, the second image may be a medical image having a higher resolution than the first image. Specifically, the second image may be a medical image of a second resolution, and the second resolution may be greater than the first resolution. In this case, it is convenient to apply the degradation kernel obtained in the low-resolution image to the high-resolution image to obtain a simulated low-resolution image. When the second degradation kernel is closer to the real degradation kernel, the degradation process of the high-resolution image can be made closer to the degradation caused by the application scenario (that is, the simulated low-resolution image can be made more accurate).

[0093] In some examples, the application scenarios of the second image and the first image may be different. In some examples, the clarity of the acquisition device of the second image and the first image may be different. In this case, the degradation kernel extracted from the first image is applied to the second image to train the super-resolution model, so that the super-resolution model trained with a certain amount of second images can super-resolve the image in the scenario of the first image to achieve super-resolution of unpaired images.

[0094] In some examples, for the first image being a capsule gastroscope image, the second image may be a gastroscope image other than the capsule gastroscope image with a higher resolution than the first image. Thus, the accuracy of the degradation kernel of the capsule gastroscope image can be improved, and the super-resolution of the unpaired capsule gastroscope image can be achieved, and the clarity of the super-resolution image corresponding to the capsule gastroscope image can be improved. For example, the second image may be an electronic gastroscope image. Specifically, the electronic gastroscope can provide high-definition images, but the inspection process may cause a certain discomfort to the patient, while the image clarity of the capsule endoscope 910 is limited, but it has the characteristics of painless, non-invasive, no cross infection, and the inspection process is comfortable. In this case, the image pair obtained by applying the degradation kernel extracted from the capsule gastroscope image to the electronic gastroscope can be used to train the super-resolution model. The trained super-resolution model can be used to perform super-resolution on the capsule gastroscope image. Thus, the clarity of the internal imaging of the stomach can be improved while improving the patient's experience.

[0095] In some examples, the second image may be unpaired with the first image. That is, the second image may be a high-resolution image that does not correspond to the first image (i.e., the low-resolution image). In most scenarios, it is difficult to meet the condition that paired images exist during acquisition. The present disclosure can support super-resolution of low-resolution images in unpaired scenarios by correcting the first degenerate kernel.

[0096] In some examples, the resolution of the third image may be less than the resolution of the second image. That is, applying the second degradation kernel to the high-resolution image may result in a simulated low-resolution image.

[0097] Figure 8 is an exemplary flow chart showing a generation method involved in the examples of the present disclosure.

[0098] Examples of the present disclosure also relate to a method for generating a training data set (which may also be referred to as a generation method, a data set construction method, etc.). In some examples, reference Figure 8 , the generation method may include:

[0099] Step S201: Acquire multiple second images.

[0100] Step S202: Acquire a third image through each of the plurality of second images.

[0101] In some examples, the processing method involved in the examples of the present disclosure may be executed to obtain a third image through each of the plurality of second images. In this case, the third image may be closer to a real low-resolution image, thereby improving the quality of the training data set.

[0102] Step S203: taking each of the plurality of second images and the third image corresponding to each of the plurality of second images as a pair of paired images to obtain a plurality of pairs of paired images.

[0103] For example, a low-resolution electronic gastroscope image can be obtained by applying the second degenerate kernel obtained from the capsule gastroscope image to the electronic gastroscope image. The original electronic gastroscope image and the low-resolution electronic gastroscope image can be used as a pair of paired images.

[0104] Step S204: using multiple pairs of paired images as training data sets.

[0105] The examples of the present disclosure also relate to a super-resolution method for medical images (which may also be referred to as a super-resolution method, or a super-resolution reconstruction method, etc.). In some examples, the super-resolution method may include inputting a first image to be super-resolved into a super-resolution model trained by a training data set to obtain a super-resolution image. The training data set may be generated by the generation method involved in the examples of the present disclosure. In this case, the training data set is generated using the corrected degenerate kernel, and the super-resolution model is trained using the training data set, which can improve the problem of false interference while improving the clarity of the super-resolution image (see Fig. 7A and Figure 7B ).

[0106] In some examples, the super-resolution model may be a deep learning-based model. Thus, it is possible to automatically identify features related to super-resolution in the training data set. In some examples, the super-resolution model may be RealESRGAN, CinCGAN, or RBSR (Efficient and Flexible Recurrent Network for Burst Super-Resolution).

[0107] In addition, when the super-resolution model is RealESRGAN, the calculation speed can be improved, and it can be applied to scenarios with high real-time requirements. For example, the first degradation kernel of the first image can be extracted and corrected based on KernelGAN (see the processing of the first degradation kernel above), and the corrected degradation kernel is used to make paired images to generate a training data set, and then the training data set is used to train RealESRGAN.

[0108] The example of the present disclosure also discloses an electronic device, including a processor and a memory. The processor executes a program stored in the memory to implement one or more steps in the above-mentioned processing method, generation method or super-resolution method.

[0109] The examples of the present disclosure also disclose a computer-readable storage medium, which can store at least one instruction, and when the at least one instruction is executed by a processor, one or more steps in the processing method, the generating method or the super-resolution method are implemented. Wherein, the computer-readable storage medium can include but is not limited to any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0110] Although the present disclosure is specifically described above in conjunction with the accompanying drawings and examples, it is to be understood that the above description does not limit the present disclosure in any form. Those skilled in the art may modify and change the present disclosure as needed without departing from the essential spirit and scope of the present disclosure, and these modifications and changes all fall within the scope of the present disclosure.

Claims

1. A medical image processing method, characterized in that: include: extracting a first degradation kernel of a first image, where the first image is a medical image of a first resolution; The first degradation kernel is processed based on a mapping relationship to obtain a second degradation kernel, wherein the mapping relationship is used to reflect the relationship between a first value of the first degradation kernel and a second value of the second degradation kernel, wherein the second value is positively correlated with the first value, an interval in which the first value is located includes a first interval and a second interval greater than the first interval, and an increasing rate of the second value in the first interval is less than an increasing rate in the second interval; and the second degradation kernel is applied to a second image to obtain a third image, wherein the second image is a medical image of a second resolution, and the second resolution is greater than the first resolution.

2. The medical image processing method according to claim 1, characterized in that: The first degenerate kernel is obtained by using KernelGAN.

3. The medical image processing method according to claim 1, characterized in that: The mapping relationship is based on a monotonically increasing function; the second degenerate kernel is obtained based on an output value of the monotonically increasing function, and the increasing rate of the monotonically increasing function in the first interval is less than the increasing rate in the second interval, wherein the mapping relationship satisfies a first formula: t1(m,n)=f(v(m,n)), wherein f represents the monotonically increasing function, v(m,n) represents a numerical value related to the first value at the position (m,n) of the first degenerate kernel, and t1(m,n) represents the output value of the monotonically increasing function.

4. The medical image processing method according to claim 3, characterized in that: The numerical value related to the first value is the first value after preprocessing, and the preprocessing is to convert the first value into a first preset interval.

5. The medical image processing method according to claim 4, characterized in that: The conversion process satisfies the formula: Among them, u(m, n) represents the first value after transformation when the position of the first degenerate core is (m, n), x(m, n) represents the first value when the position of the first degenerate core is (m, n), M and N represent the size of the first degenerate core, and max represents the maximum value function.

6. The medical image processing method according to any one of claims 1 to 5, characterized in that: The first value is processed based on the mapping relationship to output an intermediate value, and the intermediate value is divided by the sum of all the intermediate values ​​to obtain the second value.

7. The medical image processing method according to any one of claims 1 to 5, characterized in that: The first image is a capsule gastroscopy image, and the second image is a gastroscopy image other than the capsule gastroscopy image.

8. A method for generating a training data set, characterized in that: include: acquiring a plurality of second images; Execute the medical image processing method described in any one of claims 1 to 7 to obtain a third image through each of the multiple second images; use each of the multiple second images and the third image corresponding to each of the multiple second images as a pair of paired images to obtain multiple pairs of the paired images; and use the multiple pairs of the paired images as training data sets.

9. A super-resolution method for medical images, characterized in that: include: The first image to be super-resolved is input into a super-resolution model trained by a training data set generated by the method for generating a training data set according to claim 8 to obtain a super-resolution image.

10. An electronic device, characterized in that: The electronic device includes a processor and a memory, and the processor executes a program stored in the memory to implement the medical image processing method described in any one of claims 1 to 7, the method for generating a training data set described in claim 8, or the super-resolution method of medical images described in claim 9.

Images