Image processing methods, dataset acquisition methods, and image processing devices

Abstract

A method for image processing and a method for acquiring a dataset are disclosed. The image processing method includes: applying a high-resolution (HD) image to a calibration blur kernel to obtain a blurred image; injecting calibration noise to form a first noisy image; degrading the first noisy image to a low-resolution image using a downsampling method; performing lossy compression on the low-resolution image to obtain a compressed image; adding random noise simulating the transmission process to the compressed image to form a second noisy image; and performing lossy compression on the second noisy image to obtain a low-resolution image. The HD image is an image formed by taking pictures of internal organs using a capsule endoscope, and the calibration blur kernel and calibration noise are the results obtained by fitting the HD image acquisition process. By incorporating the blur caused by camera movement during capsule endoscopy and the noise effect during image formation, the above method makes the image processing closer to the actual degradation process, obtaining an accurate correspondence between HD and low-resolution images to establish an accurate dataset.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image processing method, a dataset acquisition method, and an image processing apparatus. Background Technology

[0002] With the development of technology, people have an increasing demand for high-definition, high-resolution images, especially in the field of medical imaging. High-definition medical images can greatly facilitate attending physicians in analyzing patients' conditions and making diagnoses. Currently, the method of using magnetically controlled capsule endoscopy for gastrointestinal examinations is widely used. The capsule endoscope contains a magnet, and its interaction with an external magnet controls the movement of the endoscope inside the body to capture images of the stomach and intestinal walls. Doctors can analyze the health of the patient's stomach and intestinal walls based on these images. However, due to the hardware limitations of image transmission tools, the captured images often have low resolution, and the details and textures of the stomach and intestinal walls are blurred, which greatly hinders doctors' analysis and diagnosis.

[0003] Therefore, techniques such as image super-resolution (which recovers a high-resolution image from a low-resolution image or image sequence) are needed to enhance the resolution of these images and restore their details and textures. Existing image super-resolution techniques are divided into traditional methods and deep learning-based methods. Traditional methods have a longer history of development and generally use spatial spline interpolation (bilinear) to enhance the resolution of the input image. However, the resulting high-resolution images often have significant blurriness, amplified noise, and low overall image quality. Deep learning-based methods rely heavily on the quality of the training data. If the training data is poorly designed, the trained model often fails to achieve good results. Typically, deep learning methods create datasets by first acquiring high-resolution images, then downsampling the high-resolution images to obtain corresponding low-resolution images, using these low-resolution-high-resolution image pairs as the training dataset. This results in a relatively simple dataset that cannot fully represent the image degradation process. Models trained using such datasets often perform poorly in practical applications. Therefore, models built using datasets obtained through current deep learning methods often produce high-resolution images that are still not very clear, resulting in poor image restoration and affecting image analysis and judgment. Summary of the Invention

[0004] In view of the above problems, the purpose of this invention is to provide an image processing method, a dataset acquisition method, and an image processing device that obtain low-resolution low-definition images by simulating the degradation process of high-definition images captured by a capsule endoscope inside the body, thereby acquiring high-quality high-definition-low-definition image data pairs to solve the problems in the prior art.

[0005] According to a first aspect of the present invention, an image processing method is provided, comprising:

[0006] A blurred image is obtained by applying the acquired high-resolution high-definition image to a calibrated blur kernel;

[0007] A calibration noise is injected into the blurred image to form a first noisy image;

[0008] The first noisy image is degraded into a low-resolution image using a downsampling method; and

[0009] The low-resolution image is subjected to lossy compression to reduce storage space, resulting in a compressed image.

[0010] Random noise during the simulated transmission process is added to the compressed image to form a second noise image;

[0011] The second noisy image is lossily compressed and then saved to obtain a low-resolution, low-definition image.

[0012] The high-definition image is an image of an internal organ captured using a capsule endoscope, and the calibration blur kernel and calibration noise are the results obtained by fitting the capture process of the high-definition image.

[0013] Optionally, the calibration blur kernel includes a focus blur kernel and a motion blur kernel. The focus blur kernel characterizes the image blur caused by distortion and focus during the capture of the high-definition image by the camera of the capsule endoscope, and the motion blur kernel characterizes the image blur caused by the movement of the camera inside the body.

[0014] Optionally, the blurred image is the result obtained by simultaneously operating the high-definition image with the focus blur kernel and the motion blur kernel.

[0015] Optionally, the calibration step of the focus blur kernel includes:

[0016] The camera is used to photograph the tilted line of the standard color chart, and the pixel value change curves on both sides of the tilted line are obtained as pulse signals.

[0017] Calculate the edge function corresponding to the pulse signal and the line propagation function obtained by differentiating the edge function;

[0018] By rotating the standard color chart at fixed intervals, multiple line propagation functions corresponding to the multiple inclined lines are obtained;

[0019] The multiple line propagation functions are rotated once and then synthesized into a point propagation function in three-dimensional space; and

[0020] The focus blur kernel is obtained by normalizing the point propagation function.

[0021] Optionally, a Gaussian blur kernel is used to simulate the motion blur kernel, wherein the Gaussian blur kernel includes an isotropic blur kernel and an anisotropic blur kernel.

[0022] Optionally, the calibration noise includes dark current noise and Gaussian noise. The dark current noise characterizes the noise generated during signal conversion of the image captured by the camera of the capsule endoscope, and the Gaussian noise characterizes the noise generated during the formation of the RGB image.

[0023] Optionally, the calibration steps for the dark current noise include:

[0024] Set the initial image data, place it in a black cloth environment with different camera gains, and after lighting for a fixed time, calculate the average pixel value, horizontal pixel value, and vertical pixel value of the entire image in the three RGB channels of the initial image respectively.

[0025] Obtain the pixel mean and variance of the full image for each channel under the different camera gains;

[0026] Obtain the full image pixel values ​​of the initial image under a fixed camera gain, and create a histogram of the image pixel values ​​accordingly.

[0027] Based on the pixel mean and variance, histograms were generated for various functional distributions.

[0028] By comparing the fitting degree between the histograms under various different function distributions and the histograms of the image pixel values, the function distribution that best matches is selected as the distribution of the dark current noise.

[0029] Optionally, the various function distributions include Gaussian distribution, Poisson distribution, and gamma distribution, with the best-fit function distribution being the gamma distribution.

[0030] Optionally, the calibration steps for the Gaussian noise include:

[0031] Set the initial image data and multiple different camera gains;

[0032] After placing the initial image in a darkroom environment with multiple camera gains and lighting for a fixed time, the pixel mean and variance of the entire image in the three RGB channels are calculated respectively.

[0033] Create a graph of pixel mean versus variance based on the relationship between the pixel mean and the variance;

[0034] Obtain the pixel mean of the image captured by the camera, and find the variance of the image based on the curve; and

[0035] Gaussian noise with zero pixel mean is generated based on the obtained variance, and the Gaussian noise conforms to a Gaussian distribution.

[0036] Optionally, the step of injecting calibration noise into the blurred image to form a first noisy image includes:

[0037] Obtain the function distributions corresponding to the dark current noise and the Gaussian noise, respectively;

[0038] The dark current noise and the Gaussian noise are injected into the blurred image according to their corresponding function distributions to form a first noise image.

[0039] Optionally, a bicubic interpolation downsampling method is used to perform a 2x downsampling operation on the first noisy image to obtain the low-resolution image.

[0040] Optionally, the lossy compression coefficients used in compressing the low-resolution image into a compressed image and in compressing the second noisy image into a low-resolution image are 70 and 90, respectively.

[0041] Optionally, the step of lossy compression of the image includes:

[0042] The image is converted from RGB data to YUV data, and 4:2:0 chromaticity sampling is performed simultaneously.

[0043] The image in the YUV data format is divided into 8x8 cells, and a discrete cosine transform is performed on each cell;

[0044] After performing the discrete cosine transform, the image is quantized, and high-frequency data is discarded.

[0045] Entropy encoding is performed on the matrix corresponding to the quantized image to form a compressed image.

[0046] According to a second aspect of the present invention, a data set acquisition method is provided, comprising:

[0047] Perform the image processing method described above; and

[0048] The high-resolution image and its corresponding low-resolution image are saved as a pair of image data;

[0049] A training dataset is created by acquiring multiple pairs of the aforementioned image data.

[0050] According to a third aspect of the present invention, an image processing apparatus is provided for implementing the image processing method described above, the image processing apparatus comprising:

[0051] The image blurring unit combines the acquired high-resolution image with a calibrated blur kernel to obtain a blurred image.

[0052] A first noise injection unit injects calibration noise into the blurred image to form a first noise image;

[0053] The downsampling unit degrades the first noisy image into a low-resolution image using a downsampling method; and

[0054] The first compression unit performs lossy compression on the low-resolution image to reduce storage space and obtain a compressed image.

[0055] The second noise injection unit simulates random noise during transmission and adds random noise points to the compressed image to form a second noise image;

[0056] The second compression unit performs lossy compression on the second noisy image and saves it to obtain a low-resolution, low-definition image.

[0057] The high-definition image is an image of an internal organ captured using a capsule endoscope, and the calibration blur kernel and calibration noise are the results obtained by fitting the capture process of the high-definition image.

[0058] Optionally, the image blurring unit is further configured to obtain the calibration blur kernel, which includes a focus blur kernel and a motion blur kernel. The focus blur kernel characterizes the image blur caused by distortion and focus during the capture of the high-definition image by the camera of the capsule endoscope, and the motion blur kernel characterizes the image blur caused by the movement of the camera inside the body.

[0059] The first noise injection unit is also used to acquire the calibration noise, which includes dark current noise and Gaussian noise. The dark current noise characterizes the noise generated when the image captured by the camera of the capsule endoscope is converted into a digital signal, and the Gaussian noise characterizes the noise generated when the image is formed into an RGB image.

[0060] The image processing method, dataset acquisition method, and image processing device provided by this invention simulate the degradation process of high-definition images captured by a capsule endoscope inside the body. During image processing, blurring caused by the movement of the capsule within the body and noise introduced during processing are incorporated. Downsampling and lossy compression methods are then used to repeatedly reduce the resolution of the high-definition image, resulting in a low-resolution compressed image. Noise from the transmission process is then added to the compressed image, and lossy compression is applied during transmission to obtain a more accurate degraded image (low-resolution image) corresponding to the high-definition image. This process reasonably describes the true degradation process of the image. The resulting high-definition and low-resolution image pairs effectively describe the degradation relationship between the images. The dataset established based on this data better reflects the influence of various factors on the image during degradation, resulting in a clearer and more realistic high-resolution image recovery, improving the accuracy and precision of high-definition image recovery. Furthermore, the training dataset created by the dataset acquisition method of this invention can be applied to the establishment of sample sets for model building in various image processing processes, such as image super-resolution and image denoising. Attached Figure Description

[0061] The above and other objects, features and advantages of the present invention will become more apparent from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0062] Figure 1a This diagram illustrates the process of degrading a high-definition image to a low-definition image according to the present invention.

[0063] Figure 1b A flowchart of an image processing method according to an embodiment of the present invention is shown;

[0064] Figure 2a and Figure 2b A schematic diagram and a flowchart of the signal change process for calibrating the blur kernel according to an embodiment of the present invention are shown respectively;

[0065] Figure 3 A schematic diagram illustrating the process of forming images captured by a capsule endoscope according to an embodiment of the present invention is shown;

[0066] Figure 4 A flowchart illustrating the calibration process for dark current noise according to an embodiment of the present invention is shown;

[0067] Figures 5a-5c Histograms of pixel distributions conforming to Gaussian, Poisson, and gamma distributions according to embodiments of the present invention are shown respectively.

[0068] Figure 6 A flowchart illustrating the calibration process for Gaussian noise according to an embodiment of the present invention is shown;

[0069] Figure 7 A simplified schematic diagram of an image processing apparatus according to an embodiment of the present invention is shown;

[0070] Figure 8 A flowchart of a dataset acquisition method according to an embodiment of the present invention is shown. Detailed Implementation

[0071] Various embodiments of the invention will now be described in more detail with reference to the accompanying drawings. In the various drawings, the same elements are indicated by the same or similar reference numerals. For clarity, the various parts in the drawings are not drawn to scale.

[0072] Deep learning-based methods, by employing a nonlinear nested multilayer perceptron approach to fit the mapping relationship between low-resolution and high-resolution image spaces, can effectively reflect the relationship between low-resolution images and real images. Therefore, this invention also employs deep learning methods for image processing to establish a corresponding dataset, thereby building a model applicable to various image processing fields, such as image super-resolution and noise reduction. Furthermore, this invention provides a dataset acquisition method that enables the trained model to successfully super-resolution medical images such as colonoscopy and gastroscopy. Establishing a dataset first requires establishing the relationship between high-resolution and low-resolution images. This invention uses an image processing method to establish high-resolution-low-resolution image data pairs. Based on the description in the background section, factors affecting image degradation include downsampling, blurring, noise, and compression loss during image storage. This invention primarily analyzes the degradation process of colonoscopy and gastroscopy images acquired by capsule endoscopy to achieve dataset acquisition.

[0073] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0074] Figure 1a This diagram illustrates the process of degrading a high-definition image to a low-definition image according to the present invention. Figure 1b A flowchart of an image processing method according to an embodiment of the present invention is shown.

[0075] like Figure 1aThe diagram illustrates the process of degrading high-definition images to low-definition images according to the present invention. High-definition images are formed by using a capsule endoscope, for example, moving within the human body to capture images of organs (such as the stomach and intestines). The capsule endoscope is controlled to move within the body via magnetic induction, for example, dragging it across the surface of a target area to capture images. However, due to friction between the capsule endoscope and the wall of the target area (e.g., the stomach wall), the lens or camera of the capsule endoscope twists and moves during the capturing process. Affected by factors such as distortion, defocusing, and motion, image blurring is inevitable. Furthermore, noise is introduced when the image captured by the camera is processed by the digital sensor. Even more so, to reduce transmission costs, images are often compressed or downsampled before transmission. All of these factors contribute to the degradation of high-definition images, and this embodiment fully considers these factors.

[0076] according to Figure 1a First, the high-definition image is blurred and interacted with a blur kernel to obtain a blurred image; then noise is injected; next, downsampling is performed to reduce the image resolution; then JPEG compression (lossy compression) is used to reduce the image storage space for easier image transmission; subsequently, random noise during transmission is simulated by adding random noise points that follow a specific distribution to the image; when saving the image, it is compressed again using JPEG to reduce the image storage space, and finally a low-definition image is obtained. The above process is expressed in the following formula (1):

[0077] y = (x ↓bic *k+n) jpeg Equation (1)

[0078] Where x represents the input high-resolution image, y represents the degraded low-resolution image, ↓bic represents bicubic downsampling, k represents the blur kernel, n represents noise, and jpeg represents JPEG compression (lossy compression). As can be seen from equation (1), blur, noise, downsampling, and compression all affect image degradation. Therefore, to simulate the degradation process of real colonoscopy and gastroscopy images, an image processing method matching this process is provided to obtain low-resolution-high-resolution data pairs as samples, and a dataset acquisition method is also provided. For details of the image processing method, please refer to [link to relevant documentation]. Figure 1b .

[0079] like Figure 1b As shown, the image processing method of this invention mainly includes the following steps:

[0080] In step S101, the acquired high-resolution high-definition image is processed with a calibration blur kernel to obtain a blurred image.

[0081] In this step, the acquired high-resolution image is first processed with the calibrated blur kernel to obtain a blurred image. Figure 1a Correspondingly, according to the above description, during the formation of high-definition images captured by a capsule endoscope, the source of image blurring mainly comes from two places: lens blur caused by lens (camera) focus flicker and motion blur caused by scene movement. Therefore, the calibrated blur kernel includes the focus flicker blur kernel and the motion blur kernel. The focus flicker blur kernel characterizes the image blur caused by distortion and focus flicker during the capture of high-definition images by the capsule endoscope's camera, while the motion blur kernel characterizes the image blur caused by the camera's movement within the body.

[0082] Furthermore, the blurred image is the result of simultaneously operating the high-resolution image with both a focus blur kernel and a motion blur kernel. The interaction between the high-resolution image and the blur kernel is, for example, convolution. The blur kernel is considered a matrix; through convolution, each pixel in the image becomes equivalent to the average of the pixels of its surrounding pixels, making each pixel in the image softer relative to its surroundings, thus achieving blurring of the high-resolution image, reducing the image resolution, and obtaining the blurred image. The calibration steps for the blur kernel are discussed in [the following section]. Figures 2a-2b This will be introduced in the middle.

[0083] In step S102, calibration noise is injected into the blurred image to form a first noise image.

[0084] In this step, the corresponding Figure 1a The blurred image obtained in step S101 is then subjected to noise injection. This noise is calibration noise, which mainly includes dark current noise and Gaussian noise. Dark current noise characterizes the noise generated during signal conversion of the image captured by the capsule endoscope camera, while Gaussian noise characterizes the noise generated during the formation of the RGB image. This process primarily simulates the noise impact on the image captured by the camera during image conversion.

[0085] This step specifically includes: obtaining the function distributions corresponding to dark current noise and Gaussian noise respectively; injecting the dark current noise and Gaussian noise into the blurred image according to their corresponding function distributions to form a first noise image. By injecting noise into the blurred image, the pixel value distribution of each point in the image conforms to the function distribution corresponding to each noise, thereby reducing the pixel value of each point in the image and reducing the resolution. The noise calibration step is described in... Figures 4-6 This will be introduced in the middle.

[0086] In step S103, the first noisy image is degraded into a low-resolution image using a downsampling method.

[0087] In this step, a bicubic interpolation downsampling method is used to perform a 2x downsampling operation on the first noisy image to obtain a low-resolution image. Specifically, during image transmission, to reduce transmission power consumption, the input image is downsampled by 2x, reducing the image size to half its original size. The bicubic interpolation downsampling method utilizes the 16 pixels surrounding the point to be sampled. This interpolation sampling method considers not only the influence of the four surrounding adjacent points but also the influence of the grayscale value change rate between adjacent points, resulting in a reduction effect that more closely approximates the low-resolution image.

[0088] In step S104, lossy compression is performed on the low-resolution image to obtain a compressed image.

[0089] In this step, to ensure transmission efficiency, the low-resolution image is first subjected to lossy compression (i.e., Figure 1a JPEG compression is used to reduce image storage space, resulting in compressed images, reducing transmission power consumption, and facilitating subsequent image transmission.

[0090] In step S105, random noise during transmission is simulated by adding random noise points to the compressed image to form a second noise image.

[0091] In this step, the main focus is on simulating the noise impact on the image during transmission. Noise is added to the image to reduce the pixel values ​​at each point, forming a second noisy image. In this embodiment, the distribution of noise can also conform to the distribution of a specific function.

[0092] In step S106, the second noisy image is compressed using JPEG and then saved to obtain a low-resolution, low-definition image.

[0093] In this step, lossy compression is applied to the image to reduce storage space during image saving. This embodiment primarily uses the `cv2.encode` and `cv2.decode` functions from the OpenCV library to perform two JPEG compression operations. The lossy compression coefficients used in compressing the low-resolution image into a compressed image (step S104) and compressing the second noisy image into a low-resolution image (step S106) are different, at 70 and 90 respectively. The lossy compression coefficient of 70 is chosen during transmission because experiments have shown that the minimum compression coefficient without significant loss of image quality is 70 (a smaller compression coefficient indicates a higher degree of compression). In the image saving stage, a lossy compression coefficient of 90 is chosen because experiments have shown that a compression coefficient of 90 results in visually lossless images while also reducing storage space.

[0094] Specifically, the steps for lossy compression (JPEG compression) of an image include:

[0095] First, the image is converted from RGB data to YUV data, and 4:2:0 chromaticity sampling is performed simultaneously. Then, the YUV data image is divided into 8x8 cells, and Discrete Cosine Transform (DCT) is performed on each cell. After performing DCT, the image is quantized, and high-frequency data is discarded. Finally, entropy encoding is performed on the matrix corresponding to the quantized image to form a compressed image.

[0096] The image processing method in this embodiment simulates the degradation process of high-definition images captured by a capsule endoscope inside the body. It incorporates blurring caused by the simulated movement of the capsule within the body and noise introduced during image processing. Then, it employs downsampling and lossy compression methods to repeatedly reduce the resolution of the high-definition image, resulting in a low-resolution compressed image. This yields an accurate degraded image (low-resolution compressed image) that reflects the actual image degradation process. The low-resolution image obtained using this method effectively and realistically reflects the various factors affecting the high-definition image during degradation, thus providing accurate high-definition-low-resolution image data pairs. The image data pairs obtained using this method better reflect the true degradation process of the image, resulting in a more accurate dataset. The recovered high-resolution image is clearer, with more distinct textures, and more closely resembles reality, facilitating image research and analysis in the medical field.

[0097] Figure 2a and Figure 2b A schematic diagram and a flowchart of the signal change process of the calibrating process of the blur nucleus according to an embodiment of the present invention are shown respectively.

[0098] Figure 2b The calibration process of the blur kernel in step S101 is shown. According to the theory of computational optics, the blur problem caused by lens flash can be described by the point propagation function (PSF), and then the blur kernel is generated by the PSF. Figure 2a The process of generating a point propagation function from an impulse function is shown.

[0099] like Figure 2a As shown, firstly, the pulse signal σ(x) is obtained, then the edge propagation function ESF(x) is calculated based on the pulse function, then the derivative of the edge propagation function is obtained to obtain the line propagation function LSF(x), and finally the line propagation function is rotated one full circle to synthesize the point propagation function PSF(x, y).

[0100] like Figure 2b As shown, the calibration process of the blur kernel in this embodiment mainly includes steps S1011-S1015. Specifically:

[0101] In step S1011, a camera is used to photograph the tilted line of the standard color chart, and the pixel value change curves on both sides of the tilted line are obtained as pulse signals.

[0102] In this step, the first requirement is to obtain the pulse signal. However, in practice, a true pulse signal σ(x) cannot be obtained during calibration. Therefore, the inclined lines of a standard color chart are used as the subject of the camera lens. A pixel value change curve is plotted on both sides of the inclined line. This curve approximates the pulse signal σ(x). The area to the left of the inclined line is set to be darker, and the area to the right to be brighter. The pixel values ​​change smoothly from left to right initially, then rise sharply from extremely low values ​​to extremely high values ​​near the inclined line, and then change smoothly again. This curve resembles the trend of a pulse signal. For example, a standard color chart can be disc-shaped, with multiple inclined lines dividing it into different color regions along the radius from the center. The colors on both sides of each inclined line are different, and the pixel value distribution in the image captured by the camera approximates a pulse signal.

[0103] In step S1012, the edge function corresponding to the pulse signal and the line propagation function obtained by differentiating the edge function are calculated.

[0104] In this step, the edge propagation function ESF(x) is calculated based on the pulse signal σ(x), and then the differential of the edge propagation function is taken to obtain the corresponding line propagation function LSF(x). The above process can be formally expressed as follows:

[0105] ESF(x)≈smooth(σ(x)) Equation (2)

[0106] LSF(x) = diff(ESF(x)) Equation (3)

[0107] In formula (2), ESF(x) represents the edge propagation function, and σ(x) represents the pulse signal. Formula (3) differentiates the edge propagation function to obtain a curve resembling a one-dimensional Gaussian distribution (see [reference]). Figure 2a The curve is called the line propagation function LSF(x). The signal values ​​on both sides of the curve are low and smooth and symmetrical. The middle region is extremely narrow with a high signal value. The signal value at the apex is the highest and the derivative value is zero.

[0108] In step S1013, the standard color card is rotated at fixed angles to obtain multiple line propagation functions corresponding to multiple inclined lines.

[0109] In this step, for example, the inclined lines of the standard color chart are rotated every 10 degrees, and the line propagation function of each inclined line is calculated according to the above steps, resulting in multiple line propagation functions.

[0110] In step S1014, multiple line propagation functions are rotated once and then synthesized into a point propagation function in three-dimensional space.

[0111] In this step, all these line propagation functions are rotated one full rotation, and then the rotated curves are aligned according to their maximum values ​​and synthesized into a point propagation function in three-dimensional space. The rotation is performed by rotating 360 degrees along the central axis of the line propagation function to obtain the point propagation function PSF(x, y) in three-dimensional space, as shown below. Figure 2a As shown in the rightmost figure, the point propagation function has the highest signal value at the vertex, and the signal value decreases as it extends outward from the neighborhood. This is very similar to a two-dimensional Gaussian blur kernel. Therefore, the focus blur kernel calibrated in this step can be regarded as an expression of a certain distorted two-dimensional Gaussian blur kernel in linear space.

[0112] In step S1015, the point propagation function is normalized to obtain the focus blur kernel.

[0113] In this step, the point propagation function is regarded as a two-dimensional Gaussian blur curve. Taking the vertex of the curve in the three-dimensional space as the center, a suitable range of length and width values ​​are selected, and finally normalization is performed to obtain the final focus blur kernel.

[0114] In this embodiment, through Figure 2b The steps described above calibrate the focus blur kernel, and the motion blur kernel also needs to be calibrated. During the shooting process, the camera of a capsule endoscope inevitably experiences camera shake, resulting in motion blur in the captured images. The Gaussian blur kernel is a universally applicable kernel for simulating motion blur, and it can fit motion blur scenes well in most cases. Therefore, this embodiment uses a Gaussian blur kernel to simulate the blurring process of the scene, using it as the motion blur kernel. For example, the blur kernel takes fixed values: [3, 5, 7, 9, 11, 13]. To expand the spatial range of the blur kernel, isotropic and anisotropic blur kernels are also used to represent the motion blur kernel, encompassing as many different blur scenes as possible. The formula for the Gaussian blur kernel is expressed as follows:

[0115]

[0116] Where σ represents variance, μ x μ y These represent the mean values ​​in the x and y directions, respectively.

[0117] This completes the calibration of the focus blur kernel and motion blur kernel. Based on the relevant algorithms of Gaussian blur, after obtaining the corresponding calibrated blur kernel, convolving it with the high-resolution image yields a blurred image, thus achieving blurring processing of the high-resolution image. For example, the high-resolution image can be first processed with the focus blur kernel, and then with the motion blur kernel.

[0118] Figure 3 A schematic diagram illustrating the formation process of images captured by a capsule endoscope according to an embodiment of the present invention is shown.

[0119] like Figure 3 The image shown illustrates the formation process of gastroscopy and colonoscopy images captured by a capsule endoscope. First, the capsule lens 301 acquires a light signal, which is transmitted to the electronic photosensitive instrument 302 to generate a corresponding current signal. Then, the analog front-end unit 303 converts the current signal into a digital signal. Afterward, the image passes through the compression unit 304 to obtain a RAW image. Finally, the image is processed by the de-mosaic unit 305 and the JPEG compression unit 306 to obtain an 8-bit RGB compressed image. Analysis reveals that the noise signal mainly originates from the dark current noise generated by the electronic photosensitive instrument 302 and the Gaussian noise generated by the de-mosaic unit 305 and the JPEG compression unit 306. Figure 3 This mainly simulates the noise generated during the image formation process captured by the camera. Only a few examples of the main types of noise are given here. In reality, there may be more than just these two types of noise. Figures 4-6 This section mainly introduces how to fit the noise signal generated in the above process.

[0120] Figure 4 A flowchart illustrating the dark current noise calibration process according to an embodiment of the present invention is shown. Figures 5a-5c The pixel distribution histograms conforming to Gaussian, Poisson, and gamma distributions according to embodiments of the present invention are shown respectively.

[0121] like Figure 4 As shown, the calibration process for dark current noise includes steps S1021-S1025.

[0122] In step S1021, the data of the initial image is set, and it is placed in a black cloth environment with different camera gains. After lighting for a fixed time, the pixel mean, horizontal pixel mean and vertical pixel mean of the entire image in the three RGB channels are calculated respectively.

[0123] In this step, to calibrate the dark current noise, the initial image output by the capsule endoscope is first set to 480x480 raw data and saved as a bmp file. Since different camera gain values ​​are accompanied by different noise intensities, the gain values ​​are set to 1, 2, 4, and 8 respectively, corresponding to saving 100, 102, 100, and 104 images. Then, the above images are placed in a dark environment and illuminated with LED lights for 0-20ms. After that, the pixel mean, horizontal pixel mean, and vertical pixel mean corresponding to the time mean of the entire image are calculated. Their calculation formulas are as follows: (5), (6), and (7) respectively.

[0124]

[0125]

[0126]

[0127] Where T represents the moment of the last illumination, M represents the length and width of the image, and I(x, y, t) represents the pixel value at coordinates (x, y) at the moment of the tth illumination. During calibration, the average pixel values ​​across the entire image, the horizontal average pixel value, and the vertical average pixel value were calculated for each of the three RGB channels of the initial image. Analysis revealed that the average dark current (pixel) value increases with increasing gain, and the average dark current value of the R channel is higher than that of the G channel, while the average dark current value of the G channel is higher than that of the B channel. The average dark current value of each channel shows good consistency, with weak striped textures existing in some areas.

[0128] In step S1022, the pixel mean and variance of the full image under each channel at different camera gains are obtained.

[0129] In this step, after obtaining the pixel mean of the entire image, the pixel variance of the entire image is calculated. Since the results are measured under different gain values, the relationship between the pixel mean and variance of the entire image under different channels and the gain value can be obtained. After calibration and testing, when the gain values ​​are {1, 2, 4, 8}, the pixel mean of the R channel ranges from 4.0 to 5.4, the pixel mean of the G channel ranges from 4.0 to 4.6, and the pixel mean of the B channel ranges from 4.0 to 4.7. The pixel variance (dark current variance) of the R channel ranges from 0.020 to 4.698, the dark current variance of the G channel ranges from 0.003 to 1.401, and the dark current variance of the B channel ranges from 0.009 to 2.487.

[0130] In step S1023, the full image pixel values ​​of the initial image under a fixed camera gain are obtained, and a histogram of the image pixel values ​​is created accordingly.

[0131] In this step, the pixel values ​​of the initial image are measured across all channels at a fixed gain, and a histogram of the image pixel values ​​is plotted. Since pixel values ​​measured with a smaller gain value are susceptible to random errors, a larger gain value (gain = 8) is used as the basic measurement condition. The pixel values ​​of the image are measured in each channel, and a histogram of the image pixel values ​​is plotted. In the histogram, the horizontal axis represents the pixel value, and the vertical axis represents the probability distribution of the pixel value. This yields the pixel histogram curve of the entire image. This curve conforms to a certain distribution law. The following steps use several common function distributions to fit this curve to see which distribution best matches the distribution of dark current noise.

[0132] In step S1024, histograms are generated for various functional distributions based on the pixel mean and variance.

[0133] In this step, after obtaining the mean and variance of the dark current, the next step is to fit the statistical distribution law of the dark current. Based on the pixel mean and variance, histograms are plotted under various functional distributions, where the mean μ = ab and the variance σ... 2 =ab 2 .

[0134] For example, several common noise distributions (Gaussian distribution, Poisson distribution, and Gamma distribution) are used to fit the statistical distribution law of dark current. Among them, the Gamma distribution is implemented using the built-in distribution function of MATLAB, as shown in the following equation (8):

[0135]

[0136] The probability density functions of the Gaussian and Poisson distributions are shown in equations (9) and (10) below:

[0137]

[0138]

[0139] Where μ and σ are the mean and variance of the Gaussian distribution, respectively, and λ is the built-in parameter of the Poisson distribution.

[0140] In step S1025, the fitting degree between histograms under various different function distributions and histograms of image pixel values ​​is compared, and the function distribution that best matches is selected as the distribution of dark current noise.

[0141] In this step, the histogram curves of these three distributions are compared with the histogram curves of the actual data, and the distribution that best fits the histogram curve of the actual data is selected. In the experiment, a gain value of gain = 8 was used to measure the mean and variance of the dark current under the three RGB channels. When plotting the histogram curves of the three distributions based on the mean and variance, pixels with values ​​greater than 20 only account for about 0.1%, and for ease of display, pixel values ​​greater than 20 are omitted in the statistics. The curves of the Gaussian, Poisson, and Gamma distributions are shown below. Figures 5a-5c As shown.

[0142] like Figures 5a-5cAs shown in the graphs of the three statistical distributions, the Gaussian distribution curve becomes flatter as the variance σ increases; the shape of the Poisson distribution curve is greatly affected by the value of λ, and the peak of the curve gradually shifts to the right as λ increases; while the shape of the gamma distribution curve is determined by the two parameters a and b, and its waveform changes less with the parameters. Verification revealed that the histogram curve of the gamma distribution fits the histogram curve of the actual data best, with both curve shape and pixel value being relatively close. The Gaussian distribution is second best, while the Poisson distribution curve shows the greatest difference from the histogram curve of the actual data, with significant differences in both curve shape and value. Therefore, the gamma distribution was ultimately chosen as the distribution for dark current noise. Of course, this embodiment only provides several common function distributions to fit the histogram curves of pixel values ​​in actual images, selecting the most suitable function distribution. In practical applications, other suitable function distributions can be selected for fitting based on factors such as the shape of the curve.

[0143] After obtaining the distribution of dark current noise, it is injected into the image so that the distribution of pixel values ​​at each point in the image matches the distribution of dark current noise, thereby reducing the resolution of the image.

[0144] Figure 6 A flowchart illustrating the calibration process for Gaussian noise according to an embodiment of the present invention is shown.

[0145] like Figure 6 As shown, the process of calibrating Gaussian noise is similar to the process of calibrating dark current noise. The calibration process of Gaussian noise includes steps S1031-S1035.

[0146] In step S1031, the data of the initial image and multiple different camera gains are set.

[0147] In this step, the capsule output image is set to JPEG-480x480, rawdata, and saved as a bmp image. The camera gain values ​​are set to 1, 2, 4, and 8 respectively.

[0148] In step S1032, after the initial image is placed in a dark room environment with multiple different camera gains and illuminated for a fixed time, the pixel mean and variance of the entire image under the three RGB channels are calculated respectively.

[0149] In this step, the initial image is placed in a darkroom environment with different camera gains, illuminated by LED lights for milliseconds, and photographed against a white sheet of paper in the darkroom. The camera gain values ​​are set to 1, 2, 4, and 8, respectively, corresponding to 100, 102, 100, and 104 images saved. Then, the pixel mean and pixel variance of the entire image in the RGB three channels are calculated using the following formulas (11) and (12):

[0150]

[0151]

[0152] Where T is the moment of the last lighting, and I(x, y, t) is the pixel value at the (x, y) coordinate of the image at the t-th lighting event. V represents the pixel mean, and V represents the variance.

[0153] In step S1033, a graph of pixel mean versus variance is generated based on the relationship between pixel mean and variance.

[0154] In this step, the relationship between pixel mean and variance is analyzed, and... The graph shows the mean on the x-axis and the variance on the y-axis. Experiments revealed that the three channels... The graphs all exhibit characteristic peaks, with the curves containing three distinct peaks. To fit the actual data curves, it is assumed that... The curve consists of four parts: y = y1 + y2 + y3 + y4, where y1 = kx + d.

[0155]

[0156]

[0157] Here, k, d, a1, a2, a3, and a4 are calibration parameters.

[0158] In step S1034, the average pixel value of the image captured by the camera is obtained, and the variance of the image is found based on the curve.

[0159] In this step, since variance calculation is relatively difficult, a mean-variance curve was first plotted based on the experimental data. Then, the pixel mean of the actual captured images was obtained, based on the data from the previous step. The graph shows the variance corresponding to the mean. To obtain the variance, simply substitute the mean into the appropriate formula in y1-y4 above.

[0160] In step S1035, Gaussian noise with zero pixel mean is generated based on the obtained variance. The Gaussian noise conforms to a Gaussian distribution.

[0161] In this step, zero-mean Gaussian noise is generated using the obtained variance. Since the distribution of Gaussian noise conforms to a Gaussian distribution, a Gaussian noise curve with zero pixel mean under the corresponding variance can be obtained. Adding this Gaussian noise, which conforms to this distribution, to the image makes the distribution of pixel values ​​at each point conform to a Gaussian distribution, thereby reducing the pixel value of the image.

[0162] Accordingly, the present invention also provides an image processing apparatus for implementing the image processing methods of the above embodiments, see below. Figure 7.

[0163] Figure 7 A simplified schematic diagram of an image processing apparatus according to an embodiment of the present invention is shown.

[0164] like Figure 7 As shown, the image processing apparatus 700 of this embodiment includes: an image blurring unit 701, a first noise injection unit 702, a downsampling unit 703, a first compression unit 704, a second noise injection unit 705, and a second compression unit 706.

[0165] Image blurring unit 701 combines the acquired high-resolution high-definition image with a calibration blur kernel to obtain a blurred image; first noise injection unit 702 injects calibration noise into the blurred image to form a first noise image; downsampling unit 703 degrades the first noise image into a low-resolution image using a downsampling method; first compression unit 704 performs lossy compression on the low-resolution image to reduce storage space and obtain a compressed image; second noise injection unit 705 simulates random noise during transmission and adds random noise points to the compressed image to form a second noise image; second compression unit 706 performs lossy compression on the second noise image and saves it to obtain a low-resolution low-definition image. The high-definition image is an image formed by capturing internal organs using a capsule endoscope, and the calibration blur kernel and calibration noise are results obtained by fitting the high-definition image capture process.

[0166] The image blurring unit 701 is also used to acquire a calibration blur kernel, which includes a focus blur kernel and a motion blur kernel. The focus blur kernel characterizes the image blur caused by distortion and focus during the capture of high-definition images by the camera of the capsule endoscope, and the motion blur kernel characterizes the image blur caused by the movement of the camera inside the body. The first noise injection unit 702 is also used to acquire calibration noise, which includes dark current noise and Gaussian noise. Dark current noise characterizes the noise generated during the conversion of the image captured by the camera of the capsule endoscope into a digital signal, and Gaussian noise characterizes the noise generated during the formation of an RGB image.

[0167] Furthermore, the first noise injection unit 702 and the second noise injection unit 705 can be combined into the same unit, for example, a noise injection unit; similarly, the first compression unit 704 and the second compression unit 706 can be combined into a compression unit. Thus, the image processing device of the present invention has four units: an image blurring unit, a noise injection unit, a downsampling unit, and a compression unit, each performing a corresponding function to reduce the resolution of a high-definition image, referred to as a low-definition image.

[0168] The image processing method and apparatus described in the above embodiments simulate the degradation process of high-definition images captured by a capsule endoscope inside the body. The image processing incorporates blurring caused by the simulated movement of the capsule within the body and noise introduced during processing. Downsampling and lossy compression are then used to repeatedly reduce the resolution of the high-definition image, resulting in a low-resolution compressed image. Furthermore, noise from the simulated transmission process is added to the compressed image, and lossy compression is performed again during transmission. This yields an accurate degraded image (low-definition image) that reflects the actual image degradation process. The low-definition image obtained using this image processing method effectively reflects the various factors affecting the high-definition image during degradation. Based on this, accurate high-definition-low-definition image data pairs can be obtained, allowing for the establishment of corresponding datasets and models, such as... Figure 8 As shown in the example.

[0169] Figure 8 A flowchart of a dataset acquisition method according to an embodiment of the present invention is shown.

[0170] like Figure 8 As shown, the dataset collection method in this embodiment includes steps S201-S208, and steps S201-S206 and Figure 1b Steps S101-S106 are completely identical. The dataset collection method in this embodiment is... Figures 1a-6 This implementation is based on the image processing method described in the embodiment. The same steps will not be repeated here. Here, we mainly introduce steps S207-S208.

[0171] In step S201, the acquired high-resolution high-definition image is processed with a calibration blur kernel to obtain a blurred image.

[0172] In step S202, calibration noise is injected into the blurred image to form a first noise image.

[0173] In step S203, the first noisy image is degraded into a low-resolution image using a downsampling method.

[0174] In step S204, lossy compression is performed on the low-resolution image to obtain a compressed image.

[0175] In step S205, random noise during the transmission process is simulated, and random noise points are added to the compressed image to form a second noise image.

[0176] In step S206, the second noisy image is lossily compressed and saved to obtain a low-resolution, low-definition image.

[0177] In step S207, the high-definition image and its corresponding low-definition image are saved as a pair of image data.

[0178] In this step, the degraded low-resolution image and the original unprocessed high-resolution image are saved as a pair of image data.

[0179] In step S208, multiple image data pairs are acquired to create a training dataset.

[0180] In this step, steps S201-S206 above are repeated to obtain low-resolution images corresponding to multiple high-resolution images, and multiple high-resolution-low-resolution image data pairs are established. These data pairs are used together as a training dataset. The training dataset established by the dataset acquisition method of this embodiment can be applied to the acquisition of sample sets when building models in various image processing processes, such as image super-resolution, image deblurring, and image denoising, which improves the precision and accuracy of image processing, can well reflect the texture of each part of the image, and is convenient for medical analysis and research.

[0181] The image processing method, dataset acquisition method, and image processing device provided by this invention simulate the degradation process of high-definition images captured by a capsule endoscope inside the body. During image processing, blurring caused by the movement of the capsule within the body and noise introduced during processing are incorporated. Downsampling and lossy compression methods are then used to repeatedly reduce the resolution of the high-definition image, resulting in a low-resolution compressed image. Furthermore, noise from the transmission process is simulated in the compressed image, and lossy compression is performed again during transmission to obtain a more accurate degraded image (low-resolution image). This process reasonably describes the true degradation process of the image. The resulting high-definition and low-resolution image data pairs effectively describe the degradation relationship between images. The dataset established based on this data better reflects the influence of various factors on the image during degradation. The resulting high-resolution image is clearer and closer to reality. The training dataset created by the dataset acquisition method according to this invention can be applied to the establishment of sample sets for model building in various image processing processes.

[0182] As described above, these embodiments of the present invention do not exhaustively cover all details, nor do they limit the invention to the specific embodiments described. Clearly, many modifications and variations can be made based on the above description. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to effectively utilize the invention and its modifications. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. An image processing method, comprising: A blurred image is obtained by applying the acquired high-resolution high-definition image to a calibrated blur kernel; A calibration noise is injected into the blurred image to form a first noisy image; The first noisy image is degraded into a low-resolution image using a downsampling method; The low-resolution image is subjected to lossy compression to obtain a compressed image; Random noise during the simulated transmission process is added to the compressed image to form a second noise image; The second noisy image is lossily compressed and then saved to obtain a low-resolution, low-definition image. The high-definition image is an image of an internal organ captured using a capsule endoscope, and the calibration blur kernel and calibration noise are the results obtained by fitting the capture process of the high-definition image. The calibration blur kernel includes a focus blur kernel and a motion blur kernel. The focus blur kernel characterizes the image blur caused by distortion and focus during the capture of the high-definition image by the camera of the capsule endoscope. The motion blur kernel characterizes the image blur caused by the movement of the camera inside the body. The calibration noise includes dark current noise and Gaussian noise. The dark current noise characterizes the noise generated during signal conversion of the image captured by the camera of the capsule endoscope, and the Gaussian noise characterizes the noise generated during the formation of the RGB image. The first noisy image is downsampled by a factor of two using a bicubic interpolation downsampling method to obtain the low-resolution image.

2. The image processing method according to claim 1, wherein, The blurred image is the result of simultaneously processing the high-definition image with the focus blur kernel and the motion blur kernel.

3. The image processing method according to claim 1, wherein, The calibration steps for the focus blur kernel include: The camera is used to photograph the tilted line of the standard color chart, and the pixel value change curves on both sides of the tilted line are obtained as pulse signals. Calculate the edge function corresponding to the pulse signal and the line propagation function obtained by differentiating the edge function; By rotating the standard color chart at fixed intervals, multiple line propagation functions corresponding to the multiple inclined lines are obtained; The multiple line propagation functions are rotated once and then synthesized into a point propagation function in three-dimensional space; and The focus blur kernel is obtained by normalizing the point propagation function.

4. The image processing method according to claim 1, wherein, The motion blur kernel is simulated using a Gaussian blur kernel, which includes an isotropic blur kernel and an anisotropic blur kernel.

5. The image processing method according to claim 1, wherein, The calibration steps for the dark current noise include: Set the initial image data, place it in a black cloth environment with different camera gains, and after lighting for a fixed time, calculate the average pixel value, horizontal pixel value, and vertical pixel value of the entire image in the three RGB channels of the initial image respectively. Calculate the variance of the full image for each channel under different camera gains; Obtain the full image pixel values ​​of the initial image under a fixed camera gain, and create a histogram of the image pixel values ​​accordingly. Based on the pixel mean and variance, histograms were generated for various functional distributions. By comparing the fitting degree between the histograms under various different function distributions and the histograms of the image pixel values, the function distribution that best matches is selected as the distribution of the dark current noise.

6. The image processing method according to claim 5, wherein, The various function distributions include Gaussian distribution, Poisson distribution and gamma distribution, and the best-fit function distribution is gamma distribution.

7. The image processing method according to claim 1, wherein, The calibration steps for the Gaussian noise include: Set the initial image data and multiple different camera gains; After placing the initial image in a darkroom environment with multiple camera gains and lighting for a fixed time, the pixel mean and variance of the entire image in the three RGB channels are calculated respectively. Create a graph of pixel mean versus variance based on the relationship between the pixel mean and the variance; Obtain the pixel mean of the image captured by the camera, and find the variance of the image based on the curve; and Gaussian noise with zero pixel mean is generated based on the obtained variance, and the Gaussian noise conforms to a Gaussian distribution.

8. The image processing method according to claim 1, wherein, The step of injecting calibration noise into the blurred image to form a first noisy image includes: Obtain the function distributions corresponding to the dark current noise and the Gaussian noise, respectively; The dark current noise and the Gaussian noise are injected into the blurred image according to their corresponding function distributions to form a first noise image.

9. The image processing method according to claim 1, wherein, The lossy compression coefficients used in compressing the low-resolution image into a compressed image and in compressing the second noisy image into a low-resolution image are 70 and 90, respectively.

10. The image processing method according to claim 9, wherein, The steps for lossy compression of an image include: The image is converted from RGB data to YUV data, and 4:2:0 chromaticity sampling is performed simultaneously. The image in the YUV data format is divided into 8x8 cells, and a discrete cosine transform is performed on each cell; After performing the discrete cosine transform, the image is quantized, and high-frequency data is discarded. Entropy encoding is performed on the matrix corresponding to the quantized image to form a compressed image.

11. A dataset collection method, wherein, include: The image processing method according to any one of claims 1-10; as well as The high-resolution image and its corresponding low-resolution image are saved as a pair of image data; A training dataset is created by acquiring multiple pairs of the aforementioned image data.

12. An image processing apparatus for implementing the image processing method according to any one of claims 1-10, the image processing apparatus comprising: The image blurring unit combines the acquired high-resolution image with a calibrated blur kernel to obtain a blurred image. A first noise injection unit injects calibration noise into the blurred image to form a first noise image; The downsampling unit degrades the first noisy image into a low-resolution image using a downsampling method; as well as The first compression unit performs lossy compression on the low-resolution image to reduce storage space and obtain a compressed image. The second noise injection unit simulates random noise during transmission and adds random noise points to the compressed image to form a second noise image; The second compression unit performs lossy compression on the second noisy image and saves it to obtain a low-resolution, low-definition image. The high-definition image is an image of an internal organ captured using a capsule endoscope, and the calibration blur kernel and calibration noise are the results obtained by fitting the capture process of the high-definition image.

Images