An image inpainting method, device, electronic equipment and storage medium

Abstract

The application relates to an image repairing method and device, electronic equipment and a storage medium, which comprises the following steps: determining the boundary of a current to-be-repaired region of a first image; determining a current sample block size; generating a first to-be-repaired block with a first pixel point as the center and a set of neighborhood blocks adjacent to the first to-be-repaired block according to the current sample block size; wherein the first pixel point is one of a set of pixel points on the boundary; obtaining the correlation coefficient of the first to-be-repaired block and the set of neighborhood blocks; determining the priority of the first pixel point according to the correlation coefficient, the confidence of the first pixel point and the data item of the first pixel point; determining the current to-be-repaired pixel point and the corresponding target matching block according to the priority of each pixel point in the set of pixel points on the boundary, and repairing the to-be-repaired block corresponding to the current to-be-repaired pixel point according to the target matching block. The obtained pixel point priority result is more comprehensive and accurate, and the image repairing quality and efficiency are comprehensively improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image restoration method, apparatus, electronic device, and storage medium. Background Technology

[0002] In current clinical medicine, medical images are often needed to assist in diagnosis and treatment, such as taking CT images for pathological analysis, or using marker balls that can be visualized on X-ray films to help determine the location of lesions or the position and orientation of surgical instruments during surgery. However, some areas in the generated X-ray images are subject to noise interference or are obscured by marker balls, affecting the doctor's observation of the X-ray images. Therefore, we need to use image restoration methods to repair defects such as noise interference and marker ball obstruction in X-ray images.

[0003] Image inpainting aims to repair lost areas using known information from the image according to certain rules. The Criminisi algorithm is a commonly used image inpainting algorithm. The Criminisi algorithm selects the highest-priority pixel p on the edge of the region to be repaired, then constructs an n×n pixel block centered on p. It then searches for the most similar sample block in the intact region and updates the information to be repaired in that pixel block with the found sample block, starting the next iteration until the repair is complete. The Criminisi algorithm calculates the pixel priority using the formula P(p) = C(p) * D(p), where C(p) represents the confidence of the repair block centered on pixel p, and D(p) represents the intensity of the isoluminescence lines intersecting the boundary. C(p) may gradually become zero as the image is repaired, and the isoluminescence lines and normals in D(p) may also be zero. Furthermore, C(p) and D(p) may exhibit two extreme values, one large and one small, all of which can lead to inaccurate priority calculation results. In addition, the Criminisi algorithm uses a fixed size for the sample blocks and does not adjust the size based on image information. Furthermore, the algorithm employs a global search, which is time-consuming.

[0004] The existing Criminisi algorithm suffers from shortcomings such as deviations in pixel priority calculation due to extreme cases, and high time complexity during image restoration. Therefore, there is an urgent need for a technical solution for accurate and efficient image restoration. Summary of the Invention

[0005] This invention provides an image restoration method, apparatus, electronic device, and storage medium to address the shortcomings of the existing Criminisi algorithm in pixel priority calculation, which suffers from extreme cases leading to inaccurate priority calculation results and high time complexity during image restoration, thereby achieving accurate and efficient image restoration.

[0006] This invention provides an image restoration method, comprising:

[0007] Determine the boundaries of the current region to be repaired in the first image;

[0008] Determine the current sample block size;

[0009] Based on the current sample block size, a first block to be repaired, centered on a first pixel, and a set of neighboring blocks adjacent to the first block to be repaired are generated; wherein, the first pixel is one of the set of pixels on the boundary;

[0010] Obtain the correlation coefficient between the first block to be repaired and the set of neighboring blocks;

[0011] The priority of the first pixel is determined based on the correlation coefficient, the confidence level of the first pixel, and the data items of the first pixel.

[0012] Based on the priority of each pixel in the set of pixels on the boundary, determine the current pixel to be repaired and its corresponding target matching block, and repair the block corresponding to the current pixel to be repaired based on the target matching block;

[0013] The current sample block size is obtained by updating the previous sample block size based on the initial sample block size or based on the information entropy of the target matching block in the previous repair process.

[0014] According to an image restoration method provided by the present invention, determining the boundary of the current region to be restored in a first image includes:

[0015] The first image is input into the target detection model to determine the boundary of the current region to be repaired in the first image; wherein, the target detection model is a pre-trained neural network model, the feature fusion node of the target detection model is equipped with a channel attention mechanism module, and the number of output nodes and the output size of the target detection model are preset according to the size characteristics of the region to be repaired in the first image.

[0016] According to an image restoration method provided by the present invention, obtaining the correlation coefficient between the first block to be restored and the set of neighboring blocks includes:

[0017] The correlation coefficient is generated based on the variance of the first block to be repaired and the variance of each neighboring block in the neighborhood block set; wherein, the neighborhood block set consists of eight neighboring blocks adjacent to the first block to be repaired at the center grid that do not contain pixels in the area to be repaired, determined by a grid of the same size as the first block to be repaired.

[0018] According to an image restoration method provided by the present invention, the correlation coefficient is obtained by the following formula:

[0019]

[0020] Where R(p) represents the correlation coefficient between the first block to be repaired and the set of neighboring blocks, X represents the first block to be repaired, Y represents the neighboring blocks in the set of neighboring blocks, Var[X] represents the variance of the random variable of X, Var[Y] represents the variance of the random variable of Y, Cov(X,Y) represents the covariance of X and Y, and n is the number of neighboring blocks in the set of neighboring blocks.

[0021] According to an image restoration method provided by the present invention, determining the priority of the first pixel based on the correlation coefficient, the confidence level of the first pixel, and the data items of the first pixel includes:

[0022] The priority of the first pixel is obtained by weighted summation of the correlation coefficient, the confidence level of the first pixel, and the data item of the first pixel; wherein, the correlation coefficient represents the degree of correlation between the image at the first pixel and the images in the adjacent regions, the confidence level of the first pixel represents the proportion of known information contained in the image at the first pixel, and the data item represents the linear complexity of the image structure at the first pixel.

[0023] According to an image restoration method provided by the present invention, determining the current sample block size includes:

[0024] If the current round is the first image restoration, the initial sample block size is obtained as the current sample block size; or, if the current round is not the first image restoration,

[0025] Obtain the information entropy of the target matching block from the previous repair process;

[0026] If the information entropy is less than the first threshold, increase the size of the current sample block;

[0027] If the information entropy is greater than or equal to the first threshold and less than or equal to the second threshold, the current sample block size remains unchanged;

[0028] If the information entropy is greater than the second threshold, reduce the current sample block size;

[0029] The first threshold and the second threshold are both within the range of (0, 1), and the first threshold is less than the second threshold.

[0030] According to an image restoration method provided by the present invention, the step of determining the current pixel to be restored and its corresponding target matching block based on the priority of each pixel in the set of pixels on the boundary, and restoring the block to be restored corresponding to the current pixel to be restored based on the target matching block, includes:

[0031] Multiple image blocks are generated within a first distance range of the current pixel to be repaired, and the target matching block is determined by calculating the similarity between the multiple image blocks and the block to be repaired corresponding to the pixel to be repaired.

[0032] Repair the block to be repaired corresponding to the current pixel to be repaired based on the target matching block;

[0033] The first distance is updated based on the similarity or correlation coefficient between the target matching block and the block to be repaired corresponding to the pixel to be repaired; wherein the initial value of the first distance is determined based on the distribution characteristics of the region to be repaired in the first image.

[0034] According to an image restoration method provided by the present invention, updating the first distance based on the similarity or correlation coefficient between the target matching block and the block to be restored corresponding to the pixel to be restored includes:

[0035] If the similarity is less than the third threshold, or the correlation coefficient is less than zero, the first distance is increased.

[0036] The present invention also provides an image restoration apparatus, comprising:

[0037] The first determining module is used to determine the boundary of the current region to be repaired in the first image;

[0038] The second determining module is used to determine the current sample block size;

[0039] The generation module is used to generate a first block to be repaired centered on a first pixel, and a set of neighboring blocks adjacent to the first block to be repaired, based on the current sample block size; wherein, the first pixel is one of the set of pixels on the boundary;

[0040] The acquisition module is used to acquire the correlation coefficient between the first block to be repaired and the set of neighboring blocks;

[0041] The third determining module is used to determine the priority of the first pixel based on the correlation coefficient, the confidence level of the first pixel, and the data items of the first pixel.

[0042] The repair module is used to determine the current pixel to be repaired and its corresponding target matching block according to the priority of each pixel in the set of pixels on the boundary, and repair the block to be repaired corresponding to the current pixel to be repaired according to the target matching block;

[0043] The current sample block size is obtained by updating the previous sample block size based on the initial sample block size or based on the information entropy of the target matching block in the previous repair process.

[0044] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement all or part of the steps of any of the above-described image restoration methods.

[0045] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements all or part of the steps of any of the above-described image restoration methods.

[0046] The present invention provides an image restoration method, apparatus, electronic device and storage medium, which comprehensively improves the quality and efficiency of image restoration by using dynamic sample block size. By introducing correlation coefficients combined with the confidence level of the first pixel and the data items of the first pixel to determine the priority of the first pixel, the priority result obtained is more comprehensive and accurate. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0048] Figure 1 This is one of the flowcharts illustrating an image restoration method provided by the present invention;

[0049] Figure 2 This is a schematic diagram of the channel attention mechanism module of the target detection model in an image restoration method provided by the present invention;

[0050] Figure 3 This is a schematic diagram of the target detection model in one embodiment of an image restoration method provided by the present invention;

[0051] Figure 4 This is a schematic diagram of the structure of an image restoration device provided by the present invention;

[0052] Figure 5This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0054] The following is combined Figures 1-5 This invention describes an image restoration method, apparatus, electronic device, and storage medium.

[0055] Figure 1 This is a flowchart illustrating an image restoration method according to the present invention, as shown below. Figure 1 As shown, the method includes:

[0056] S11. Determine the boundary of the current area to be repaired in the first image;

[0057] Specifically, starting from the edge of the area to be repaired and gradually repairing the image can achieve better results. For the first image to be repaired, the boundary of the area to be repaired needs to be determined. Understandably, if there are multiple areas to be repaired, they need to be repaired separately, and the boundary of each area needs to be determined separately.

[0058] S12. Determine the current sample block size; wherein, the current sample block size is obtained by updating the sample block size of the previous round based on the initial sample block size or based on the information entropy of the target matching block in the previous repair process.

[0059] Specifically, in the prior art, the Criminisi algorithm generates a block to be repaired with a fixed sample block size and then starts repairing it step by step. In this invention, the size of the sample block is dynamic. Before generating the block to be repaired in each round of repair, the current sample block size needs to be determined first.

[0060] The current sample block size is determined based on the initial sample block size or by updating the previous round's sample block size based on the information entropy of the target matching block in the previous repair process. That is, if this is the first repair attempt, the current sample block size is determined based on the preset initial sample block size; if this is not the first repair attempt, the current sample block size is determined by updating the previous round's sample block size according to preset rules based on the information entropy of the target matching block in the previous repair process. The initial sample block size is set according to actual needs or experience, for example, 5×5, where "5×5" corresponds to a pixel block with 5 pixels per row and column. The target matching block is an image block determined from an intact image region for repairing the block to be repaired, and the information entropy characterizes the amount of information in the target matching block.

[0061] S13. Based on the current sample block size, generate a first block to be repaired centered on the first pixel, and a set of neighboring blocks adjacent to the first block to be repaired; wherein, the first pixel is one of the set of pixels on the boundary;

[0062] Specifically, taking a current sample block size of 7×7 as an example, a 7×7 pixel block is determined on the first image centered on the first pixel as the first block to be repaired. Then, multiple pixel blocks of the same size adjacent to the first block to be repaired are generated on the first image, forming a neighborhood block set of the first block to be repaired. The first pixel is one of the pixels in the set on the boundary; that is, for pixels on the boundary of the current area to be repaired, priority needs to be determined separately, and steps S13 to S15 are executed separately. Of course, it is understandable that for pixels that have not changed in the previous repair process, the priority of that pixel in the previous round can be directly used without recalculation.

[0063] S14. Obtain the correlation coefficient between the first block to be repaired and the set of neighboring blocks;

[0064] Specifically, there are multiple neighboring blocks in the neighborhood block set. The correlation between each neighboring block and the first block to be repaired is calculated, and then the correlation coefficient between the first block to be repaired and the neighborhood block set is obtained. The correlation coefficient represents the degree of similarity between the first block to be repaired and the surrounding area.

[0065] S15. Determine the priority of the first pixel based on the correlation coefficient, the confidence level of the first pixel, and the data items of the first pixel;

[0066] Specifically, the correlation coefficient represents the degree of correlation between the image at the first pixel and the images in the adjacent regions, the confidence of the first pixel represents the proportion of known information contained in the image at the first pixel, and the data item represents the linear complexity of the image structure at the first pixel. By combining the correlation coefficient between the first block to be repaired and the set of neighboring blocks, the confidence of the first pixel, and the data item of the first pixel, priority can be obtained more comprehensively and accurately.

[0067] S16. Based on the priority of each pixel in the set of pixels on the boundary, determine the current pixel to be repaired and its corresponding target matching block, and repair the block to be repaired corresponding to the current pixel to be repaired based on the target matching block;

[0068] Specifically, after determining the priority of each pixel in the set of pixels on the boundary, the order of pixel repair can be determined, that is, which position on the boundary of the region to be repaired should be prioritized for repair. After determining the current pixel to be repaired, a matching block of the same size as the sample block is further searched in the intact image region, and the target matching block with the highest similarity to the current pixel to be repaired is determined from multiple matching blocks. The current pixel to be repaired is repaired according to the target matching block, and then the process iterates to the next round of repair.

[0069] In this embodiment, the image restoration quality and efficiency are comprehensively improved by using dynamic sample block size. The priority of the first pixel is determined by introducing correlation coefficients, combining the confidence level of the first pixel and the data items of the first pixel, making the obtained priority result more comprehensive and accurate.

[0070] Based on any of the above embodiments, in one embodiment, determining the boundary of the current region to be repaired in the first image includes:

[0071] The first image is input into the target detection model to determine the boundary of the current region to be repaired in the first image; wherein, the target detection model is a pre-trained neural network model, the feature fusion node of the target detection model is equipped with a channel attention mechanism module, and the number of output nodes and the output size of the target detection model are preset according to the size characteristics of the region to be repaired in the first image.

[0072] Specifically, in this embodiment, an object detection model is used to detect the region to be repaired, and then the boundary of the current region to be repaired is determined. The object detection model is a pre-trained neural network model. The object detection model in this embodiment has a channel attention mechanism module in the feature fusion node. Figure 2 This is a structural diagram of the channel attention mechanism module, such as... Figure 2As shown, to better calculate the importance of each feature, the channel attention mechanism module uses global max pooling and global average pooling to compress the input feature map F, resulting in the global max pooling feature map. Global average pooling feature map The elements are then summed one by one, and the result is fed into the first fully connected layer (FC) to construct the correlation between channels and compress the dimension. The dimension is then restored to its initial dimension through the first activation function layer (ReLU) and the second fully connected layer (FC). Finally, the calculated weight values ​​are normalized to between 0 and 1 through the second activation function layer (Sigmoid), resulting in the channel attention weight feature map M. c (F)∈R C×1×1 Finally, the feature map F is multiplied by the original input feature map F at the feature labeling layer to obtain a new feature map F′ = M. c (F)×F, the feature fusion node can output the feature-enhanced image based on the new feature map, thereby accurately determining the area to be repaired.

[0073] Furthermore, in this embodiment, the number of output nodes and the output size of the target detection model are preset based on the size characteristics of the region to be repaired in the first image. For example, for an image to be repaired where the marked ball position needs to be repaired, since the actual size of the marked ball used is known in advance, such as 5mm, the diameter of the marked ball region to be repaired in the image is determined to be approximately 10 pixels. Therefore, the neural network model is set to include two feature map output sizes: 52×52 and 26×26, eliminating the existing neural network model's 13×13 prediction scale. Figure 3 This is a schematic diagram of the corresponding neural network model, such as... Figure 3 As shown in the figure, the output of the existing neural network model at the 13×13 prediction scale has been removed, and only the 13×13 feature image extracted from the features is used as reference information for the upper-level 26×26 image. This is because the 13×13 prediction scale has a deep receptive field and is responsible for detecting larger target regions, while the marker ball in the input first image is small, and this feature cannot be extracted in the deep network. Therefore, the output of the 13×13 prediction scale of this layer is removed and used only as reference information for the upper-level scale.

[0074] In this embodiment, the image features are enhanced through the channel attention mechanism, making the detection of the current region to be repaired more accurate. By pre-setting the number of output nodes and the output size of the target detection model according to the size features of the region to be repaired in the first image, the efficiency of detecting the current region to be repaired is improved by utilizing the prior knowledge of the size features of the known region to be repaired.

[0075] Based on any of the above embodiments, in one embodiment, obtaining the correlation coefficient between the first block to be repaired and the set of neighboring blocks includes:

[0076] The correlation coefficient is generated based on the variance of the first block to be repaired and the variance of each neighboring block in the neighborhood block set; wherein, the neighborhood block set consists of eight neighboring blocks adjacent to the first block to be repaired at the center grid that do not contain pixels in the area to be repaired, determined by a grid of the same size as the first block to be repaired.

[0077] Specifically, in the first image to be repaired, in the "nine grid" with the first block to be repaired as the center, the size of the eight surrounding grids is the same as the size of the first block to be repaired at the center grid. The neighboring blocks of the eight surrounding grids that do not contain the pixels of the area to be repaired constitute the neighborhood block set. The correlation coefficient between the first block to be repaired and the neighborhood block set is determined based on the variance of the first block to be repaired and the variance of each neighboring block in the neighborhood block set.

[0078] In this embodiment, pixels in the region to be repaired are excluded during the determination of the correlation coefficient, and the size of the neighboring block is the same as that of the first block to be repaired, so that the variance of the first block to be repaired and the variance of each neighboring block in the neighboring block set are comparable, and the correlation coefficient calculation is more accurate.

[0079] Based on any of the above embodiments, in one embodiment, the correlation coefficient is obtained by the following formula:

[0080]

[0081] Where R(p) represents the correlation coefficient between the first block to be repaired and the set of neighboring blocks, X represents the first block to be repaired, Y represents the neighboring blocks in the set of neighboring blocks, Var[X] represents the variance of the random variable of X, Var[Y] represents the variance of the random variable of Y, Cov(X,Y) represents the covariance of X and Y, and n is the number of neighboring blocks in the set of neighboring blocks.

[0082] In this embodiment, the correlation coefficient was accurately obtained using the above formula.

[0083] Based on any of the above embodiments, in one embodiment, determining the priority of the first pixel based on the correlation coefficient, the confidence level of the first pixel, and the data items of the first pixel includes:

[0084] The priority of the first pixel is obtained by weighted summation of the correlation coefficient, the confidence level of the first pixel, and the data item of the first pixel; wherein, the correlation coefficient represents the degree of correlation between the image at the first pixel and the images in the adjacent regions, the confidence level of the first pixel represents the proportion of known information contained in the image at the first pixel, and the data item represents the linear complexity of the image structure at the first pixel.

[0085] Specifically, the priority of the first pixel is obtained by weighted summation based on the correlation coefficient between the first block to be repaired and its neighboring block set, the confidence level of the first pixel, and the data items of the first pixel. The weights can be preset or dynamically adjusted based on the actual image restoration effect. Setting weights makes the calculation of pixel priority more comprehensive and tailored to the characteristics of specific image types, thereby improving the image restoration effect.

[0086] In this embodiment, the priority of the first pixel is determined by weighted summation based on the correlation coefficient between the first block to be repaired and the set of neighboring blocks, the confidence level of the first pixel, and the data items of the first pixel. This makes the priority calculation more comprehensive and accurate, avoiding the defect of inaccurate pixel priority in extreme cases.

[0087] Based on any of the above embodiments, in one embodiment, the priority of the first pixel is obtained by the following formula:

[0088] P(p)=a·C(p)+b·D(p)+c·R(p);

[0089] Where p is the first pixel, P(p) is the priority of the first pixel, a, b, c are weight values, requiring a + b + c = 1 and 0 ≤ a, b, c ≤ 1, C(p) is the confidence level of the first pixel, D(p) is the data item of the first pixel, and R(p) is the correlation coefficient.

[0090] Based on any of the above embodiments, in one embodiment, the confidence level of the first pixel is obtained by the following formula:

[0091]

[0092] Where p is the first pixel, C(p) is the confidence level of the first pixel p, I represents the first image, Ω represents the region to be repaired, and ψ p This represents the first block to be repaired. q is a pixel in the first block to be repaired that does not belong to the region to be repaired.

[0093] Based on any of the above embodiments, in one embodiment, the data item of the first pixel is obtained by the following formula:

[0094]

[0095] Where p is the first pixel, and D(p) is the data item of the first pixel. Indicates the direction of isoluminescence, n p Let α represent the normal vector of pixel p, and α = 255 represent the normalization factor.

[0096] Based on the above embodiments, in one embodiment, determining the current sample block size includes:

[0097] If the current round is the first image restoration, the initial sample block size is obtained as the current sample block size; or, if the current round is not the first image restoration, ...

[0098] Obtain the information entropy of the target matching block from the previous repair process;

[0099] If the information entropy is less than the first threshold, increase the size of the current sample block;

[0100] If the information entropy is greater than or equal to the first threshold and less than or equal to the second threshold, the current sample block size remains unchanged;

[0101] If the information entropy is greater than the second threshold, reduce the current sample block size;

[0102] The first threshold and the second threshold are both within the range of (0, 1), and the first threshold is less than the second threshold.

[0103] Specifically, in the first round of image restoration, the initial sample block size is used as the current sample block size. In the first round of image restoration, based on the information entropy of the target matching block in the previous restoration process, if the information entropy is less than a first threshold, the current sample block size is increased; if the information entropy is greater than or equal to the first threshold and less than or equal to the second threshold, the current sample block size is kept unchanged; if the information entropy is greater than the second threshold, the current sample block size is decreased.

[0104] Image information entropy represents the amount of information contained in an image; it is a statistical form of a feature that reflects the average amount of information in the image. One-dimensional image information entropy represents the amount of information contained in the clustering features of gray-level distribution in the image. When the information entropy of the target sample block is less than a certain level, it can be determined that the image quality of the target sample block is poor. Increasing the size of the current sample block will correspondingly increase the size of the target matching block in the current round, thus improving the image restoration quality. Conversely, when the information entropy of the target sample block is greater than a certain level, it can be determined that the image quality of the target sample block is redundant. Decreasing the size of the current sample block will correspondingly decrease the size of the target matching block in the current round, thus improving the image restoration efficiency.

[0105] In this embodiment, the current sample block size is dynamically determined based on the information entropy of the target matching block in the previous repair process, thereby dynamically determining the pixel priority and the target matching block size, which comprehensively improves the image repair quality and image repair efficiency.

[0106] Based on the above embodiments, in one embodiment, the current sample block size is obtained by the following formula:

[0107]

[0108] Where n represents the nth iteration of the repair, N n N represents the sample block size during the nth round of repair. n-1 H represents the sample block size during the (n-1)th round of repair. n-1 K1 represents the information entropy of the target matching block in the (n-1)th round of repair process, K2 represents the first threshold, and K1 represents the second threshold.

[0109] In this embodiment, the current sample block size is accurately obtained using the above formula.

[0110] Based on the above embodiments, in one embodiment, the information entropy of the target matching block is obtained according to the following formula:

[0111]

[0112] Where H is the information entropy, i is the gray value of a pixel, and P i It is the probability of a pixel with a grayscale value of i appearing. The formula assumes P i When = 0, log(P) i ) = 0.

[0113] Based on the above embodiments, in one embodiment, determining the current pixel to be repaired and its corresponding target matching block according to the priority of each pixel in the set of pixels on the boundary, and repairing the block to be repaired corresponding to the current pixel to be repaired according to the target matching block, includes:

[0114] Multiple image blocks are generated within a first distance range of the current pixel to be repaired, and the target matching block is determined by calculating the similarity between the multiple image blocks and the block to be repaired corresponding to the pixel to be repaired.

[0115] Repair the block to be repaired corresponding to the current pixel to be repaired based on the target matching block;

[0116] The first distance is updated based on the similarity or correlation coefficient between the target matching block and the block to be repaired corresponding to the pixel to be repaired; wherein the initial value of the first distance is determined based on the distribution characteristics of the region to be repaired in the first image.

[0117] Specifically, the existing Criminisi algorithm uses a global search to determine pixel priority and target matching blocks for pixels, which is time-consuming as it involves searching the entire image. This embodiment uses a dynamic local search to determine the target matching blocks: multiple image blocks are generated within a first distance range of the current pixel to be repaired. The initial value of the first distance is determined based on the distribution characteristics of the repaired area in the first image. For example, if there are multiple marked ball regions to be repaired in the first image, and the minimum distance between adjacent marked balls is L0 pixels, then the first distance can be set to L0. The optimal target matching block is determined by calculating the similarity between the generated image blocks and the corresponding repaired blocks of the pixel to be repaired based on the first distance. Then, the repaired block corresponding to the current pixel to be repaired is repaired based on the target matching block. After repair, the first distance is updated based on the similarity between the target matching block and the corresponding repaired block of the pixel to be repaired, or the correlation coefficient of the current round, to adjust the matching range of the repaired block in the next round of repair based on the repair quality of the current round, thus balancing image repair efficiency and quality. The similarity between the target matching block and the corresponding block to be repaired can be mutual information similarity, histogram distance, Bach distance, etc.

[0118] In this embodiment, the initial matching range of the target matching block is determined based on the distribution characteristics of the region to be repaired in the first image, thereby improving the initial image repair efficiency. The first distance is updated based on the similarity between the target matching block and the corresponding region to be repaired, or the correlation coefficient of the current round, which balances the image repair efficiency and image repair quality.

[0119] Based on the above embodiments, in one embodiment, updating the first distance according to the similarity between the target matching block and the block to be repaired corresponding to the pixel to be repaired or the correlation coefficient includes: increasing the first distance when the similarity between the target matching block and the block to be repaired corresponding to the pixel to be repaired is less than a third threshold, or the correlation coefficient is less than zero.

[0120] Specifically, if the similarity between the target matching block and the block to be repaired corresponding to the pixel to be repaired is less than the third threshold, it indicates that the image of the target matching block and the block to be repaired are significantly different and the image repair quality is poor. It is necessary to find a target matching block with a higher degree of fit to repair the block to be repaired. Therefore, the first distance is increased to expand the search matching range. If the correlation coefficient is less than zero, it indicates that the image of the area surrounding the first pixel is significantly different from the block to be repaired. It is necessary to find a target matching block with a higher degree of fit to repair the block to be repaired. Therefore, the first distance is increased to expand the search matching range.

[0121] In this embodiment, the first distance is updated based on the similarity or correlation coefficient between the target matching block and the block to be repaired corresponding to the pixel to be repaired, thereby improving the image repair quality.

[0122] Based on the above embodiments, in one embodiment, the first distance is updated according to the following formula:

[0123]

[0124] Where n represents the nth iteration of the repair, L n R(p) represents the first distance in the nth round of repair process, R(p) represents the correlation coefficient, S represents the similarity between the target matching block and the block to be repaired corresponding to the pixel to be repaired, and T represents the third threshold.

[0125] The image restoration apparatus provided by the present invention is described below. The image restoration apparatus described below can be referred to in correspondence with the image restoration method described above.

[0126] Figure 4 This is a schematic diagram of the structure of an image restoration device provided by the present invention, as shown below. Figure 4 As shown, the device includes:

[0127] The first determining module 41 is used to determine the boundary of the current area to be repaired in the first image;

[0128] The second determining module 42 is used to determine the current sample block size;

[0129] The generation module 43 is used to generate a first block to be repaired centered on a first pixel, and a set of neighboring blocks adjacent to the first block to be repaired, based on the current sample block size; wherein, the first pixel is one of the set of pixels on the boundary;

[0130] The acquisition module 44 is used to acquire the correlation coefficient between the first block to be repaired and the set of neighboring blocks;

[0131] The third determining module 45 is used to determine the priority of the first pixel based on the correlation coefficient, the confidence level of the first pixel, and the data items of the first pixel.

[0132] Repair module 46 is used to determine the current pixel to be repaired and its corresponding target matching block according to the priority of each pixel in the set of pixels on the boundary, and repair the block to be repaired corresponding to the current pixel to be repaired according to the target matching block;

[0133] The current sample block size is obtained by updating the previous sample block size based on the initial sample block size or based on the information entropy of the target matching block in the previous repair process.

[0134] In this embodiment, the image restoration quality and efficiency are comprehensively improved by using dynamic sample block size. The priority of the first pixel is determined by introducing correlation coefficients, combining the confidence level of the first pixel and the data items of the first pixel, making the obtained priority result more comprehensive and accurate.

[0135] Based on the above embodiments, in one embodiment, the first determining module 41 is specifically used to input the first image into the target detection model to determine the boundary of the current region to be repaired in the first image; wherein, the target detection model is a pre-trained neural network model, the feature fusion node of the target detection model is equipped with a channel attention mechanism module, and the number of output nodes and the output size of the target detection model are preset according to the size characteristics of the region to be repaired in the first image.

[0136] In this embodiment, the image features are enhanced through the channel attention mechanism, making the detection of the current region to be repaired more accurate. By pre-setting the number of output nodes and the output size of the target detection model according to the size features of the region to be repaired in the first image, the efficiency of detecting the current region to be repaired is improved by utilizing the prior knowledge of the size features of the known region to be repaired.

[0137] Based on any of the above embodiments, in one embodiment, the acquisition module 44 is used to generate the correlation coefficient based on the variance of the first block to be repaired and the variance of each neighboring block in the neighboring block set; wherein, the neighboring block set consists of eight neighboring blocks adjacent to the first block to be repaired at the center grid that do not contain pixels of the area to be repaired, determined by a grid of the same size as the first block to be repaired.

[0138] In this embodiment, pixels in the region to be repaired are excluded during the determination of the correlation coefficient, and the size of the neighboring block is the same as that of the first block to be repaired, so that the variance of the first block to be repaired and the variance of each neighboring block in the neighboring block set are comparable, and the correlation coefficient calculation is more accurate.

[0139] Based on any of the above embodiments, in one embodiment, the third determining module 45 is used to perform a weighted summation of the correlation coefficient, the confidence level of the first pixel, and the data item of the first pixel to obtain the priority of the first pixel; wherein, the correlation coefficient represents the degree of correlation between the image at the first pixel and the images in the adjacent regions, the confidence level of the first pixel represents the proportion of known information contained in the image at the first pixel, and the data item represents the linear complexity of the image structure at the first pixel.

[0140] In this embodiment, the priority of the first pixel is determined by weighted summation based on the correlation coefficient between the first block to be repaired and the set of neighboring blocks, the confidence level of the first pixel, and the data items of the first pixel. This makes the priority calculation more comprehensive and accurate, avoiding the defect of inaccurate pixel priority in extreme cases.

[0141] Based on any of the above embodiments, in one embodiment, the second determining module 42 includes:

[0142] The first determining subunit is used to obtain the initial sample block size as the current sample block size when the current round is the first image restoration.

[0143] The second determining subunit is used to: obtain the information entropy of the target matching block in the previous repair process when the current round is not the first image repair; increase the current sample block size when the information entropy is less than a first threshold; keep the current sample block size unchanged when the information entropy is greater than or equal to the first threshold and less than or equal to a second threshold; and decrease the current sample block size when the information entropy is greater than the second threshold; wherein the first threshold and the second threshold are in the range of (0, 1), and the first threshold is less than the second threshold.

[0144] In this embodiment, the current sample block size is dynamically determined based on the information entropy of the target matching block in the previous repair process, thereby dynamically determining the pixel priority and the target matching block size, which comprehensively improves the image repair quality and image repair efficiency.

[0145] Based on any of the above embodiments, in one embodiment, the repair module 46 includes:

[0146] The first repair subunit is used to generate multiple image blocks within a first distance range of the current pixel to be repaired, and to determine the target matching block by calculating the similarity between the multiple image blocks and the repair block corresponding to the pixel to be repaired.

[0147] The second repair subunit is used to repair the block to be repaired corresponding to the current pixel to be repaired based on the target matching block;

[0148] The third repair subunit is used to update the first distance based on the similarity or correlation coefficient between the target matching block and the repair block corresponding to the repair pixel; wherein the initial value of the first distance is determined based on the distribution characteristics of the repair area in the first image.

[0149] In this embodiment, the initial matching range of the target matching block is determined based on the distribution characteristics of the region to be repaired in the first image, thereby improving the initial image repair efficiency. The first distance is updated based on the similarity between the target matching block and the corresponding region to be repaired, or the correlation coefficient of the current round, which balances the image repair efficiency and image repair quality.

[0150] Based on any of the above embodiments, in one embodiment, the third repair subunit is specifically used to increase the first distance when the similarity is less than a third threshold or the correlation coefficient is less than zero.

[0151] In this embodiment, the first distance is updated based on the similarity or correlation coefficient between the target matching block and the block to be repaired corresponding to the pixel to be repaired, thereby improving the image repair quality.

[0152] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute all or part of the steps of the image restoration methods provided above. The method includes: determining the boundary of the current region to be restored in the first image; determining the current sample block size; generating a first region to be restored centered on a first pixel, and a set of neighboring blocks adjacent to the first region to be restored, based on the current sample block size; wherein the first pixel is one of the pixels in the set on the boundary; obtaining the correlation coefficient between the first region to be restored and the set of neighboring blocks; determining the priority of the first pixel based on the correlation coefficient, the confidence level of the first pixel, and the data items of the first pixel; determining the current region to be restored pixel and its corresponding target matching block based on the priority of each pixel in the set of pixels on the boundary, and restoring the region to be restored corresponding to the current region to be restored pixel based on the target matching block; wherein the current sample block size is obtained by updating the previous sample block size based on the initial sample block size or based on the information entropy of the target matching block in the previous restoration process.

[0153] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0154] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is capable of performing all or part of the steps of the image restoration methods provided above, the method comprising: determining the boundary of the current region to be restored in a first image; determining the current sample block size; generating a first region to be restored centered on a first pixel, and a set of neighboring blocks adjacent to the first region to be restored, according to the current sample block size; wherein the first pixel is one of the pixels in the set of pixels on the boundary; obtaining the correlation coefficient between the first region to be restored and the set of neighboring blocks; determining the priority of the first pixel according to the correlation coefficient, the confidence level of the first pixel, and the data items of the first pixel; determining the current region to be restored pixel and its corresponding target matching block according to the priority of each pixel in the set of pixels on the boundary, and restoring the region to be restored corresponding to the current region to be restored pixel according to the target matching block; wherein the current sample block size is obtained by updating the previous sample block size according to the initial sample block size or according to the information entropy of the target matching block in the previous restoration process.

[0155] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs all or part of the steps of the image restoration methods provided above. The method includes: determining the boundary of a current region to be restored in a first image; determining a current sample block size; generating a first region to be restored centered on a first pixel, and a set of neighboring blocks adjacent to the first region to be restored, based on the current sample block size; wherein the first pixel is one of the pixels in the set on the boundary; obtaining a correlation coefficient between the first region to be restored and the set of neighboring blocks; determining the priority of the first pixel based on the correlation coefficient, the confidence level of the first pixel, and the data items of the first pixel; determining the current region to be restored pixel and its corresponding target matching block based on the priority of each pixel in the set of pixels on the boundary, and restoring the region to be restored corresponding to the current region to be restored pixel based on the target matching block; wherein the current sample block size is obtained by updating the previous sample block size based on an initial sample block size or based on the information entropy of the target matching block in the previous restoration process.

[0156] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0157] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image restoration method, characterized in that, include: Determine the boundaries of the current region to be repaired in the first image; Determine the current sample block size; wherein, if the current round is the first image restoration, the current sample block size is obtained based on the initial sample block size; if the current round is not the first image restoration, the current sample block size is obtained by updating the sample block size of the previous round based on the information entropy of the target matching block in the previous restoration process. Based on the current sample block size, a first block to be repaired, centered on a first pixel, and a set of neighboring blocks adjacent to the first block to be repaired are generated; wherein, the first pixel is one of the set of pixels on the boundary; Obtain the correlation coefficient between the first block to be repaired and the set of neighboring blocks; The priority of the first pixel is determined based on the correlation coefficient, the confidence level of the first pixel, and the data items of the first pixel. Based on the priority of each pixel in the set of pixels on the boundary, determine the current pixel to be repaired; determine the target matching block corresponding to the current pixel to be repaired, and repair the block corresponding to the current pixel to be repaired based on the target matching block; The step of determining the target matching block corresponding to the current pixel to be repaired, and repairing the block corresponding to the current pixel to be repaired according to the target matching block, includes: Multiple image blocks are generated within a first distance range of the current pixel to be repaired, and the target matching block is determined by calculating the similarity between the multiple image blocks and the block to be repaired corresponding to the pixel to be repaired. Repair the block to be repaired corresponding to the current pixel to be repaired based on the target matching block; The first distance is updated based on the similarity or correlation coefficient between the target matching block and the block to be repaired corresponding to the pixel to be repaired; wherein the initial value of the first distance is determined based on the distribution characteristics of the region to be repaired in the first image.

2. The image restoration method according to claim 1, characterized in that, Determining the boundary of the current region to be repaired in the first image includes: The first image is input into the target detection model to determine the boundary of the current region to be repaired in the first image; wherein, the target detection model is a pre-trained neural network model, the feature fusion node of the target detection model is equipped with a channel attention mechanism module, and the number of output nodes and the output size of the target detection model are preset according to the size characteristics of the region to be repaired in the first image.

3. The image restoration method according to claim 1, characterized in that, The step of obtaining the correlation coefficient between the first block to be repaired and the set of neighboring blocks includes: The correlation coefficient is generated based on the variance of the first block to be repaired and the variance of each neighboring block in the neighborhood block set; wherein, the neighborhood block set consists of eight neighboring blocks adjacent to the first block to be repaired at the center grid that do not contain pixels in the area to be repaired, determined by a grid of the same size as the first block to be repaired.

4. The image restoration method according to claim 1, characterized in that, The correlation coefficient is obtained using the following formula: R(p) = ; Where R(p) represents the correlation coefficient between the first block to be repaired and the set of neighboring blocks, X represents the first block to be repaired, and Y represents a neighboring block in the set of neighboring blocks. The variance of the random variable X is represented by X. The variance of the random variable Y is represented. Let X represent the covariance of X and Y, and n be the number of neighborhood blocks in the neighborhood block set.

5. The image restoration method according to claim 1, characterized in that, The step of determining the priority of the first pixel based on the correlation coefficient, the confidence level of the first pixel, and the data items of the first pixel includes: The priority of the first pixel is obtained by weighted summation of the correlation coefficient, the confidence level of the first pixel, and the data item of the first pixel; wherein, the correlation coefficient represents the degree of correlation between the image at the first pixel and the images in the adjacent regions, the confidence level of the first pixel represents the proportion of known information contained in the image at the first pixel, and the data item represents the linear complexity of the image structure at the first pixel.

6. The image restoration method according to claim 1, characterized in that, Determining the current sample block size includes: If the current round is the first image restoration, the initial sample block size is obtained as the current sample block size; or, if the current round is not the first image restoration, ... Obtain the information entropy of the target matching block from the previous repair process; If the information entropy is less than the first threshold, increase the size of the current sample block; If the information entropy is greater than or equal to the first threshold and less than or equal to the second threshold, the current sample block size remains unchanged; If the information entropy is greater than the second threshold, reduce the current sample block size; The values ​​of the first threshold and the second threshold are in the range of (0, 1), and the first threshold is less than the second threshold.

7. The image restoration method according to claim 1, characterized in that, The step of updating the first distance based on the similarity or correlation coefficient between the target matching block and the block to be repaired corresponding to the pixel to be repaired includes: If the similarity is less than the third threshold, or the correlation coefficient is less than zero, the first distance is increased.

8. An image restoration device, characterized in that, include: The first determining module is used to determine the boundary of the current region to be repaired in the first image; The second determining module is used to determine the current sample block size; wherein, when the current round is the first image restoration, the current sample block size is obtained based on the initial sample block size; when the current round is not the first image restoration, the current sample block size is obtained by updating the sample block size of the previous round based on the information entropy of the target matching block in the previous restoration process. The generation module is used to generate a first block to be repaired centered on a first pixel, and a set of neighboring blocks adjacent to the first block to be repaired, based on the current sample block size; wherein, the first pixel is one of the set of pixels on the boundary; The acquisition module is used to acquire the correlation coefficient between the first block to be repaired and the set of neighboring blocks; The third determining module is used to determine the priority of the first pixel based on the correlation coefficient, the confidence level of the first pixel, and the data items of the first pixel. The repair module is used to determine the current pixel to be repaired based on the priority of each pixel in the set of pixels on the boundary; determine the target matching block corresponding to the current pixel to be repaired; and repair the repair block corresponding to the current pixel to be repaired based on the target matching block. The repair module is further configured to generate multiple image blocks within a first distance range of the current pixel to be repaired, determine a target matching block by calculating the similarity between the multiple image blocks and the corresponding block to be repaired of the pixel to be repaired, repair the block to be repaired corresponding to the current pixel to be repaired based on the target matching block, and update the first distance based on the similarity between the target matching block and the corresponding block to be repaired of the pixel to be repaired or the correlation coefficient; wherein, the initial value of the first distance is determined based on the distribution characteristics of the region to be repaired in the first image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements all or part of the steps of the image restoration method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements all or part of the steps of the image restoration method as described in any one of claims 1 to 7.

Images