Image inpainting method and device, electronic equipment and storage medium

Abstract

The application discloses an image repairing method and device, electronic equipment and storage medium. The image repairing method comprises the following steps: acquiring a to-be-processed image, extracting a to-be-processed tree structure corresponding to a target object in the to-be-processed image, and determining a to-be-processed connected component corresponding to the to-be-processed tree structure; determining the number of the to-be-processed connected components, determining a candidate connection weight between each two adjacent to-be-processed connected components in the case that the number is more than one, and determining a weight connection matrix corresponding to the to-be-processed tree structure based on the candidate connection weight; and repairing the to-be-processed tree structure according to the weight connection matrix to obtain a repaired tree structure, so as to obtain a repaired image. Based on the technical scheme of the embodiment of the application, the efficiency and accuracy of image repairing can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image repairing method and device, an electronic device and a storage medium. BACKGROUND

[0002] Based on the tree topology structure in the image, analyzing the image is a common image data analysis method. At present, a semantic segmentation model is usually used to extract the tree structure. However, the existing semantic segmentation model cannot guarantee the integrity of the extracted tree structure, that is, the extracted tree structure will usually have a disconnected node connection. Therefore, post-processing is needed to repair the disconnected node connection of the tree structure and guarantee the integrity of the tree structure.

[0003] The existing tree topology repairing method is an artificial repairing method. Specifically, the original image and the tree topology structure extracted by the semantic segmentation model are compared by artificial comparison to identify and manually repair the disconnected node connection. However, manual repair is time-consuming, and the repair result depends on the subjective judgment of the repairer and requires certain experience. Therefore, the tree structure in the image is usually repaired incorrectly. In summary, the efficiency and accuracy of the image repair in the prior art are poor. SUMMARY

[0004] The present application provides an image repairing method, device, electronic device and storage medium to solve the technical problem of poor efficiency and accuracy of image repair.

[0005] According to an aspect of the present application, an image repairing method is provided, wherein the method comprises:

[0006] An image to be processed is obtained, a tree structure to be processed corresponding to a target object in the image to be processed is extracted, and a connected component to be processed corresponding to the tree structure to be processed is determined.

[0007] The number of connected components to be processed is determined, and in the case that the number is more than one, a candidate connection weight between each two adjacent connected components to be processed is determined, and a weight connection matrix corresponding to the tree structure to be processed is determined based on the candidate connection weight.

[0008] The tree structure to be processed is repaired according to the weight connection matrix to obtain a repaired tree structure, so as to obtain a repaired image.

[0009] According to another aspect of the present application, an image repairing device is provided, wherein the device comprises:

[0010] The communication component determination module is configured to acquire a to-be-processed image, extract a to-be-processed tree structure corresponding to a target object in the to-be-processed image, and determine a to-be-processed communication component corresponding to the to-be-processed tree structure;

[0011] The matrix determination module is configured to determine the number of to-be-processed communication components, determine a candidate connection weight between each two adjacent to-be-processed communication components in a case where the number is more than one, and determine a weight connection matrix corresponding to the to-be-processed tree structure based on the candidate connection weight.

[0012] The image repairing module is configured to repair the to-be-processed tree structure according to the weight connection matrix, obtain a repaired tree structure, and obtain a repaired image.

[0013] According to another aspect of the present application, an electronic device is provided, which comprises:

[0014] at least one processor; and

[0015] a memory connected to the at least one processor in communication; wherein,

[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the image repairing method according to any one of the embodiments of the present application.

[0017] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to execute the image repairing method according to any one of the embodiments of the present application.

[0018] The technical scheme of the embodiments of the present application comprises the following steps: acquiring a to-be-processed image, extracting a to-be-processed tree structure corresponding to a target object in the to-be-processed image, and determining a to-be-processed communication component corresponding to the to-be-processed tree structure; determining the number of to-be-processed communication components, determining a candidate connection weight between each two adjacent to-be-processed communication components in a case where the number is more than one, and determining a weight connection matrix corresponding to the to-be-processed tree structure based on the candidate connection weight; and repairing the to-be-processed tree structure according to the weight connection matrix, obtaining a repaired tree structure, and obtaining a repaired image. The image repairing is based on the extracted communication component, which improves the efficiency of image repairing. The tree structure repairing is based on the determined weight connection matrix, which improves the accuracy of image repairing.

[0019] It is to be understood that the description of the present section is not intended to identify key or critical elements of the embodiments of the present application or to limit the scope of the present application. Other features of the present application will be apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort based on these drawings.

[0021] Figure 1 is a flow chart of an image repairing method according to the first embodiment of the present application;

[0022] Figure 2a is a scene diagram of a fundus image according to the embodiment of the present application;

[0023] Figure 2b is a scene diagram of an arterial blood vessel structure in a fundus image according to the embodiment of the present application;

[0024] Figure 2c is a scene diagram of a repaired arterial blood vessel structure according to the embodiment of the present application;

[0025] Figure 3a is a scene diagram in a lung CT image according to the embodiment of the present application;

[0026] Figure 3b is a scene diagram of a pulmonary artery structure in a lung CT image according to the embodiment of the present application;

[0027] Figure 3c is a scene diagram of a pulmonary artery structure to be repaired according to the embodiment of the present application;

[0028] Figure 3d is a scene diagram of a repaired pulmonary artery structure according to the embodiment of the present application;

[0029] Figure 4 is a flow chart of an image repairing method according to the second embodiment of the present application;

[0030] Figure 5 is a scene diagram of determining a reference connection weight according to the embodiment of the present application;

[0031] Figure 6 is a whole flow chart of an image repairing method according to the embodiment of the present application;

[0032] Figure 7 This is a schematic diagram of the structure of an image restoration device according to Embodiment 3 of the present invention;

[0033] Figure 8 This is a schematic diagram of the structure of an electronic device that implements the image restoration method of this invention. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] Example 1

[0037] Figure 1 The flowchart illustrates an image restoration method according to Embodiment 1 of the present invention. This embodiment is applicable to the case of spanning tree restoration. The method can be executed by an image restoration device, which can be implemented in hardware and / or software and can be configured within computer software. Figure 1 As shown, the method includes:

[0038] S110. Obtain the image to be processed, extract the tree structure to be processed corresponding to the target object in the image to be processed, and determine the connected components to be processed corresponding to the tree structure to be processed.

[0039] The image to be processed can be understood as an image to be repaired. Optionally, the image to be processed may be an image including a tree structure. In this embodiment of the invention, the image to be processed can be preset according to the needs of the scenario, and is not specifically limited here. For example, in a medical scenario, the image to be processed may be a fundus image including arterial vascular structures (see reference). Figure 2a ) or CT images of the lungs including the pulmonary artery structures (reference) Figure 3a ).

[0040] The target object can be understood as an object in the image to be processed that has a tree-like structure. Optionally, if the image to be processed is a fundus image, the target object can be an artery. If the image to be processed is a lung CT image, the target object can be the pulmonary artery.

[0041] The tree structure to be processed can be understood as the tree structure corresponding to the target object in the image to be processed. Optionally, if the image to be processed is a fundus image, the tree structure to be processed can be an arterial vascular structure. If the image to be processed is a lung CT image, the tree structure to be processed can be a pulmonary artery structure (see reference). Figure 2b and Figure 3b ).

[0042] The connected component to be processed can be understood as at least one connected component corresponding to the tree structure to be processed (see reference). Figure 2b and Figure 3c ).

[0043] Optionally, extracting the tree structure corresponding to the target object in the image to be processed and determining the connected components to be processed corresponding to the tree structure includes:

[0044] The skeleton of the target object in the image to be processed is extracted using the LEE algorithm to obtain the tree structure to be processed.

[0045] Traverse and mark each connected component in the tree structure to be processed to obtain the connected components to be processed corresponding to the tree structure to be processed.

[0046] Optionally, the entire tree structure to be processed can be traversed using a depth-first search algorithm, and each connected component in the tree structure to be processed can be marked to obtain the connected components to be processed corresponding to the tree structure to be processed.

[0047] In this embodiment of the invention, the Lee algorithm is used to extract the skeleton from the image to be processed. Based on the edge information of the image to be processed, a skeletalized thin-line structure can be constructed in the image to be processed to obtain the tree structure to be processed. This improves the efficiency of skeleton extraction and reduces redundant information while retaining the tree structure to be processed, thus ensuring the accuracy of the extracted tree structure to be processed.

[0048] S120. Determine the number of connected components to be processed. If the number is more than one, determine the candidate connection weights between every two adjacent connected components to be processed, and determine the weighted connection matrix corresponding to the tree structure to be processed based on the candidate connection weights.

[0049] The candidate connection weights can be understood as the connection weights between every two adjacent connected components to be processed. The weighted connection matrix can be understood as the matrix corresponding to the tree structure to be processed, composed of the candidate connection weights between every two adjacent connected components to be processed.

[0050] S130. Repair the tree structure to be processed according to the weighted connection matrix to obtain the repaired tree structure and thus obtain the repaired image.

[0051] The repaired tree structure can be understood as the tree structure obtained after repairing the tree structure to be processed according to the weighted connection matrix (see reference). Figure 2c and Figure 3d ).

[0052] The repaired image can be understood as the image corresponding to the repaired tree structure.

[0053] Optionally, the step of repairing the tree structure to be processed according to the weighted connection matrix to obtain a repaired tree structure, thereby obtaining a repaired image, includes:

[0054] The smallest candidate connection weight in the weighted connection matrix is ​​taken as the target connection weight;

[0055] Connect the two connected components to be processed corresponding to the target connection weight to obtain the repaired tree structure, and then obtain the repaired image corresponding to the repaired tree structure.

[0056] The target connection weight can be understood as the smallest candidate connection weight in the weighted connection matrix.

[0057] Optionally, after obtaining the repaired image corresponding to the repaired tree structure, the method further includes:

[0058] The operation involves determining the number of connected components to be processed in the repaired tree structure corresponding to the repaired image. If the number is more than one, the operation returns to determine the candidate connection weights between every two adjacent connected components to be processed. Based on the candidate connection weights, the weight connection matrix corresponding to the tree structure to be processed is determined. The tree structure to be processed is repaired according to the weight connection matrix to obtain the repaired tree structure and thus obtain the repaired image.

[0059] Until the number of unprocessed connected components of the repaired tree structure corresponding to the repaired image is one, the current repaired image is taken as the target image.

[0060] The target image can be understood to include the repaired image corresponding to the repaired tree structure of the connected component to be processed.

[0061] The technical solution of this invention involves acquiring an image to be processed, extracting the tree structure corresponding to the target object in the image, and determining the connected components to be processed corresponding to the tree structure. The number of connected components to be processed is determined; if the number is greater than one, candidate connection weights are determined between every two adjacent connected components; a weighted connection matrix corresponding to the tree structure to be processed is determined based on the candidate connection weights; and the tree structure to be processed is repaired according to the weighted connection matrix to obtain a repaired tree structure, thus obtaining a repaired image. Image repair based on the extracted connected components improves the efficiency of image repair, and tree structure repair based on the determined weighted connection matrix improves the accuracy of image repair.

[0062] Example 2

[0063] Figure 4 This is a flowchart of an image restoration method provided in Embodiment 2 of the present invention. This embodiment refines the determination of candidate connection weights between every two adjacent connected components to be processed as described in the above embodiments. Figure 4 As shown, the method includes:

[0064] S210. Obtain the image to be processed, extract the tree structure to be processed corresponding to the target object in the image to be processed, and determine the connected components to be processed corresponding to the tree structure to be processed.

[0065] S220. Determine the number of connected components to be processed, if the number is more than one.

[0066] S230. The two adjacent connected components to be processed are determined as the first connected component and the second connected component, and the pixels in the first connected component are determined as the first pixel and the pixels in the second connected component are determined as the second pixel.

[0067] Wherein, the first connected component and the second connected component are two currently adjacent connected components to be processed. The first pixel is a pixel in the first connected component. The second pixel is a pixel in the second connected component. It can be understood that the first connected component may include multiple first pixels, and the second connected component may include multiple second pixels.

[0068] S240. For the current first pixel and the current second pixel, determine the first dominant direction of the first pixel and the second dominant direction of the second pixel, and determine the reference connection weight between the first pixel and the second pixel based on the first dominant direction and the second dominant direction.

[0069] Here, the first dominant direction can be understood as the dominant direction of the first pixel.

[0070] The second dominant direction can be understood as the dominant direction of the second pixel.

[0071] The reference connection weight can be understood as the connection weight between the first pixel and the second pixel.

[0072] Optionally, determining the first dominant direction of the first pixel and the second dominant direction of the second pixel includes:

[0073] Obtain the first coordinates of the first pixel and the second coordinates of the second pixel;

[0074] The dominant directions of the first point and the second point are determined based on the first coordinate and the second coordinate, wherein the first coordinate and the second coordinate are multidimensional coordinates with the same dimension.

[0075] Here, the first coordinate can be understood as the coordinate of the current first pixel. In this embodiment of the invention, the dimension of the first coordinate can be set according to the scene requirements, and is not specifically limited here. For example, the first coordinate can be (x1, x2) or (x1, x2, ..., x...). n )wait.

[0076] The second coordinate can be understood as the coordinate of the current second pixel. In this embodiment of the invention, the dimension of the second coordinate can be set according to the scene requirements, and is not specifically limited here. For example, the second coordinate can be (y1, y2) or (y1, y2, ..., y2). n )wait.

[0077] Optionally, determining the dominant direction of the first point and the dominant direction of the second point based on the first coordinates and the second coordinates includes:

[0078] Using the first coordinates and the second coordinates, the dominant directions of the first point and the second point are determined in the following manner:

[0079] d1 = (y1-x1, y2-x2, ..., y n -x n )

[0080] d2=(x1-y1,x2-y2,…,x n -y n )

[0081] Where d1 represents the dominant direction of the first point, d2 represents the dominant direction of the second point, x1 represents the one-dimensional coordinates of the current first pixel, and x2 represents the two-dimensional coordinates of the current first pixel. n Let y1 represent the n-dimensional coordinates of the current first pixel, y2 represent the one-dimensional coordinates of the current second pixel, and y3 represent the two-dimensional coordinates of the current second pixel. n This represents the n-dimensional coordinates of the current second pixel.

[0082] Optionally, determining the reference connection weight between the first pixel and the second pixel based on the first dominant direction and the second dominant direction includes:

[0083] Determine candidate paths for the first pixel and the second pixel, and determine the Euclidean distance based on the candidate paths;

[0084] Determine the first angle between the dominant direction of the first point and the dominant direction of the second point, the second angle between the dominant direction of the first point and the candidate path, and the third angle between the dominant direction of the first point and the dominant direction of the second point;

[0085] The reference connection weight is determined based on the Euclidean distance, the first included angle, the second included angle, and the third included angle.

[0086] Wherein, the candidate path can be understood as a straight-line path between the first pixel and the second pixel. The Euclidean distance can be understood as the distance corresponding to the candidate path. The first included angle can be understood as the angle between the dominant direction of the first point and the dominant direction of the second point. The second included angle can be understood as the angle between the dominant direction of the first point and the candidate path. The third included angle can be understood as the angle between the dominant direction of the first point and the dominant direction of the second point. (Reference) Figure 5 Where L represents the candidate path, α represents the first included angle, θ1 represents the second included angle, and θ2 represents the third included angle.

[0087] S250. Determine multiple reference connection weights between each first pixel and each second pixel, and determine the smallest reference connection weight among the multiple reference connection weights as the candidate connection weight between the two adjacent connected components to be processed.

[0088] Optionally, for each first pixel and each second pixel, the reference connection weight is determined based on the Euclidean distance, the first included angle, the second included angle, and the third included angle, and the smallest reference connection weight among the plurality of reference connection weights is determined as the candidate connection weight between the two adjacent connected components to be processed. The specific calculation method is as follows:

[0089] w ij =min{||L||2(sin(α)+sin(θ1)+sin(θ2)+1)}

[0090] Among them, w ij Let L represent the candidate connection weights, ||L||2 represent the Euclidean distance, α represent the first included angle, θ1 represent the second included angle, and θ2 represent the third included angle.

[0091] S260. Determine the weighted connection matrix corresponding to the tree structure to be processed based on the candidate connection weights.

[0092] Optionally, when the number of connected components to be processed in the image to be processed is n, the weight connection matrix corresponding to the tree structure to be processed can be an n-dimensional adjacency matrix:

[0093]

[0094] Where W represents the weighted connection matrix, w 11 w represents the candidate connection weights between the first first connected component and the first second connected component. n1 w represents the candidate connection weights between the first first connected component and the nth second connected component. nnThis represents the candidate connection weights between the nth first connected component and the nth second connected component.

[0095] S270. Repair the tree structure to be processed according to the weighted connection matrix to obtain the repaired tree structure, thereby obtaining the repaired image.

[0096] The technical solution of this invention involves determining two adjacent connected components to be processed as a first connected component and a second connected component, defining pixels in the first connected component as first pixels and pixels in the second connected component as second pixels. For the current first pixel and the current second pixel, a first dominant direction for the first pixel and a second dominant direction for the second pixel are determined, and a reference connection weight between the first pixel and the second pixel is determined based on these directions. Multiple reference connection weights are determined between each first pixel and each second pixel, and the smallest of these reference connection weights is determined as the candidate connection weight between the two adjacent connected components to be processed. Determining the candidate connection weight based on the dominant direction improves the accuracy of the determined candidate connection weights between each pair of adjacent connected components to be processed, further ensuring the accuracy of the weighted connection matrix corresponding to the tree structure to be processed obtained based on the candidate connection weights.

[0097] Figure 6 This is an overall flowchart of an image restoration method provided according to an embodiment of the present invention. Figure 6 As shown, the overall process of the image restoration method can be as follows:

[0098] 1. Extracting the Tree Structure to be Processed. The Lee algorithm is used to extract the skeleton of the image to be processed, thus obtaining the tree structure. In this invention, the Lee algorithm is used to extract the skeleton of the image to be processed. Based on the edge information of the image to be processed, a skeletalized thin-line structure can be constructed in the image to obtain the tree structure. This method can efficiently extract important structures from the image to be processed, obtain the tree structure, reduce redundant information, and ensure the accuracy of the extracted tree structure.

[0099] 2. Mark the connected components to be processed. Using a depth-first search algorithm, traverse and mark each connected component in the tree structure to be processed to obtain the connected components to be processed corresponding to the tree structure.

[0100] 3. Calculating the Dominant Direction. It is understood that tree structures are typically formed through growth and branching processes, and their curvature is usually smooth due to biological and physical constraints. Therefore, this invention introduces a point-dominant direction to repair the tree structure being processed, improving the accuracy of tree structure repair and image restoration.

[0101] 4. Calculate candidate connection weights. Determine the candidate connection weights between every two adjacent connected components to be processed based on the Euclidean distance, the first included angle, the second included angle, and the third included angle.

[0102] 5. Repair the tree structure to be processed. If the number of connected components to be processed in the image to be processed is n, then the candidate connection weights between different connected components to be processed can form an n-dimensional adjacency matrix W; use the Bresenham algorithm to connect the two connected components to be processed corresponding to the smallest candidate connection weight in W to obtain the repaired image; for the repaired image, update the adjacency matrix W; until the repaired image includes one connected component to be processed, the target image is obtained.

[0103] This invention introduces a tree structure of the target object, transforming the problem of repairing breaks in the tree structure of the image into a problem of connecting connected components. By analyzing and utilizing the relationships between the connected components, the broken parts of the tree structure are effectively connected, thus achieving image restoration.

[0104] Based on the characteristic that the tree structure to be processed has a certain degree of extension in the current direction, this invention introduces the coordinates of the pixels to determine the dominant direction of the points, and uses the dominant direction of the points as a metric for the connection of the connected components to be processed, thereby improving the accuracy and stability of image restoration.

[0105] This invention is applicable to two-dimensional, three-dimensional and multi-dimensional images to be processed, and has a wider range of applicability.

[0106] Example 3

[0107] Figure 7 This is a schematic diagram of an image restoration device provided in Embodiment 3 of the present invention. Figure 7 As shown, the device includes: a connected component determination module 310, a matrix determination module 320, and an image restoration module 330; wherein,

[0108] The connected component determination module 310 is used to acquire the image to be processed, extract the tree structure to be processed corresponding to the target object in the image to be processed, and determine the connected components to be processed corresponding to the tree structure to be processed; the matrix determination module 320 is used to determine the number of connected components to be processed, and if the number is more than one, determine the candidate connection weights between every two adjacent connected components to be processed, and determine the weight connection matrix corresponding to the tree structure to be processed based on the candidate connection weights; the image restoration module 330 is used to restore the tree structure to be processed according to the weight connection matrix to obtain a restored tree structure, so as to obtain a restored image.

[0109] The technical solution of this invention involves acquiring an image to be processed, extracting the tree structure corresponding to the target object in the image, and determining the connected components to be processed corresponding to the tree structure. The number of connected components to be processed is determined; if the number is greater than one, candidate connection weights are determined between every two adjacent connected components; a weighted connection matrix corresponding to the tree structure to be processed is determined based on the candidate connection weights; and the tree structure to be processed is repaired according to the weighted connection matrix to obtain a repaired tree structure, thus obtaining a repaired image. Image repair based on the extracted connected components improves the efficiency of image repair, and tree structure repair based on the determined weighted connection matrix improves the accuracy of image repair.

[0110] Optionally, the matrix determination module 320 includes: a pixel determination unit, a pixel processing unit, and a candidate connection weight determination unit; wherein,

[0111] The pixel point determination unit is used to determine two adjacent connected components to be processed as a first connected component and a second connected component, and to determine the pixel point in the first connected component as a first pixel point and the pixel point in the second connected component as a second pixel point.

[0112] The pixel processing unit is configured to determine, for the current first pixel and the current second pixel, a first dominant direction of the first pixel and a second dominant direction of the second pixel, and to determine a reference connection weight between the first pixel and the second pixel based on the first dominant direction and the second dominant direction.

[0113] The candidate connection weight determination unit is used to determine multiple reference connection weights between each first pixel and each second pixel, and to determine the smallest reference connection weight among the multiple reference connection weights as the candidate connection weight between the two adjacent connected components to be processed.

[0114] Optionally, the pixel processing unit is used for:

[0115] Obtain the first coordinates of the first pixel and the second coordinates of the second pixel;

[0116] The dominant directions of the first point and the second point are determined based on the first coordinate and the second coordinate, wherein the first coordinate and the second coordinate are multidimensional coordinates with the same dimension.

[0117] Optionally, the pixel processing unit is used for:

[0118] Determine candidate paths for the first pixel and the second pixel, and determine the Euclidean distance based on the candidate paths;

[0119] Determine the first angle between the dominant direction of the first point and the dominant direction of the second point, the second angle between the dominant direction of the first point and the candidate path, and the third angle between the dominant direction of the first point and the dominant direction of the second point;

[0120] The reference connection weight is determined based on the Euclidean distance, the first included angle, the second included angle, and the third included angle.

[0121] Optionally, the image inpainting module 330 includes: a connection between the weight processing unit and the image inpainting unit; wherein,

[0122] The connection weight processing unit is used to take the smallest candidate connection weight in the weighted connection matrix as the target connection weight.

[0123] The image restoration unit is used to connect the two connected components to be processed corresponding to the target connection weight to obtain a restoration tree structure, so as to obtain the restoration image corresponding to the restoration tree structure.

[0124] Optionally, the image restoration module 330 further includes a cyclic restoration unit and a target image determination unit; wherein,

[0125] The loop repair unit is used to determine the number of connected components to be processed in the repair tree structure corresponding to the repair image after obtaining the repair image corresponding to the repair tree structure. If the number is more than one, it returns to the operation of determining the candidate connection weights between every two adjacent connected components to be processed, determining the weight connection matrix corresponding to the tree structure to be processed based on the candidate connection weights, and repairing the tree structure to be processed according to the weight connection matrix to obtain the repair tree structure and thus obtain the repair image.

[0126] The target image determination unit is used to take the current repaired image as the target image when the number of unprocessed connected components of the repaired tree structure corresponding to the repaired image is one.

[0127] Optionally, the connected component determination module 310 is used for:

[0128] The skeleton of the target object in the image to be processed is extracted using the LEE algorithm to obtain the tree structure to be processed.

[0129] Traverse and mark each connected component in the tree structure to be processed to obtain the connected components to be processed corresponding to the tree structure to be processed.

[0130] The image restoration apparatus provided in this embodiment of the invention can execute the image restoration method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0131] Example 4

[0132] Figure 8 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0133] like Figure 8 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0134] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0135] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as image inpainting methods.

[0136] In some embodiments, the image restoration method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the image restoration method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the image restoration method by any other suitable means (e.g., by means of firmware).

[0137] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0139] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0140] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0141] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0142] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0143] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0144] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An image restoration method, characterized in that, include: Obtain the image to be processed, extract the tree structure to be processed corresponding to the target object in the image to be processed, and determine the connected components to be processed corresponding to the tree structure to be processed. The number of connected components to be processed is determined. If the number is more than one, the two adjacent connected components to be processed are determined as the first connected component and the second connected component, and the pixels in the first connected component are determined as the first pixel and the pixels in the second connected component are determined as the second pixel. For the current first pixel and the current second pixel, determine the first dominant direction of the first pixel and the second dominant direction of the second pixel, and determine the reference connection weight between the first pixel and the second pixel based on the first dominant direction and the second dominant direction; Determine multiple reference connection weights between each first pixel and each second pixel, and determine the smallest reference connection weight among the multiple reference connection weights as the candidate connection weight between the two adjacent connected components to be processed. The weighted connection matrix corresponding to the tree structure to be processed is determined based on the candidate connection weights; The smallest candidate connection weight in the weighted connection matrix is ​​taken as the target connection weight; Connect the two connected components to be processed corresponding to the target connection weight to obtain the repaired tree structure, and then obtain the repaired image corresponding to the repaired tree structure.

2. The method according to claim 1, characterized in that, Determining the first dominant direction of the first pixel and the second dominant direction of the second pixel includes: Obtain the first coordinates of the first pixel and the second coordinates of the second pixel; The dominant directions of the first point and the second point are determined based on the first coordinate and the second coordinate, wherein the first coordinate and the second coordinate are multidimensional coordinates with the same dimension.

3. The method according to claim 1, characterized in that, Determining the reference connection weight between the first pixel and the second pixel based on the first dominant direction and the second dominant direction includes: Determine candidate paths for the first pixel and the second pixel, and determine the Euclidean distance based on the candidate paths; Determine the first angle between the dominant direction of the first point and the dominant direction of the second point, the second angle between the dominant direction of the first point and the candidate path, and the third angle between the dominant direction of the first point and the dominant direction of the second point; The reference connection weight is determined based on the Euclidean distance, the first included angle, the second included angle, and the third included angle.

4. The method according to claim 1, characterized in that, After obtaining the repaired image corresponding to the repaired tree structure, the method further includes: The operation involves determining the number of connected components to be processed in the repaired tree structure corresponding to the repaired image. If the number is more than one, the operation returns to determine the candidate connection weights between every two adjacent connected components to be processed. Based on the candidate connection weights, the weight connection matrix corresponding to the tree structure to be processed is determined. The tree structure to be processed is repaired according to the weight connection matrix to obtain the repaired tree structure and thus obtain the repaired image. Until the number of unprocessed connected components of the repaired tree structure corresponding to the repaired image is one, the current repaired image is taken as the target image.

5. The method according to claim 1, characterized in that, The step of extracting the tree structure corresponding to the target object in the image to be processed and determining the connected components to be processed corresponding to the tree structure includes: The skeleton of the target object in the image to be processed is extracted using the LEE algorithm to obtain the tree structure to be processed. Traverse and mark each connected component in the tree structure to be processed to obtain the connected components to be processed corresponding to the tree structure to be processed.

6. An image restoration device, characterized in that, include: The connected component determination module is used to acquire the image to be processed, extract the tree structure to be processed corresponding to the target object in the image to be processed, and determine the connected components to be processed corresponding to the tree structure to be processed. The matrix determination module is used to determine the number of connected components to be processed. If the number is more than one, it determines the candidate connection weights between every two adjacent connected components to be processed, and determines the weight connection matrix corresponding to the tree structure to be processed based on the candidate connection weights. The image inpainting module is used to repair the tree structure to be processed according to the weighted connection matrix to obtain a repaired tree structure, thereby obtaining a repaired image. The matrix determination module includes: A pixel point determination unit is used to determine two adjacent connected components to be processed as a first connected component and a second connected component, and to determine the pixel point in the first connected component as the first pixel point and the pixel point in the second connected component as the second pixel point. A pixel processing unit is configured to determine, for the current first pixel and the current second pixel, a first point dominance direction of the first pixel and a second point dominance direction of the second pixel, and to determine a reference connection weight between the first pixel and the second pixel based on the first point dominance direction and the second point dominance direction. A candidate connection weight determination unit is used to determine multiple reference connection weights between each first pixel and each second pixel, and to determine the smallest reference connection weight among the multiple reference connection weights as the candidate connection weight between the two adjacent connected components to be processed. The image restoration module includes: A connection weight processing unit is used to take the smallest candidate connection weight in the weight connection matrix as the target connection weight. An image inpainting unit is used to connect the two connected components to be processed corresponding to the target connection weight to obtain a repaired tree structure, so as to obtain the repaired image corresponding to the repaired tree structure.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the image restoration method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the image restoration method according to any one of claims 1-5.

Images