Image restoration method and electronic equipment

Abstract

The invention provides an image restoration method and electronic equipment, and relates to the technical field of images, in particular to the technical field of image restoration. According to the specific implementation scheme, the method comprises the steps of determining a to-be-filled image block containing a first pixel region and a second pixel region in a to-be-restored image, determining an image block with the highest matching degree with the to-be-filled image block based on a complete region in the to-be-restored image and a reference image, and filling the first pixel region based on the image block. The first pixel region comprises a part of pixels in a to-be-restored region of the to-be-restored image, the second pixel region comprises a part of pixels in a complete region of the to-be-restored image, the size of the image block is the same as that of the to-be-filled image block, and the reference image and the to-be-restored image are images shot for the same object. Through the method, the to-be-restored image can be restored, and the phenomena of large-area bar-shaped artifacts and black holes of the reconstructed image are effectively reduced.

Description

Technical Field

[0001] This disclosure relates to the field of image technology, and more particularly to the field of image restoration technology, specifically to an image restoration method and an electronic device. Background Technology

[0002] During cone beam computed tomography (CBCT) image acquisition, image data loss frequently occurs due to equipment hardware failures or environmental interference. This data loss can lead to large-area stripe artifacts and black holes in the reconstructed images.

[0003] Therefore, how to repair missing image data is an urgent problem to be solved. Summary of the Invention

[0004] This disclosure provides an image restoration method and electronic device that can effectively reduce the occurrence of large-area stripe artifacts and black holes in the reconstructed image.

[0005] In a first aspect, this disclosure provides an image restoration method, the method comprising:

[0006] In the image to be repaired, identify the image block to be filled, which contains the first pixel region and the second pixel region. Based on the complete region in the image to be repaired and the reference image, identify the image block with the highest matching degree to the image block to be filled. Based on the image block, fill the first pixel region.

[0007] The first pixel region includes a portion of the pixels in the region to be repaired of the image to be repaired, and the second pixel region includes a portion of the pixels in the complete region of the image to be repaired. The size of the image block is the same as the size of the image block to be filled. The reference image and the image to be repaired are images taken of the same object.

[0008] In some embodiments, determining the image block with the highest matching degree to the image block to be filled based on the complete region in the image to be repaired and the reference image may specifically include: determining multiple candidate image blocks from the complete region in the image to be repaired and the reference image, and determining the image block with the highest matching degree to the image block to be filled from the multiple candidate image blocks based on the similarity between each candidate image block and the image block to be filled.

[0009] In some embodiments, the above-mentioned method of determining the image block with the highest matching degree to the image block to be filled from multiple candidate image blocks based on the similarity between each candidate image block and the image block to be filled can specifically include: determining the matching degree between each candidate image block and the image block to be filled based on the similarity between each candidate image block and the image block to be filled, and the weight corresponding to the image containing the candidate image block; and determining the image block with the highest matching degree to the image block to be filled from multiple candidate image blocks based on the matching degree between each candidate image block and the image block to be filled.

[0010] In some embodiments, the similarity between the candidate image patch and the image patch to be filled can be measured by at least one of pixel intensity or gradient distribution.

[0011] In some embodiments, the plurality of candidate image blocks may include a first candidate image block and a second candidate image block. The first candidate image block is an image block within the complete region of the image to be repaired, and the second candidate image block is an image block in the reference image. Based on this, the determination of the plurality of candidate image blocks from the complete region of the image to be repaired and the reference image may specifically include: determining texture boundaries in the complete region; based on the texture boundaries of the complete region, determining a first candidate image block from the complete region whose similarity to the image block to be filled is greater than a threshold; determining a third pixel region from the reference image; and based on the third pixel region, determining a second candidate image block from the reference image.

[0012] In this context, the third pixel region is located in the same position in the reference image as the image block to be filled in the image to be repaired. Texture boundaries refer to the boundaries between different texture regions.

[0013] In some embodiments, the number of texture regions contained in the first candidate image block, the second candidate image block, and the image block to be filled are the same.

[0014] In some embodiments, the above-described determination of the image block to be filled in the image to be repaired may specifically include: determining the boundary line between the region to be repaired and the complete region in the image to be repaired, and determining the image block to be filled in the image to be repaired based on the weight of the boundary points located on the boundary line.

[0015] In some embodiments, the above-mentioned determination of the boundary line between the region to be repaired and the complete region in the image to be repaired may specifically include: determining multiple boundary points from the image to be repaired based on the degree of change of gray values ​​around each pixel in the image to be repaired and the direction of the largest change of gray values ​​around each pixel, and connecting the multiple boundary points to obtain the boundary line.

[0016] In some embodiments, the aforementioned boundary points may include multiple boundary points, each of which may correspond to a candidate region. Based on this, the aforementioned determination of the image block to be filled in the image to be repaired based on the boundary points located on the boundary line may specifically include: for each boundary point, determining the weight of the boundary point based on the number of pixels in the region to be repaired in the candidate region corresponding to the boundary point and the number of texture regions contained in the candidate region corresponding to the boundary point, and taking the candidate region corresponding to the boundary point with the highest weight as the image block to be filled.

[0017] In some embodiments, the image to be repaired may be a first projection image of the target object under the current fractionation radiotherapy, and the reference image may include a second projection image of the target object under the current fractionation radiotherapy, or a third projection image of the target object under a historical fractionation radiotherapy.

[0018] The first and second projected images have different imaging angles, and the first and third projected images may have the same or different imaging angles.

[0019] Secondly, this disclosure also provides an image restoration apparatus, the apparatus comprising:

[0020] The first determining unit is used to determine an image block to be filled in the image to be repaired. The image block to be filled includes a first pixel region and a second pixel region. The first pixel region includes a portion of pixels in the region to be repaired of the image to be repaired, and the second pixel region includes a portion of pixels in the complete region of the image to be repaired.

[0021] The second determining unit is used to determine the image block with the highest matching degree to the image block to be filled, based on the complete region and the reference image. The size of the image block is the same as the size of the image block to be filled, and the reference image and the image to be repaired are images taken for the same object.

[0022] A fill unit is used to fill the first pixel region based on an image block.

[0023] Thirdly, this disclosure also provides an electronic device comprising: a processor and a memory configured to store processor-executable instructions; wherein the processor is configured to execute the instructions to implement any of the optional image restoration methods described in the first aspect above.

[0024] The above technical solution allows for the filling of image blocks containing the area to be repaired within the image to be repaired, based on the complete region of the image to be repaired and a reference image. This effectively reduces the occurrence of large-area stripe artifacts and black holes in the reconstructed image. Furthermore, this solution can determine similar image blocks not only based on the image to be repaired but also based on the reference image. This effectively expands the range of non-missing regions, thereby increasing the probability of identifying similar image blocks and improving the accuracy of repairing the area to be repaired in the image to be repaired. Attached Figure Description

[0025] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0026] Figure 1 This is a schematic diagram of a scene for an image restoration system provided in an embodiment of this disclosure;

[0027] Figure 2 A schematic diagram of an image to be repaired provided in an embodiment of this disclosure;

[0028] Figure 3 A flowchart of an image restoration method provided in this disclosure embodiment;

[0029] Figure 4 A schematic diagram illustrating the determination of an image block to be filled, provided as an embodiment of this disclosure;

[0030] Figure 5 This is a schematic diagram illustrating the filling of an image block to be filled, as provided in an embodiment of this disclosure.

[0031] Figure 6 A flowchart illustrating another image restoration method provided in this disclosure embodiment;

[0032] Figure 7 This is a schematic block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0033] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0034] In the description of this disclosure, it should be understood that the terms "center," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing this disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this disclosure. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," or "third" may explicitly or implicitly include one or more of the stated features. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly and specifically defined.

[0035] In the description of this disclosure, the term "exemplary" is used to mean "serving as an example, illustration, or description." Any embodiment described as "exemplary" in this disclosure is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use this disclosure. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that this disclosure can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of this disclosure with unnecessary detail. Therefore, this disclosure is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed herein.

[0036] It should be noted that since the method of this embodiment is executed in an electronic device, the processing objects of each electronic device exist in the form of data or information, such as time, which is essentially time information. It is understood that if size, quantity, position, etc. are mentioned in subsequent embodiments, they are all corresponding data that exist so that the computer device can process them. Specific details will not be elaborated here.

[0037] During CBCT image acquisition, image data loss frequently occurs due to equipment hardware malfunctions or environmental interference. This data loss can lead to large-area stripe artifacts and black holes in the reconstructed images.

[0038] Taking scattering correction using a beam stopping array (BSA) as an example, during image acquisition, the BSA's blocking of rays prevents the detector from receiving projection data for the corresponding area. The structural information of the blocked area is lost during the acquisition phase, making it impossible to reconstruct the lost structural information using conventional data restoration methods such as interpolation algorithms.

[0039] Therefore, how to repair missing image data is an urgent problem to be solved.

[0040] Based on the above-mentioned technical problems, this disclosure provides an image restoration method, which can determine an image block to be filled in the image to be restored, which includes a first pixel region and a second pixel region, determine the image block with the highest matching degree with the image block to be filled based on the complete region in the image to be restored and a reference image, and fill the first pixel region based on the image block.

[0041] The first pixel region includes a portion of the pixels in the region to be repaired of the image to be repaired, and the second pixel region includes a portion of the pixels in the complete region of the image to be repaired. The size of the image block is the same as the size of the image block to be filled. The reference image and the image to be repaired are images taken of the same object.

[0042] The above technical solution allows for the filling of image blocks containing the areas to be repaired within the image to be repaired, based on the complete region of the image to be repaired and a reference image, thereby achieving image repair. This effectively reduces the occurrence of large-area stripe artifacts and black holes in the reconstructed image. Furthermore, this solution can determine similar image blocks not only based on the image to be repaired but also based on the reference image. This effectively expands the range of non-missing regions, thereby increasing the probability of identifying similar image blocks and improving the accuracy of repairing the areas to be repaired in the image to be repaired.

[0043] The image restoration method provided in this disclosure can be applied to various scenarios such as medical imaging and computer vision. The following description uses the application of the image restoration method in medical imaging as an example to illustrate the scenario of the image restoration method provided in this disclosure.

[0044] Figure 1 This is a schematic diagram of an exemplary image restoration system, which may include an image-guided radiotherapy device 101 and an imaging computer device 102.

[0045] Image-guided radiotherapy equipment 101 may include a gantry 103 and an image guiding device mounted on the gantry 103. The image guiding device includes an imaging source 104, a blocking device 105, and a detector 106. The cone-shaped imaging beam emitted by the imaging source 104 passes through the blocking device 105 and is received by the detector 106, generating a two-dimensional projection image of the target object.

[0046] In this embodiment, the occlusion device 105 is located between the imaging source 104 and the detector 106. During the scattering correction process of the imaging beam emitted by the imaging source 104, the occlusion device 105 will block part of the imaging beam, causing data loss in the two-dimensional image generated by the detector 106. That is, the two-dimensional image generated by the detector 106 is a two-dimensional image containing the area to be repaired (hereinafter referred to as the image to be repaired). Figure 2 The image shown in (a) has an area to be repaired. Figure 2 The blank area in region 201 shown in (a) of the diagram.

[0047] In this embodiment of the disclosure, the shape of the blocking device 105 is not specifically limited. For example, the blocking device 105 can be a BSA or a grating collimator, etc.

[0048] The detector 106 can be a flat panel detector or a curved surface detector. The present disclosure does not specifically limit the shape of the detector 106.

[0049] Image guidance devices may include CBCT devices. This disclosure does not specifically limit the form of the image guidance device.

[0050] When the image guidance device is a CBCT device, the imaging source 104 can be an X-ray tube, and the detector 106 can be a flat panel detector.

[0051] The frame 103 can be a ring frame, a C-arm frame, a drum frame, a multi-layer bowl / cylindrical structure frame, etc. The frame 103 can be a rotating frame that can move around the rotation axis or a fixed frame that cannot move.

[0052] The image computing device 102 is communicatively connected to the detector 106. Accordingly, after the detector 106 generates an image to be repaired, it can send the image to be repaired to the image computing device 102. In this way, the image computing device 102 can receive the image to be repaired.

[0053] In some embodiments, the imaging computing device 102 may be a computer device with a graphical user interface (GUI), which may include one or more processors, memory, and one or more application programs. For example, the processor of the imaging computing device 102 may perform the following: determining an image block to be filled in the image to be repaired, containing a first pixel region and a second pixel region; determining an image block with the highest matching degree to the image block to be filled based on the complete region in the image to be repaired and a reference image; and filling the first pixel region based on the image block. The first pixel region includes a portion of pixels in the region to be repaired in the image to be repaired. The second pixel region includes a portion of pixels in the complete region of the image to be repaired. The size of the image block is the same as the size of the image block to be filled. The reference image and the image to be repaired are images taken of the same object.

[0054] For example, taking the image to be repaired as... Figure 2 The image shown in (a) has an area to be repaired. Figure 2 Taking the blank area in region 201 shown in (a) as an example, the image computer device 102 can be based on Figure 2 The blank areas in region 201 of the image shown in (a) are filled with the complete region of the image and the reference image to obtain the result. Figure 2 Image (b) is shown in the image.

[0055] In this embodiment of the disclosure, the imaging computer device 102 can be a standalone server, or a server network or server cluster composed of servers. For example, the imaging computer device described in this embodiment of the disclosure includes, but is not limited to, a computer, a network host, a single network server, a set of multiple network servers, or a cloud server composed of multiple servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.

[0056] In this embodiment, the imaging computer device 102 can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device can be a desktop computer, a portable computer, a network server, a PDA (Personal Digital Assistant), a mobile phone, a tablet computer, a wireless terminal device, a communication device, an embedded device, etc. This embodiment does not limit the type of computer device.

[0057] The image restoration method provided in this disclosure is described below using BSA (Browser-Assisted Image Processing) as an example, with both the image computer device and the occlusion device being BSA. Figure 3 This is a flowchart of an image restoration method provided in an embodiment of the present disclosure. The executing entity can be an image computer device, such as... Figure 3As shown, the method includes:

[0058] S301, Identify the image block to be filled in the image to be repaired.

[0059] The image block to be filled contains a first pixel region and a second pixel region. The first pixel region includes a portion of pixels from the area to be repaired (also called the damaged area) of the image to be repaired, and the second pixel region includes a portion of pixels from the intact area (also called the undamaged area) of the image to be repaired. In one example, such as... Figure 4 As shown in (a), the image to be repaired 401 includes a region to be repaired, a, and a complete region, b. Correspondingly, the image block to be filled may include a portion of pixels from the region to be repaired, a portion of pixels from the complete region, a portion of pixels from the complete region, a portion of pixels from the complete region, a second pixel region. In one example, refer to... Figure 4 The image patch to be filled, as shown in (a) of the image to be repaired, can be referenced. Figure 4 Image block 402 is shown in (b) of the image.

[0060] In the embodiments of this disclosure, the size and shape of the image block to be filled are not specifically limited. For example, the shape of the image block to be filled can be a square with a side length of 1 mm, a square with a side length of 5 mm, a circle with a radius of 2 mm, or a circle with a radius of 3 mm, etc.

[0061] In this embodiment of the disclosure, the method by which the imaging computer device determines the region to be repaired in the image to be repaired is not specifically limited. In one example, the region to be repaired in the image to be repaired may be manually marked by a user (such as a healthcare worker). In another example, the region to be repaired in the image to be repaired may be automatically determined by the imaging computer device. Specifically, after receiving the image to be repaired, the imaging computer device may determine the region to be repaired in the image to be repaired based on the prior information of BSA.

[0062] This disclosure does not specifically limit the prior information of the BSA. For example, the prior information of the BSA may include, but is not limited to, the shape and arrangement of the occlusion units in the BSA, the geometric parameters of the occlusion units (such as the size and spacing of the occlusion units), and the position of the BSA (such as the distance between the BSA and the imaging source, the distance between the BSA and the detector).

[0063] In this embodiment of the disclosure, the method by which the image computer device determines the complete region in the image to be repaired is not specifically limited. In one example, the complete region in the image to be repaired may be manually marked by the user. In another example, the complete region in the image to be repaired may be automatically determined by the image computer device. Specifically, after the image computer device determines the region to be repaired in the image to be repaired in the above manner, it may determine the remaining regions in the image to be repaired as the complete region in the image to be repaired.

[0064] After the image computer equipment determines the region to be repaired and the complete region in the image to be repaired in the above manner, it can determine the image block to be filled from the image to be repaired based on the region to be repaired and the complete region in the image to be repaired.

[0065] S302, based on the complete region and the reference image, determine the image block with the highest matching degree to the image block to be filled.

[0066] The size and shape of the image patch are the same as those of the image patch to be filled. For example, if the image patch to be filled is a 1mm*1mm square, the image patch that best matches the image patch to be filled is also a 1mm*1mm square.

[0067] The reference image and the image to be repaired are both images taken of the same object.

[0068] In the embodiments disclosed herein, the reference image is not specifically limited. For example, when the image to be repaired is a projection image of the target object under the current fractionation radiotherapy (hereinafter referred to as the first projection image, or may also be a kilovolt (KV) image), the reference image may include the other projection images of the target object under the current fractionation radiotherapy besides the first projection image (hereinafter referred to as the second projection image), or projection images of the target object under historical fractionation radiotherapy (hereinafter referred to as the third projection image).

[0069] The imaging angles of the first and second projected images can be different. In one example, the imaging angles of the first and second projected images can be adjacent. For example, the imaging angle of the first projected image can be 90 degrees, and the imaging angle of the second projected image can be 89 degrees.

[0070] In this embodiment, the imaging angle of the third projected image is not specifically limited. In one example, the imaging angles of the first and third projected images can be the same. For example, the imaging angles of both the first and third projected images are 90 degrees. In another example, the imaging angles of the first and third projected images can be different. For example, taking the imaging angles of the first and third projected images as adjacent imaging angles, the imaging angle of the first projected image can be 90 degrees, and the imaging angle of the third projected image can be 89 degrees.

[0071] After identifying the image patch to be filled from the image to be repaired, the image computer equipment can determine multiple candidate image patches from the complete region of the image to be repaired and the reference image. Then, the image computer equipment can determine the similarity between each candidate image patch and the image patch to be filled, and based on the similarity between each candidate image patch and the image patch to be filled, determine the image patch with the highest matching degree from the multiple candidate image patches.

[0072] For example, suppose the imaging computer device determines multiple candidate image patches including Figure 5 Image blocks 501 and 502 are shown in (a) above, and the similarity between image block 501 and the image block to be filled is greater than the similarity between image block 502 and the image block to be filled. The image computing device may use image block 501 as the image block with the highest matching degree to the image block to be filled.

[0073] S303, based on image blocks, fills the first pixel region.

[0074] After the image computer device determines the image block with the highest matching degree to the image block to be filled in the above manner, it can use the area in the image block that corresponds to the first pixel area in the image block to be filled to fill the first pixel area.

[0075] For example, the image patch with the highest matching degree to the image patch to be filled is used as... Figure 5 Image block 501 shown in (a) is the image block to be filled. Figure 4 (b) or Figure 5 Taking image block 402 as shown in (a) as an example, the image computer device can use the area in image block 501 that corresponds to the first pixel area in image block 402 to fill the first pixel area in image block 402 (e.g., Figure 5 (a) shows the blank area in image block 402, and obtains Figure 5 Image (b) is shown in the image.

[0076] Using the above method, image processing equipment can identify similar image blocks (i.e., the image blocks with the highest matching degree to the image block to be filled) in non-missing regions (such as the complete region and reference image mentioned above), and repair the areas to be repaired in the image to be repaired, effectively reducing the phenomenon of large-area stripe artifacts and black holes in the reconstructed image. Furthermore, in this scheme, similar image blocks can be identified not only based on the image to be repaired but also based on the reference image. This effectively expands the range of non-missing regions, thereby significantly increasing the probability of identifying similar image blocks and improving the accuracy of repairing the areas to be repaired in the image to be repaired.

[0077] The following will describe embodiments related to step S301 described above.

[0078] Since the area to be repaired in the image to be repaired contains little (almost no) information for reference, the image computing device can repair the area to be repaired starting from the edge of the area to be repaired, combining information from the entire area of ​​the image to be repaired. In an optional implementation, the above-mentioned determination of the image block to be filled in the image to be repaired may specifically include: determining the boundary line between the area to be repaired and the complete area in the image to be repaired, and determining the image block to be filled in the image to be repaired based on the weights of boundary points located on the boundary line.

[0079] The following describes the process by which image computer equipment determines the boundary lines between the region to be repaired and the complete region in an image to be repaired, using two different methods.

[0080] Method 1

[0081] After the image computer device determines the region to be repaired and the complete region in the image to be repaired in the manner described in S301, it can use the boundary line between the region to be repaired and the complete region in the image to be repaired as the boundary line between the region to be repaired and the complete region.

[0082] Method 2

[0083] Image processing equipment can determine multiple target boundary points in the image to be repaired based on the degree of change in grayscale values ​​around each pixel and the direction of the greatest change in grayscale values ​​around each pixel. Then, the image processing equipment can connect these multiple target boundary points to obtain the boundary lines between the area to be repaired and the complete area.

[0084] In one example, the degree of change in grayscale values ​​around a pixel can be characterized by the pixel's gradient value (also known as the gradient at the pixel). The direction of the greatest change in grayscale values ​​around a pixel can be characterized by the pixel's gradient direction. The direction perpendicular to the gradient direction can be defined as the boundary normal vector, which points in the direction of the most gradual change in grayscale values. After determining the gradient value and gradient direction of each pixel in the image to be repaired, the image processing equipment can identify pixels with gradient values ​​greater than a preset gradient value threshold as potential boundary points, and the direction perpendicular to the gradient direction of each pixel as the boundary normal vector of that pixel. For each potential boundary point, the image processing equipment can check whether the boundary normal vectors of the pixels surrounding that potential boundary point are consistent. The image processing equipment can identify potential boundary points with consistent boundary normal vectors of surrounding pixels as target boundary points. The image processing equipment can connect multiple target boundary points to obtain the boundary lines between the area to be repaired and the complete area.

[0085] Consistency of boundary normal vectors of each pixel means that the direction of the boundary normal vectors of each pixel is the same, or the angle between the boundary normal vectors of each pixel is less than a preset angle threshold.

[0086] This disclosure does not specifically limit the method for determining the gradient value of each pixel. For example, gradient operators such as the Sobel operator, Prewitt operator, or Canny operator can be used to determine the gradient value of each pixel.

[0087] The method for determining the gradient direction of each pixel can be found in relevant technologies, and will not be elaborated here.

[0088] After determining the boundary lines using the aforementioned method, the image processing equipment identifies a candidate region centered on each boundary point located on the boundary lines within the image to be repaired. Based on the number of pixels within the candidate region corresponding to that boundary point that are located within the image to be repaired, and the number of texture regions contained within that candidate region, the equipment determines the weight of that boundary point. Subsequently, the image processing equipment can use the candidate region corresponding to the boundary point with the highest weight as the image patch to be filled.

[0089] The number of pixels in the candidate region corresponding to a boundary point that are located in the region to be repaired is negatively correlated with the weight of the boundary point. That is, the more pixels in the candidate region corresponding to a boundary point that are located in the region to be repaired, the lower the weight of the boundary point; the fewer pixels in the candidate region corresponding to a boundary point that are located in the region to be repaired, the higher the weight of the boundary point.

[0090] The number of texture regions contained in the candidate region corresponding to a boundary point is positively correlated with the weight of the boundary point. That is, the more texture regions contained in the candidate region corresponding to a boundary point, the higher the weight of the boundary point; the fewer texture regions contained in the candidate region corresponding to a boundary point, the lower the weight of the boundary point.

[0091] In one example, the number of textured regions contained in the candidate region corresponding to a boundary point can be represented by the dot product of the boundary point's gradient value and the boundary normal vector. A larger dot product indicates a larger number of textured regions in the candidate region, and a smaller dot product indicates a smaller number of textured regions. Accordingly, for each boundary point located on the boundary line, the image computing device can determine a candidate region centered on that boundary point based on a preset shape and size. Then, the image computing device can represent the number of textured regions contained in the candidate region based on the dot product of the boundary point's gradient value and the boundary normal vector. Next, the image computing device can determine the weight of the boundary point based on the number of pixels in the candidate region corresponding to the boundary point located in the region to be repaired, as well as the dot product of the boundary point's gradient value and the boundary normal vector. The image computing device can then use the candidate region corresponding to the boundary point with the highest weight as the image patch to be filled.

[0092] The method by which image computing devices determine the number of pixels in the candidate region that are located in the region to be repaired can be found in relevant technologies, and will not be elaborated here.

[0093] In another example, after the image computing device identifies the candidate region corresponding to each boundary point on the boundary line, it can determine the proportion of pixels in the candidate region that are located in the complete region of the image to be repaired (also known as the confidence level of the region to be repaired) based on the number of pixels in the candidate region corresponding to that boundary point and the number of pixels in that candidate region that are located in the complete region of the image to be repaired. The image computing device can determine the confidence level of each pixel in the region to be repaired based on the confidence level of the region to be repaired and the number of pixels in the region to be repaired. Then, the image computing device can determine the weight of each boundary point based on the confidence level of each pixel in the region to be repaired corresponding to each boundary point, as well as the dot product of the gradient value and the boundary normal vector of that boundary point, and select the candidate region corresponding to the boundary point with the highest weight as the image patch to be filled.

[0094] For example, suppose the boundary point located on the boundary line is point p. The gradient value of point p is represented by gradient value (p), the weight of point p is represented by P(p), the boundary normal vector of point p is represented by M(p), the dot product of the gradient value and the boundary normal vector of point p is represented by D(p), and the confidence of each pixel in the candidate region to be repaired corresponding to point p is represented by C(p). The image computing device can determine the dot product of the gradient value and the boundary normal vector of point p, D(p), using the following formula 1; determine the confidence of each pixel in the candidate region to be repaired corresponding to point p, C(p), using the following formula 2; and determine the weight of point p, P(p), using the following formula 3.

[0095] D(p) = gradient value(p)·M(p) (Formula 1)

[0096]

[0097] P(p)=D(p)*C(p) (Formula 3)

[0098] After determining the weight of each boundary point on the boundary line using the above method, the image computer equipment can use the candidate region corresponding to the boundary point with the highest weight as the image block to be filled.

[0099] The following will describe embodiments related to step S302 described above.

[0100] In one alternative implementation, the plurality of candidate image blocks may include an image block located in the complete region of the image to be repaired (hereinafter referred to as the first candidate image block) and an image block located in the reference image (hereinafter referred to as the second candidate image block). Accordingly, the above-described determination of a plurality of candidate image blocks from the complete region of the image to be repaired and the reference image may be replaced by determining the first candidate image block from the complete region of the image to be repaired and determining the second candidate image block from the reference image.

[0101] The process of determining the first candidate image block and the second candidate image block by the image computer device will be described below.

[0102] 1. Determine the first candidate image block.

[0103] The image computing device can determine the texture boundary line in the complete region, and based on the texture boundary line of the complete region, determine the first candidate image patch from the complete region whose similarity to the image patch to be filled is greater than a threshold.

[0104] Here, the texture boundary line refers to the boundary line between different texture regions. In one example, continuing to refer to... Figure 4As shown in (a), the complete region b contains two distinct regions, region c and region d. Therefore, the texture boundary line refers to the boundary line between region c and region d, i.e. Figure 4 The boundary line m is shown in (a) in the diagram.

[0105] Specifically, after determining the texture boundary line in the complete region, the image computer device can identify multiple image blocks (hereinafter referred to as image blocks A) with the same size and shape as the image block to be filled, along the texture boundary line in the complete region. Then, the image computer device can determine the similarity between each image block A and the image block to be filled, and select image blocks A with a similarity greater than a threshold as the first candidate image blocks.

[0106] This disclosure does not specifically limit the method for determining the similarity between each image block A and the image block to be filled. In one example, the image computing device can determine the similarity between each image block A and the image block to be filled using methods such as Fourier transform and wavelet transform. In another example, the image computing device can determine the similarity between image block A and the image block to be filled based on the pixel intensity and / or gradient distribution of each image block A. Here, pixel intensity is used to characterize the grayscale value of each pixel in the image block, and gradient distribution is used to characterize the structural information of the image block.

[0107] The specific process of determining the similarity between image block A and the image block to be filled can be found in relevant technologies, and will not be elaborated here.

[0108] 2. Determine the second candidate image block.

[0109] The image computing device can determine a third pixel region from a reference image and, based on the third pixel region, determine a second candidate image block from the reference image.

[0110] The third pixel region is located in the same position in the reference image as the image block to be filled is located in the image to be repaired.

[0111] Specifically, after the image computing device determines the image block to be filled, it can further determine the position of the image block to be filled in the image to be repaired (hereinafter referred to as position A). The image computing device can use the pixel region located at position A in the reference image as the third pixel region. Then, the image computing device can use the third pixel region as the second candidate image block. And / or, the image computing device can determine multiple image blocks (hereinafter referred to as image blocks B) with the same size and shape as the image block to be filled within a preset range of the third pixel region. Then, the image computing device can determine the similarity between each image block B and the image block to be filled, and use the image block B with a similarity greater than a threshold as the second candidate image block.

[0112] In an optional implementation, in order to improve the referenceability of the first candidate image block and the second candidate image block and effectively improve the accuracy of repairing the image block to be filled, in an optional implementation, the number of texture regions contained in the first candidate image block, the second candidate image block and the image block to be filled can be the same.

[0113] After determining multiple candidate image blocks in the above manner, the image computer device can determine the image block with the highest matching degree with the image block to be filled from the multiple candidate image blocks based on the similarity between each candidate image block and the image block to be filled.

[0114] In an optional implementation, the above-mentioned determination of the image block with the highest matching degree from multiple candidate image blocks based on the similarity between each candidate image block and the image block to be filled can be specifically implemented as follows: determining the matching degree between each candidate image block and the image block to be filled based on the similarity between each candidate image block and the image block to be filled, and the weight corresponding to the image containing the candidate image block; and determining the image block with the highest matching degree between multiple candidate image blocks based on the matching degree between each candidate image block and the image block to be filled.

[0115] Images containing candidate image patches include the image to be repaired and / or a reference image.

[0116] The embodiments disclosed herein do not specifically limit the weights corresponding to the image to be repaired and the reference image. Taking the image to be repaired as the first projected image and the reference images including the second and third projected images as an example, in one example, the weight corresponding to the first projected image may be less than the weight corresponding to the second projected image, and the weight corresponding to the second projected image may be less than the weight corresponding to the third projected image.

[0117] The method for determining the similarity between the candidate image block and the image block to be filled can refer to the method for determining the similarity between each image block A and the image block to be filled, as described above, and will not be repeated here.

[0118] Specifically, after obtaining the similarity between each candidate image patch and the image patch to be filled, the image computing device can obtain the weight corresponding to the image containing each candidate image patch. Then, the image computing device can determine the matching degree between each candidate image patch and the image patch to be filled based on the following formula (Formula 4).

[0119] W = R * Q (Formula 4)

[0120] Where W represents the matching degree between the candidate image patch and the image patch to be filled, R represents the similarity between the candidate image patch and the image patch to be filled, and Q represents the weight of the image containing the candidate image patch.

[0121] After obtaining the matching degree between each candidate image block and the image block to be filled using Formula 4 above, the image computer device can select the candidate image block with the highest matching degree with the image block to be filled as the image block with the highest matching degree with the image block to be filled.

[0122] The imaging computer equipment determines the image block with the highest matching degree to the image block to be filled using the above method. After filling the first pixel area of ​​the image block to be filled with this image block, local texture reconstruction of the image to be repaired can be achieved. This causes the boundary line between the area to be repaired and the complete area in the image to be repaired to dynamically shrink into the area to be repaired, thereby reducing the area to be repaired in the image to be repaired. Subsequently, the imaging computer equipment can further repair the filled image to be repaired using the above method until the area to be repaired in the image to be repaired is completely filled, resulting in the repaired image.

[0123] The following will combine the above-described embodiments to discuss... Figure 3 The image restoration method described above will be further introduced.

[0124] Figure 6 A flowchart of another image restoration method provided in this disclosure embodiment is shown below. Figure 6 As shown, the method includes:

[0125] S601, Determine the boundary lines between the region to be repaired and the complete region in the image to be repaired.

[0126] S602, for each boundary point on the boundary line, determine the weight of the boundary point based on the number of pixels in the region to be repaired in the candidate region corresponding to the boundary point, and the number of texture regions contained in the candidate region corresponding to the boundary point.

[0127] S603, the candidate region corresponding to the boundary point with the highest weight is selected as the image block to be filled. The image block to be filled contains the first pixel region and the second pixel region.

[0128] S604, determines the texture boundary line in the complete region.

[0129] S605, based on the texture boundary line of the complete region, determine the first candidate image block from the complete region whose similarity to the image block to be filled is greater than a threshold.

[0130] S606, determine from the reference image a third pixel region that is at the same position as the image block to be filled in the image to be repaired.

[0131] S607, Based on the third pixel region, determine the second candidate image block from the reference image.

[0132] S608, based on the similarity between each candidate image block (including the first candidate image block and the second candidate image block) and the image block to be filled, and the weight corresponding to the image containing each candidate image block, determine the matching degree between each candidate image block and the image block to be filled.

[0133] S609, based on the matching degree between each candidate image block and the image block to be filled, determine the image block with the highest matching degree with the image block to be filled from multiple candidate image blocks.

[0134] S610, based on the image block that has the highest matching degree with the image block to be filled, fill the first pixel region in the image block to be filled.

[0135] S611, Does the image to be repaired still contain the area to be repaired?

[0136] If the image to be repaired still contains areas to be repaired, then execute S612, and repeat the above S601-S611.

[0137] If the image to be repaired does not contain the area to be repaired, then execute S613.

[0138] S612, the filled image to be repaired is used as the new image to be repaired.

[0139] S613, End.

[0140] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein. In some embodiments, the electronic device may be as described above. Figure 1 The image computer device shown.

[0141] like Figure 7As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory 702 or a computer program loaded from a storage unit 708 into a random access memory 703. The random access memory (RAM) 703 can also store various programs and data required for the operation of the electronic device 700. The computing unit 701, the read-only memory (ROM) 702, and the RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0142] Multiple components in electronic device 700 are connected to input / output interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0143] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit, a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors, and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the reset method. For example, in one embodiment, the reset method may be implemented as a computer software program tangibly included in a machine-readable medium, such as storage unit 708. In one embodiment, part or all of the computer program may be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the reset method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the reset method by any other suitable means (e.g., by means of firmware).

[0144] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays, application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0145] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0146] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0147] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user, such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0148] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0149] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0150] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0151] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image restoration method, characterized in that, The method includes: In the image to be repaired, an image block to be filled is determined. The image block to be filled includes a first pixel region and a second pixel region. The first pixel region includes a portion of the pixels in the region to be repaired of the image to be repaired, and the second pixel region includes a portion of the pixels in the complete region of the image to be repaired. Based on the complete region and the reference image, the image block with the highest matching degree to the image block to be filled is determined. The size of the image block is the same as the size of the image block to be filled. The reference image and the image to be repaired are images taken of the same object. The first pixel region is filled based on the image block.

2. The method according to claim 1, characterized in that, The step of determining the image block with the highest matching degree to the image block to be filled, based on the complete region and the reference image, includes: Multiple candidate image blocks are determined from the complete region of the image to be repaired and the reference image; Based on the similarity between each candidate image block and the image block to be filled, the image block with the highest matching degree with the image block to be filled is determined from the plurality of candidate image blocks.

3. The method according to claim 2, characterized in that, The step of determining the image block with the highest matching degree to the image block to be filled from the plurality of candidate image blocks based on the similarity between each candidate image block and the image block to be filled includes: Based on the similarity between each candidate image patch and the image patch to be filled, and the weight corresponding to the image containing the candidate image patch, the matching degree between each candidate image patch and the image patch to be filled is determined; Based on the matching degree between each candidate image block and the image block to be filled, the image block with the highest matching degree with the image block to be filled is determined from the plurality of candidate image blocks.

4. The method according to claim 2, characterized in that, The similarity between the candidate image patch and the image patch to be filled is measured by at least one of the following: Pixel intensity; Gradient distribution.

5. The method according to claim 2, characterized in that, The plurality of candidate image blocks includes a first candidate image block and a second candidate image block, wherein the first candidate image block is an image block in the complete region and the second candidate image block is an image block in the reference image; The step of determining multiple candidate image patches from the complete region of the image to be repaired and the reference image includes: Determine the texture boundary lines in the complete region, where the texture boundary lines refer to the boundary lines between different texture regions; Based on the texture boundary line of the complete region, the first candidate image block with a similarity greater than a threshold to the image block to be filled is determined from the complete region; A third pixel region is determined from the reference image, the position of which in the reference image is the same as the position of the image block to be filled in the image to be repaired; The second candidate image block is determined from the reference image based on the third pixel region.

6. The method according to claim 5, characterized in that, The first candidate image block, the second candidate image block, and the image block to be filled contain the same number of texture regions.

7. The method according to claim 1, characterized in that, The step of determining the image block to be filled in the image to be repaired includes: Determine the boundary lines between the region to be repaired and the complete region in the image to be repaired; Based on the weights of the boundary points located on the boundary line, the image blocks to be filled are determined in the image to be repaired.

8. The method according to claim 7, characterized in that, The boundary points include multiple points, and each boundary point corresponds to a candidate region; The step of determining the image block to be filled in the image to be repaired based on the weights of boundary points located on the boundary line includes: For each boundary point, the weight of the boundary point is determined based on the number of pixels in the region to be repaired in the candidate region corresponding to the boundary point, and the number of texture regions contained in the candidate region corresponding to the boundary point. The candidate region corresponding to the boundary point with the highest weight is taken as the image block to be filled.

9. The method according to claim 1, characterized in that, The image to be repaired is the first projection image of the target object under the current fractionation radiotherapy, and the reference image includes the second projection image of the target object under the current fractionation radiotherapy, or the third projection image of the target object under a historical fractionation radiotherapy. The first projected image and the second projected image have different imaging angles, and the first projected image and the third projected image have the same or different imaging angles.

10. An electronic device, characterized in that, The electronic device includes: processor; A memory configured to store processor-executable instructions; The processor is configured to execute the instructions to implement the image restoration method as described in any one of claims 1-9.

Images