Image repair method, device and medium

Abstract

Disclosed are an image repair method, device and medium. The method comprises: determining a first set of three-dimensional layout diagrams and a second set of three-dimensional layout diagrams based on a first panoramic image and a second panoramic image taken for the same space, respectively, wherein the first panoramic image and the second panoramic image are taken at different locations; in the first set of three-dimensional layout diagrams, determining an image where a target object is located as an image to be repaired, and in the second set of three-dimensional layout diagrams, determining an image where a target object is located as a reference image, wherein the position of the target object in space when the first panoramic image is taken is different from the position of the target object in space when the second panoramic image is taken; in the reference image, determining a matched area that does not contain the target object and matches the area containing the target object in the image to be repaired; and generating a repaired image by replacing the area containing the target image in the image to be repaired with the matched area in the reference image.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision, and more particularly, to the field of image inpainting for panoramic camera vision. Background Art

[0002] In the field of panoramic vision, in a captured panorama or in a three-dimensional layout diagram (such as a three-dimensional layout of a house) obtained based on a panorama conversion, objects that should not belong to the captured three-dimensional space, such as the camera used to capture the panorama, people who accidentally enter, etc., often appear. However, the existence of such objects in a panorama or a three-dimensional layout diagram looks very abrupt. Therefore, how to delete unwanted objects in a panorama or a three-dimensional layout diagram has attracted widespread attention in the field of computer vision. The problem with the existing repair methods is that the repaired area often loses texture and the amount of calculation is large. Summary of the invention

[0003] In view of the above situation, it is desirable to provide an image repair method, device and medium that can restore detailed texture and has a lightweight processing process.

[0004] According to one aspect of the present disclosure, there is provided an image repair method, comprising: determining a first group of three-dimensional layout diagrams and a second group of three-dimensional layout diagrams based on a first panoramic image and a second panoramic image respectively obtained by taking pictures of the same space, wherein the first panoramic image and the second panoramic image are taken at different positions, and the first group of three-dimensional layout diagrams and the second group of three-dimensional layout diagrams both include multiple images that can reflect the structure of the space; in the first group of three-dimensional layout diagrams, determining an image where a target object is located as an image to be repaired, and in the second group of three-dimensional layout diagrams, determining an image where the target object is located as a reference image, wherein the position of the target object in the space when the first panoramic image is taken is different from the position of the target object in the space when the second panoramic image is taken; in the reference image, determining a matched area that does not contain the target object and matches the area containing the target object in the image to be repaired; and generating a repaired image by replacing the area containing the target image in the image to be repaired with the matched area in the reference image.

[0005] In addition, in the method according to the embodiment of the present disclosure, in the reference image, a matched area that matches the area in the image to be repaired that contains the target object and does not contain the target object is determined, including: determining the first position and the second position of the target object in the image to be repaired and the reference image; determining the overlapping area containing the first position and the second position in the image to be repaired and the reference image, respectively, as the first matching area and the second matching area; performing feature point matching between the first matching area and the second matching area to determine a transformation matrix; based on the transformation matrix, transforming the second matching area into a third matching area; and generating the matched area by removing the area containing the target object in the third matching area.

[0006] In addition, in the method according to the embodiment of the present disclosure, based on the first position and the second position, the overlapping area containing the first position and the second position is determined in the image to be repaired and the reference image, respectively, as the first matching area and the second matching area, including: in the image to be repaired, marking the second position, determining a first area centered on the first position and a second area centered on the second position, and taking the overlapping area of ​​the first area and the second area as the first matching area; and in the reference image, marking the first position, determining a third area centered on the first position and a fourth area centered on the second position, and taking the overlapping area of ​​the third area and the fourth area as the second matching area.

[0007] In addition, in the method according to the embodiment of the present disclosure, determining a first area centered on the first position and a second area centered on the second position includes: drawing a rectangle as the first area with the first position as the center, the line connecting the first position and the second position as the diagonal direction, and the product of the distance between the first position and the second position and the first coefficient as half of the diagonal length; and drawing a rectangle as the second area with the second position as the center, the line connecting the first position and the second position as the diagonal direction, and the product of the distance between the first position and the second position and the second coefficient as half of the diagonal length.

[0008] In addition, in the method according to the embodiment of the present disclosure, before determining the matched area, it further includes: performing color calibration on the image to be repaired.

[0009] In addition, in the method according to the embodiment of the present disclosure, color calibration is performed on the image to be repaired, including: transforming the image to be repaired and the reference image from RGB color space to LAB color space; respectively calculating the mean and variance of each channel of the image to be repaired and the reference image in the LAB color space; normalizing the value of each channel of the image to be repaired in the LAB color space based on the mean and variance of each channel of the image to be repaired in the LAB color space; denormalizing the normalized value of each channel of the image to be repaired in the LAB color space based on the mean and variance of each channel of the reference image in the LAB color space; and transforming the denormalized image to be repaired from the LAB color space back to the RGB color space.

[0010] In addition, in the method according to the embodiment of the present disclosure, after the repaired image is generated, the method further includes: generating a repaired first panoramic image based on the repaired image and other images in the first group of three-dimensional layout images.

[0011] According to another aspect of the present disclosure, an image repair device is provided, comprising: a conversion unit, for determining a first group of three-dimensional layout diagrams and a second group of three-dimensional layout diagrams based on a first panoramic image and a second panoramic image taken for the same space, respectively, wherein the first panoramic image and the second panoramic image are taken at different locations, and the first group of three-dimensional layout diagrams and the second group of three-dimensional layout diagrams both include multiple images that can reflect the structure of the space; a determination unit, for determining an image where a target object is located in the first group of three-dimensional layout diagrams as an image to be repaired, and for determining an image where the target object is located in the second group of three-dimensional layout diagrams as a reference image, wherein the position of the target object in the space when the first panoramic image is taken is different from the position of the target object in the space when the second panoramic image is taken; a matching unit, for determining, in the reference image, a matched area that does not contain the target object and matches the area containing the target object in the image to be repaired; and a repair unit, for generating a repaired image by replacing the area containing the target image in the image to be repaired with the matched area in the reference image.

[0012] In addition, in the device according to an embodiment of the present disclosure, the matching unit is configured to perform the following processing: determine the first position and the second position of the target object in the image to be repaired and the reference image; determine the overlapping area containing the first position and the second position in the image to be repaired and the reference image, respectively, as the first matching area and the second matching area; perform feature point matching between the first matching area and the second matching area to determine the transformation matrix; based on the transformation matrix, transform the second matching area into a third matching area; and generate the matched area by removing the area containing the target object in the third matching area.

[0013] In addition, in the device according to an embodiment of the present disclosure, the matching unit is further configured to determine a first matching area and a second matching area by performing the following processing: in the image to be repaired, marking the second position, determining a first area centered on the first position and a second area centered on the second position, and taking the overlapping area of ​​the first area and the second area as the first matching area; and in the reference image, marking the first position, determining a third area centered on the first position and a fourth area centered on the second position, and taking the overlapping area of ​​the third area and the fourth area as the second matching area.

[0014] In addition, in the device according to an embodiment of the present disclosure, the matching unit is further configured to determine a first area centered on the first position and a second area centered on the second position by performing the following processing: drawing a rectangle as the first area with the first position as the center, a line connecting the first position and the second position as the diagonal direction, and a product of the distance between the first position and the second position and the first coefficient as half of the diagonal length; and drawing a rectangle as the second area with the second position as the center, a line connecting the first position and the second position as the diagonal direction, and a product of the distance between the first position and the second position and the second coefficient as half of the diagonal length.

[0015] In addition, the device according to the embodiment of the present disclosure may further include: a color calibration unit, configured to perform color calibration on the image to be repaired.

[0016] In addition, in the device according to the embodiment of the present disclosure, the color calibration unit is configured to perform the following processing: transforming the image to be repaired and the reference image from the RGB color space to the LAB color space; respectively calculating the mean and variance of each channel of the image to be repaired and the reference image in the LAB color space; normalizing the value of each channel of the image to be repaired in the LAB color space based on the mean and variance of each channel of the image to be repaired in the LAB color space; denormalizing the normalized value of each channel of the image to be repaired in the LAB color space based on the mean and variance of each channel of the reference image in the LAB color space; and transforming the denormalized image to be repaired from the LAB color space back to the RGB color space.

[0017] In addition, the device according to the embodiment of the present disclosure may further include: a reverse conversion unit, configured to generate a repaired first panoramic image based on the repaired image and other images in the first group of three-dimensional layout images.

[0018] According to another aspect of the present disclosure, an image repair device is provided, comprising: a memory for storing a computer program thereon; and a processor for implementing the following processing when executing the computer program: determining a first group of three-dimensional layout diagrams and a second group of three-dimensional layout diagrams based on a first panoramic image and a second panoramic image taken for the same space, respectively, wherein the first panoramic image and the second panoramic image are taken at different locations, and the first group of three-dimensional layout diagrams and the second group of three-dimensional layout diagrams both include multiple images that can reflect the structure of the space; in the first group of three-dimensional layout diagrams, determining an image where a target object is located as an image to be repaired, and in the second group of three-dimensional layout diagrams, determining an image where the target object is located as a reference image, wherein the position of the target object in the space when the first panoramic image is taken is different from the position of the target object in the space when the second panoramic image is taken; in the reference image, determining a matched area that does not contain the target object and matches the area containing the target object in the image to be repaired; and generating a repaired image by replacing the area containing the target image in the image to be repaired with the matched area in the reference image.

[0019] According to another aspect of the present disclosure, a computer-readable recording medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described above is performed.

[0020] In the image repair method, device and medium according to the present disclosure, by using reference images taken at different shooting positions where the target object is located at different positions in the same space to repair the area containing the target object in the image to be repaired, the target object can be removed while still retaining the original detail texture, so that the repaired image has no trace of repair and is more natural. In addition, the amount of calculation involved in the image repair method is small, and the processing process can be kept lightweight. In addition, by using the reference image to perform color calibration on the image to be repaired before the repair process, the color consistency between the image to be repaired and the reference image can be maintained, avoiding the situation where color differences in the repaired image cause unnatural repair. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flowchart illustrating a process of an image repair method according to an embodiment of the present disclosure;

[0022] Figure 2 An example of an equirectangular projection image obtained by taking a panoramic shot of a room is shown;

[0023] Figure 3 A schematic diagram showing a portion of a room structure restored based on a set of three-dimensional layout graphs;

[0024] Figure 4A and Figure 4B Examples of the image to be repaired and the reference image are shown respectively;

[0025] Figure 5 is a flowchart illustrating a specific process of matching processing according to an embodiment of the present disclosure;

[0026] Fig. 6A and Figure 6B are schematic diagrams showing, respectively, determining a first matching region and a second matching region in an image to be repaired and a reference image according to an embodiment;

[0027] Figure 7 A schematic diagram showing a method of replacing a region in an image to be repaired containing the target image with a matched region in a reference image;

[0028] Fig. 8A and Figure 8B A comparison of the image to be repaired and the repaired image is shown.

[0029] Fig. 9 is a flowchart illustrating a procedure of a color calibration process according to an embodiment of the present disclosure;

[0030] Fig.10 is a functional block diagram illustrating a configuration of an image inpainting apparatus according to an embodiment of the present disclosure; and

[0031] Fig.11 A schematic diagram showing the architecture of an exemplary computing device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0032] The various preferred embodiments of the present invention will be described below with reference to the accompanying drawings. The following description with reference to the accompanying drawings is provided to assist in the understanding of the exemplary embodiments of the present invention as defined by the claims and their equivalents. It includes various specific details to assist in understanding, but they are to be viewed as exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Moreover, in order to make the specification more clear and concise, detailed descriptions of functions and configurations well known in the art will be omitted.

[0033] First, refer to Figure 1 The image repair method according to an embodiment of the present disclosure is described to remove a target object. Figure 1 As shown, the method includes the following steps.

[0034] First, in step S101, a first set of three-dimensional layout diagrams and a second set of three-dimensional layout diagrams are determined based on a first panoramic image and a second panoramic image captured for the same space, respectively.

[0035] The first panoramic image and the second panoramic image are shot at different positions. For example, the first panoramic image is obtained by placing a panoramic camera at a first shooting position in the space (for example, the space can be a specific room), and the second panoramic image is obtained by placing a panoramic camera at a second shooting position in the space. In addition, the first set of three-dimensional layout diagrams is generated based on the first panoramic image, and the second set of three-dimensional layout diagrams is generated based on the second panoramic image. Moreover, the first set of three-dimensional layout diagrams and the second set of three-dimensional layout diagrams both include multiple images that can reflect the structure of the space.

[0036] For example, the panoramic image may be a cylindrical panoramic image. Two fisheye images of a space (e.g., corresponding to a 360-degree space) may be acquired by a panoramic camera including two fisheye lenses (e.g., Ricoh's Theta panoramic camera). Then, by performing processing such as projection transformation on the two fisheye images, an equirectangular cylindrical projection image is obtained as the panoramic image. Figure 2 An example of an equirectangular projection image obtained by taking a panoramic shot of a room is shown. Of course, the present disclosure is not limited thereto. Those skilled in the art will appreciate that any other type of panoramic image may also be similarly applied to the present disclosure and should be included within the scope of the present disclosure.

[0037] In addition, based on the panoramic image, a set of three-dimensional layout diagrams of the space can be obtained. For example, a set of three-dimensional layout diagrams for reconstructing the three-dimensional structure in the space can be obtained by performing corner point detection, line segment extraction, plane segmentation and other processing on the panoramic image. Specifically, a set of three-dimensional layout diagrams may include six facet diagrams of the space. Through these six facet diagrams, the three-dimensional structure of the space can be restored. Figure 3 A schematic diagram showing a portion of the room structure restored based on a set of three-dimensional layout graphs.

[0038] Then, return to the reference Figure 1 In step S102, in the first set of three-dimensional layout images, an image where the target object is located is determined as the image to be repaired, and in the second set of three-dimensional layout images, an image where the target object is located is determined as the reference image. The position of the target object in the space when the first panoramic image is taken is different from the position of the target object in the space when the second panoramic image is taken.

[0039] The target object refers to an object that is not desired to appear in the image of the space (e.g., a three-dimensional layout map or a panoramic image). For example, the target object may be a camera used for panoramic photography. Figure 3 In FIG. 3 , the position of the panoramic camera used to perform the shooting is shown by box 300. It can be seen that Figure 3 In the example, the panoramic camera is placed on the floor. In this case, the target object is located in the facet image corresponding to the floor. Of course, the target object is not limited to the camera. For example, the target object may also include a camera and a device holding the camera (such as a tripod or a person). Alternatively, the target object may also be a person, animal, or other object that accidentally enters the space. In the following, the target object is described as a camera. Of course, it can be understood that other examples of target objects are also similarly applicable.

[0040] Figure 4A and Figure 4B Examples of the image to be repaired and the reference image are shown respectively. Figure 4A and Figure 4B In , both the image to be repaired 401 and the reference image 402 correspond to the floor image of the room. Figure 4A and Figure 4B In FIG. 4 , the positions of the cameras 403 and 404 as the target objects are shown respectively. It can be seen that the positions of the cameras are different in the two images. This means that the reference image can be used to repair the area covered by the camera on the image to be repaired.

[0041] Therefore, return to the reference Figure 1In step S103, in the reference image, a matched area that matches the area containing the target object in the image to be repaired and does not contain the target object is determined. Since the reference image is an image obtained by photographing the same space, the image to be repaired is repaired by repairing the corresponding area in the reference image, so that the detail texture can be restored. For example, Figures 2 to 4B In the case shown, the inpainting using the reference image preserves the tile texture on the floor, making the inpainted image look more natural.

[0042] Will refer to Figure 5 Describe the specific process of the matching process involved in step S103. Figure 5 As shown, the matching process further includes the following steps.

[0043] First, in step S501, the first position and the second position of the target object in the image to be repaired and the reference image are determined. For example, in the case where the target object is a camera, through the conversion process from the panoramic image to the three-dimensional layout map, in addition to being able to determine the three-dimensional layout map, the position of the camera used for shooting can also be determined. In addition, for example, in the case where the target object is a person or other object other than a camera, the position of the target object can be determined through the process of target detection and recognition.

[0044] Then, in step S502, the overlapping area including the first position and the second position is determined in the image to be repaired and the reference image, respectively, as the first matching area and the second matching area. The overlapping area including the first position and the second position can be considered as an area captured by cameras at different positions, or can be considered as an area including target objects at different positions at the same time. In the image repair process, such an overlapping area is important for preserving the detailed texture in the repaired image.

[0045] Specifically, since the image to be repaired only includes the target object at the first position, the second position needs to be additionally marked. Then, as a possible implementation, a first area centered on the first position and a second area centered on the second position are determined, and the overlapping area of ​​the first area and the second area is used as the first matching area.

[0046] Similar to the image to be repaired, since the reference image only includes the target object at the second position, the first position needs to be additionally marked. Then, as a possible implementation, a third area centered on the first position and a fourth area centered on the second position are determined, and the overlapping area of ​​the third area and the fourth area is used as the second matching area.

[0047] Fig. 6A and Figure 6B Schematic diagrams showing how to determine a first matching region and a second matching region in an image to be repaired and a reference image according to an embodiment. Figure 4A and Figure 4B By comparison, it can be seen that for the sake of clarity, Fig. 6A and Figure 6B The detail texture in the floor image is omitted. Fig. 6A and Figure 6B In FIG. 6 , the image to be repaired and the reference image are represented by 601 and 602 respectively, and the first position and the second position are represented by O1 and O2 respectively.

[0048] exist Fig. 6A In the image to be repaired 601, the second position O2 is further marked on the basis of the original first position O1. Then, with the first position O1 as the center, the line connecting the first position O1 and the second position O2 as the diagonal direction, and the product of the distance D between the first position O1 and the second position O2 and the first coefficient as half of the diagonal length, a rectangle is drawn as the first area 6011. Here, the first coefficient is a real number greater than 1. Therefore, Fig. 6A As shown, half of the diagonal length of the drawn first area 6011 is greater than the distance D between the first position O1 and the second position O2, so as to cover a larger range.

[0049] In a similar manner, a rectangle is drawn as the second area 6012 with the second position O2 as the center, the line connecting the first position O1 and the second position O2 as the diagonal direction, and the product of the distance D between the first position O1 and the second position O2 and the second coefficient as half of the diagonal length. Here, the second coefficient is also a real number greater than 1. In addition, the second coefficient may be the same as the first coefficient or different from the first coefficient.

[0050] After determining the first area 6011 and the second area 6012, the overlapping area (in Fig. 6A ) as the first matching area.

[0051] exist Figure 6B In a similar manner, in the reference image 602, a third region 6021 and a fourth region 6022 are determined, and the overlapping region (in Figure 6B ) as the second matching area.

[0052] Of course, in addition to determining the first area and the second area with the first position and the second position as the center respectively, the first area and the second area respectively including the first position and the second position may also be determined in any other manner, and should also be included in the scope of the present disclosure.

[0053] Next, return Figure 5 After determining the first matching region in the image to be repaired and the second matching region in the reference image, the process proceeds to step S503. In step S503, feature point matching is performed between the first matching region and the second matching region to determine a transformation matrix.

[0054] Since the first matching area and the second matching area are based on images obtained from different shooting positions, the two matching areas determined by the above method may not be completely aligned. For example, in the case of a floor image, there may be situations such as tile gaps that cannot be aligned. Therefore, the matching results are further fine-tuned by feature point matching to ensure complete alignment between the two matching areas.

[0055] Then, in step S504, based on the transformation matrix, the second matching region is transformed into a third matching region. For example, by multiplying the second matching region by the transformation matrix, the second matching region can be rotated and / or translated, and the rotated and translated region is used as the third matching region. The position of each pixel in the third matching region is completely corresponding to the position of each pixel in the first matching region.

[0056] Here, it should be pointed out that since the third matching area contains the target object, but the target object is not desired to appear in the image, it is necessary to remove the target object contained in the third matching area before performing the repair process.

[0057] Therefore, in step S505, the matched area is generated by removing the area containing the target object from the third matching area. For example, the matched area is generated by directly cutting out the area containing the target object from the third matching area. In other words, the matched area is a part of the third matching area. Since the position of each pixel in the third matching area is completely corresponding to the position of each pixel in the first matching area, as a part of the third matching area, the position of each pixel in the matched area is also completely corresponding to the position of each pixel in the corresponding part of the first matching area.

[0058] Next, return to the reference Figure 1. After determining the matched area, the processing proceeds to step S104. In step S104, a repaired image is generated by replacing the area containing the target image in the image to be repaired with the matched area in the reference image. As described above, the positions of each pixel point in the matched area are completely corresponding to the positions of each pixel point in the corresponding part of the first matching area. Therefore, by replacing the pixel values ​​at the corresponding pixel points in the corresponding part of the first matching area with the pixel values ​​at the pixel points in the matched area, it is possible to repair the area containing the target image in the image to be repaired with the matched area.

[0059] Figure 7 FIG. 1 is a schematic diagram showing a method of replacing a region in an image to be repaired containing the target image with a matched region in a reference image. Figure 7 As shown, the matched area 704 in the reference image 702 is an area that does not contain the target object, and the image 701 to be repaired is repaired by replacing the corresponding area 703 in the image 701 to be repaired with the matched area 703 in the reference image 702.

[0060] Fig. 8A and Figure 8B A comparison diagram of the image to be repaired and the repaired image is shown. Fig. 8A and Figure 8B It can be seen from FIG. 8 that the region 803 containing the target object in the image to be repaired 801 has been replaced by a region 804 not containing the target object. Figure 8B It can be clearly seen that the detailed texture of the floor is still retained in the area 804 where the target object is removed, and it seems that there is no trace of being repaired.

[0061] In addition, after the repaired image is generated, depending on the specific application scenario and design requirements, a repaired panoramic image can be further obtained. Figure 1 After step S104, the method may further include step S105. In step S105, a first panorama image after repair is generated based on the repaired image and other images in the first group of three-dimensional layout images. Of course, in other possible implementations, step S105 may also be omitted. Figure 1 In FIG. 5 , step S105 is shown in a dotted box.

[0062] As another possible implementation, before the step of determining the matched area described above, the method may further include: in step S106, color calibration of the image to be repaired. Of course, in other possible implementations, step S106 may also be omitted. Figure 1In the figure, step S106 is shown in a dotted box. By pre-calibrating the color of the image to be repaired before matching the image to be repaired with the reference image, the color difference between the image to be repaired and the reference image due to shooting conditions (such as shooting lighting, etc.) can be reduced. Therefore, when the image to be repaired is repaired with the matched area in the reference image, the color inconsistency in the repaired image is avoided.

[0063] Fig. 9 An example implementation of a color calibration process is shown. Fig. 9 , color calibration of the image to be repaired may include the following steps.

[0064] First, in step S901, the image to be repaired and the reference image are transformed from the RGB color space to the LAB color space. RGB color is the abbreviation of the three primary colors of red, green and blue. In the RGB color space, various different colors are generated by superimposing the three basic colors to different degrees. This color space can cover all colors that can be perceived by human vision and is one of the most widely used color spaces.

[0065] Compared with the RGB color space, the LAB color space is a less commonly used color space. In the LAB color space, similar to the RGB color space, three channels, L, A, and B, are used to represent different colors. The L channel is used to represent the brightness of the pixel, and the value range is [0,100], representing from pure black to pure white; the A channel represents the range from red to green, and the value range is [127,-128]; the B channel represents the range from yellow to blue, and the value range is [127,-128].

[0066] For example, the image is transformed from the RGB color space to the LAB color space indirectly via XYZ as an intermediate mode. Specifically, the first step is the conversion from RGB to XYZ, as shown in Formula 1-1.

[0067] [X, Y, Z] = M*[R, G, B] (1-1)

[0068] In the formula, RGB is the color component after Gamma correction: R = g(r), G = g(g), B = g(b), r, g, b are the original color components.

[0069] The conversion between XYZ and LAB is shown in formula 1-2.

[0070]

[0071] In the formula, X1, Y1, and Z1 are the linearly normalized values ​​of X, Y, and Z, respectively.

[0072] Then, in step S902, the average value and variance of each channel of the image to be repaired and the reference image in the LAB color space are calculated respectively.

[0073] Next, in step S903, based on the average value and variance of each channel of the image to be repaired in the LAB color space, the value of each channel of the image to be repaired in the LAB color space is normalized. Specifically, the above normalization can be achieved by the following formula (1-3):

[0074]

[0075] Where, L N represents the normalized value of the L channel of the image to be repaired at a certain pixel point, L represents the initial value of the L channel of the image to be repaired at the pixel point, E L represents the average value of the L channel of the image to be repaired, and σ L Represents the variance of the L channel of the image to be inpainted.

[0076] In addition, the normalized values ​​of the A channel and the B channel can also be calculated similarly.

[0077] Then, in step S904, based on the mean value and variance of each channel of the reference image in the LAB color space, the normalized value of each channel of the image to be repaired in the LAB color space is denormalized. Specifically, the denormalization can be achieved by the following formula (1-4):

[0078] L DN =L N ×σ LR +E LR (1-4)

[0079] Where, L DN Represents the denormalized value of the L channel of the image to be repaired at a certain pixel, E LR represents the average value of the L channel of the reference image, and σ LR Represents the variance of the L channel of the reference image.

[0080] Finally, in step S905, the denormalized image to be repaired is transformed from the LAB color space back to the RGB color space. The inverse transformation method from the LAB color space to the RGB color space can be derived by inversely deriving the transformation method from the RGB color space to the LAB color space.

[0081] In the above, reference has been made to Figures 1 to 9The image repair method according to various embodiments of the present disclosure is described in detail. In the image repair method, by using reference images captured at different shooting positions and with the target object located at different positions in the same space to repair the area containing the target object in the image to be repaired, the target object can be removed while still retaining the original detail texture, so that the repaired image has no trace of repair at all and is more natural. In addition, the amount of calculation involved in the image repair method is small, and the processing process can be kept lightweight. In addition, by using the reference image to perform color calibration on the image to be repaired before the repair process, the color consistency between the image to be repaired and the reference image can be maintained, avoiding the situation where color differences in the repaired image cause unnatural repair.

[0082] Next, we will refer to Fig.10 An image repair device according to an embodiment of the present disclosure is described, which is used to remove a target object. Fig.10 As shown, the image repair device 1000 includes: a conversion unit 1001, a determination unit 1002, a matching unit 1003 and a repair unit 1004.

[0083] The conversion unit 1001 is used to determine a first group of three-dimensional layout diagrams and a second group of three-dimensional layout diagrams based on a first panoramic image and a second panoramic image taken for the same space, respectively. The shooting positions of the first panoramic image and the second panoramic image are different. For example, the first panoramic image is obtained by placing a panoramic camera at a first shooting position in the space (for example, the space can be a specific room), and the second panoramic image is obtained by placing the panoramic camera at a second shooting position in the space. In addition, the conversion unit 1001 generates the first group of three-dimensional layout diagrams by converting the first panoramic image, and generates the second group of three-dimensional layout diagrams by converting the second panoramic image. The first group of three-dimensional layout diagrams and the second group of three-dimensional layout diagrams both include multiple images that can reflect the structure of the space.

[0084] The determining unit 1002 is used to determine, in the first set of three-dimensional layout images, an image where the target object is located as the image to be repaired, and to determine, in the second set of three-dimensional layout images, an image where the target object is located as the reference image, wherein the position of the target object in the space when the first panoramic image is taken is different from the position of the target object in the space when the second panoramic image is taken.

[0085] The matching unit 1003 is used to determine, in the reference image, a matched area that matches the area in the image to be repaired that contains the target object and does not contain the target object. Since the reference image is an image obtained by photographing the same space, the image to be repaired is repaired by repairing the corresponding area in the reference image, so that the detail texture can be restored.

[0086] Specifically, the matching unit 1003 may be configured to perform the following processing.

[0087] First, the matching unit 1003 determines the first position and the second position of the target object in the image to be repaired and the reference image. For example, when the target object is a camera, through the conversion process from the panoramic image to the three-dimensional layout map, in addition to determining the three-dimensional layout map, the position of the camera used for shooting can also be determined. In addition, for example, when the target object is a person or other object other than a camera, the position of the target object can be determined through the processing of target detection and recognition.

[0088] Then, the matching unit 1003 determines the overlapping area including the first position and the second position in the image to be repaired and the reference image, respectively, as the first matching area and the second matching area. The overlapping area including the first position and the second position can be considered as an area captured by cameras at different positions, or can be considered as an area including target objects at different positions at the same time. In the image repair process, such an overlapping area is important for retaining the detailed texture in the repaired image.

[0089] Specifically, since the image to be repaired only includes the target object at the first position, the second position needs to be additionally marked. Then, as a possible implementation, the matching unit 1003 determines a first area centered on the first position and a second area centered on the second position, and uses the overlapping area of ​​the first area and the second area as the first matching area.

[0090] Similar to the image to be repaired, since the reference image only includes the target object at the second position, the first position needs to be additionally marked. Then, as a possible implementation, the matching unit 1003 determines a third area centered on the first position and a fourth area centered on the second position, and uses the overlapping area of ​​the third area and the fourth area as the second matching area.

[0091] For example, the matching unit 1003 can be further configured to determine a first area centered on the first position and a second area centered on the second position by performing the following processing: drawing a rectangle as the first area with the first position as the center, a line connecting the first position and the second position as the diagonal direction, and a product of the distance between the first position and the second position and the first coefficient as half of the diagonal length; and drawing a rectangle as the second area with the second position as the center, a line connecting the first position and the second position as the diagonal direction, and a product of the distance between the first position and the second position and the second coefficient as half of the diagonal length.

[0092] Of course, similarly, the matching unit 1003 can be further configured to determine a third area centered on the first position and a fourth area centered on the second position by performing the following processing: drawing a rectangle as the third area with the first position as the center, the line connecting the first position and the second position as the diagonal direction, and the product of the distance between the first position and the second position and the first coefficient as half of the diagonal length; and drawing a rectangle as the fourth area with the second position as the center, the line connecting the first position and the second position as the diagonal direction, and the product of the distance between the first position and the second position and the second coefficient as half of the diagonal length.

[0093] Of course, in addition to determining the first area and the second area with the first position and the second position as the center respectively, the matching unit 1003 can also determine the first area and the second area respectively including the first position and the second position in any other way, which should also be included in the scope of the present disclosure.

[0094] After determining the first matching area and the second matching area, the matching unit 1003 may perform feature point matching between the first matching area and the second matching area to determine a transformation matrix. Furthermore, based on the transformation matrix, the matching unit 1003 may transform the second matching area into a third matching area. The position of each pixel in the third matching area is completely corresponding to the position of each pixel in the first matching area. Finally, the matching unit 1003 may generate the matched area by removing the area containing the target object in the third matching area. In other words, the matched area is a part of the third matching area. Since the position of each pixel in the third matching area is completely corresponding to the position of each pixel in the first matching area, as a part of the third matching area, the position of each pixel in the matched area is also completely corresponding to the position of each pixel in the corresponding part of the first matching area.

[0095] The repair unit 1004 is used to generate a repaired image by replacing the area containing the target image in the image to be repaired with the matched area in the reference image. As described above, the positions of each pixel point in the matched area are completely corresponding to the positions of each pixel point in the corresponding part of the first matching area. Therefore, by replacing the pixel values ​​at the corresponding pixel point positions in the corresponding part of the first matching area with the pixel values ​​at the respective pixel point positions in the matched area, the area containing the target image in the image to be repaired can be repaired with the matched area.

[0096] In addition, after generating the patched image, depending on the specific application scenario and design requirements, the image patching device 1000 may further include: a reverse conversion unit 1005, which is used to generate a patched first panoramic image based on the patched image and other images in the first group of three-dimensional layout images. Of course, in other possible implementations, the reverse conversion unit 1005 may also be omitted. Fig.10 In FIG. 1 , the inverse conversion unit 1005 is shown in a dotted frame.

[0097] As another possible implementation, the image repair device 1000 may further include: a color calibration unit 1006, which is used to perform color calibration on the image to be repaired. Since in other possible implementations, the color calibration unit 1006 may also be omitted. Fig.10 In FIG. 1 , the color calibration unit 1006 is shown in a dotted box.

[0098] Specifically, for example, the color calibration unit 1006 may be configured to perform the following processing.

[0099] First, the color calibration unit 1006 transforms the image to be repaired and the reference image from the RGB color space to the LAB color space.

[0100] Then, the color calibration unit 1006 calculates the average value and variance of each channel of the image to be repaired and the reference image in the LAB color space respectively.

[0101] Next, the color calibration unit 1006 normalizes the value of each channel of the image to be repaired in the LAB color space based on the average value and variance of each channel of the image to be repaired in the LAB color space.

[0102] Then, the color calibration unit 1006 denormalizes the normalized value of each channel of the image to be repaired in the LAB color space based on the average value and variance of each channel of the reference image in the LAB color space.

[0103] Finally, the color calibration unit 1006 transforms the denormalized image to be repaired from the LAB color space back to the RGB color space.

[0104] By pre-calibrating the color of the image to be repaired before matching it with the reference image, the color difference between the image to be repaired and the reference image due to shooting conditions (such as shooting lighting, etc.) can be reduced. Therefore, when the image to be repaired is repaired with the matched area in the reference image, the color inconsistency in the repaired image is avoided.

[0105] Since the processing performed by each component in the image repair device according to the embodiment of the present disclosure completely corresponds to the image repair method described above, in order to avoid redundancy, the processing details performed by each component in the image repair device are not elaborated.

[0106] In addition, the image processing device according to the present disclosure may include: a memory for storing a computer program thereon; and a processor for performing the following processing when executing the computer program stored on the memory: determining a first group of three-dimensional layout maps and a second group of three-dimensional layout maps based on a first panoramic image and a second panoramic image taken for the same space, respectively, wherein the shooting positions of the first panoramic image and the second panoramic image are different, and the first group of three-dimensional layout maps and the second group of three-dimensional layout maps both include multiple images that can reflect the structure of the space; in the first group of three-dimensional layout maps, determining an image where the target object is located as the image to be repaired, and in the second group of three-dimensional layout maps, determining an image where the target object is located as a reference image, wherein the position of the target object in the space when the first panoramic image is taken is different from the position of the target object in the space when the second panoramic image is taken; in the reference image, determining a matched area that does not contain the target object and matches the area containing the target object in the image to be repaired; and generating a repaired image by replacing the area containing the target image in the image to be repaired with the matched area in the reference image.

[0107] Specifically, the method or device according to the embodiment of the present disclosure can also be used by Fig.11 The architecture of the computing device 1100 shown in FIG. Fig.11As shown, the computing device 1100 may include a bus 1110, one or more CPUs 1120, a read-only memory (ROM) 1130, a random access memory (RAM) 1140, a communication port 1150 connected to a network, an input / output component 1160, a hard disk 1170, etc. The storage device in the computing device 1100, such as the ROM 1130 or the hard disk 1170, may store various data or files used for processing and / or communication of the image repair method provided by the present disclosure, as well as program instructions executed by the CPU. Of course, Fig.11 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Fig.11 One or more components of a computing device are shown.

[0108] The embodiments of the present disclosure may also be implemented as a computer-readable storage medium. Computer-readable instructions are stored on the computer-readable storage medium according to the embodiments of the present disclosure. When the computer-readable instructions are executed by a processor, the image repair method according to the embodiments of the present disclosure described with reference to the above figures may be executed. The computer-readable storage medium includes, but is not limited to, for example, a volatile memory and / or a non-volatile memory. The volatile memory may, for example, include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, etc.

[0109] So far, reference has been made to Figures 1 to 11 The image repair method, device and medium according to the embodiments of the present disclosure are described in detail. In the image repair method, device and medium according to the present disclosure, by using reference images taken at different shooting positions and with the target object located at different positions in the same space to repair the area containing the target object in the image to be repaired, the target object can be removed while still retaining the original detail texture, so that the repaired image has no trace of repair and is more natural. In addition, the amount of calculation involved in the image repair method is small, and the processing process can be kept lightweight. In addition, by using the reference image to perform color calibration on the image to be repaired in advance before the repair process, the color consistency between the image to be repaired and the reference image can be maintained, avoiding the situation where the color difference in the repaired image causes unnatural repair.

[0110] It should be noted that, in this specification, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "includes..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0111] Finally, it should be noted that the above series of processing includes not only processing executed in time series in the order described here, but also processing executed in parallel or separately rather than in time series.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary hardware platform, and of course can also be implemented entirely by software. Based on such an understanding, all or part of the contribution of the technical solution of the present invention to the background technology can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0113] The present invention has been introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. An image repair method, comprising: Determining a first set of three-dimensional layout diagrams and a second set of three-dimensional layout diagrams based on a first panoramic image and a second panoramic image respectively obtained by taking pictures of the same space, wherein the first panoramic image and the second panoramic image are taken at different locations, and the first set of three-dimensional layout diagrams and the second set of three-dimensional layout diagrams both include a plurality of images capable of reflecting the structure of the space; In the first set of three-dimensional layout images, an image where a target object is located is determined as a to-be-painted image, and in the second set of three-dimensional layout images, an image where the target object is located is determined as a reference image, wherein a position of the target object in the space when the first panoramic image is taken is different from a position of the target object in the space when the second panoramic image is taken; In the reference image, determining a matched area that matches the area in the image to be repaired that contains the target object and does not contain the target object; as well as Generate a repaired image by replacing a region in the image to be repaired that includes the target image with a matched region in the reference image; Wherein, in the reference image, determining a matched area that matches an area in the image to be repaired that contains the target object and does not contain the target object comprises: Determine a first position and a second position of the target object in the image to be repaired and the reference image; Determine, in the image to be repaired and the reference image, respectively, overlapping areas including the first position and the second position as a first matching area and a second matching area; Performing feature point matching between the first matching area and the second matching area to determine a transformation matrix; Based on the transformation matrix, transforming the second matching area into a third matching area; and The matched area is generated by removing the area including the target object from the third matching area.

2. The method according to claim 1, wherein based on the first position and the second position, determining an overlapping area including the first position and the second position in the image to be repaired and the reference image as a first matching area and a second matching area, respectively, comprises: In the image to be repaired, marking the second position, determining a first area centered on the first position and a second area centered on the second position, and taking an overlapping area of ​​the first area and the second area as the first matching area; as well as In the reference image, the first position is marked, a third area centered on the first position and a fourth area centered on the second position are determined, and an overlapping area between the third area and the fourth area is used as the second matching area.

3. The method according to claim 2, wherein determining a first area centered on the first position and a second area centered on the second position comprises: Draw a rectangle as the first area with the first position as the center, a line connecting the first position and the second position as the diagonal direction, and a product of the distance between the first position and the second position and the first coefficient as half of the diagonal length; and A rectangle is drawn as the second area with the second position as the center, the line connecting the first position and the second position as the diagonal direction, and the product of the distance between the first position and the second position and the second coefficient as half of the diagonal length.

4. The method according to claim 1, wherein before determining the matched area, further comprising: Color calibration is performed on the image to be repaired.

5. The method according to claim 4, wherein color calibration is performed on the image to be repaired, comprising: Converting the image to be repaired and the reference image from the RGB color space to the LAB color space; Calculate the mean value and variance of each channel of the image to be repaired and the reference image in the LAB color space respectively; Normalizing the value of each channel of the image to be repaired in the LAB color space based on the average value and variance of each channel of the image to be repaired in the LAB color space; Denormalizing the normalized value of each channel of the image to be repaired in the LAB color space based on the average value and variance of each channel of the reference image in the LAB color space; as well as The denormalized image to be inpainted is transformed from the LAB color space back to the RGB color space.

6. The method according to claim 1, further comprising: after generating the patched image; A patched first panoramic image is generated based on the patched image and other images in the first set of three-dimensional layout images.

7. An image repair device, comprising: a conversion unit, configured to determine a first group of three-dimensional layout diagrams and a second group of three-dimensional layout diagrams based on a first panoramic image and a second panoramic image respectively obtained by taking pictures of the same space, wherein the first panoramic image and the second panoramic image are taken at different locations, and the first group of three-dimensional layout diagrams and the second group of three-dimensional layout diagrams both include a plurality of images capable of reflecting the structure of the space; a determining unit, configured to determine, in the first set of three-dimensional layout images, an image where a target object is located as an image to be repaired, and to determine, in the second set of three-dimensional layout images, an image where the target object is located as a reference image, wherein a position of the target object in the space when the first panoramic image is taken is different from a position of the target object in the space when the second panoramic image is taken; A matching unit, configured to determine, in the reference image, a matched area that matches an area in the image to be repaired that contains a target object and does not contain the target object; as well as a patching unit, configured to generate a patched image by replacing a region including the target image in the image to be patched with a matched region in the reference image; The matching unit is used for: Determine a first position and a second position of the target object in the image to be repaired and the reference image; Determine, in the image to be repaired and the reference image, respectively, overlapping areas including the first position and the second position as a first matching area and a second matching area; Performing feature point matching between the first matching area and the second matching area to determine a transformation matrix; Based on the transformation matrix, transforming the second matching area into a third matching area; as well as The matched area is generated by removing the area including the target object from the third matching area.

8. An image repair device, comprising: a memory for storing a computer program thereon; as well as A processor, configured to implement the following processing when executing the computer program: Determining a first set of three-dimensional layout diagrams and a second set of three-dimensional layout diagrams based on a first panoramic image and a second panoramic image respectively obtained by taking pictures of the same space, wherein the first panoramic image and the second panoramic image are taken at different locations, and the first set of three-dimensional layout diagrams and the second set of three-dimensional layout diagrams both include a plurality of images capable of reflecting the structure of the space; In the first set of three-dimensional layout images, an image where a target object is located is determined as a to-be-painted image, and in the second set of three-dimensional layout images, an image where the target object is located is determined as a reference image, wherein a position of the target object in the space when the first panoramic image is taken is different from a position of the target object in the space when the second panoramic image is taken; In the reference image, determining a matched area that matches the area in the image to be repaired that contains the target object and does not contain the target object; as well as Generate a repaired image by replacing a region in the image to be repaired that includes the target image with a matched region in the reference image; Wherein, in the reference image, determining a matched area that matches an area in the image to be repaired that contains the target object and does not contain the target object comprises: Determine a first position and a second position of the target object in the image to be repaired and the reference image; Determine, in the image to be repaired and the reference image, respectively, overlapping areas including the first position and the second position as a first matching area and a second matching area; Performing feature point matching between the first matching area and the second matching area to determine a transformation matrix; Based on the transformation matrix, transforming the second matching area into a third matching area; and The matched area is generated by removing the area including the target object from the third matching area.

9. A computer-readable recording medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is performed.

Images