A method of automatically removing non-moving objects from a photograph, processing system

Abstract

The present application provides a method and a processing system for automatically removing a static object from a photo, comprising: providing a reference photo containing at least one object to be removed from the reference photo; providing at least one continuous photo taken from another angle relative to the reference photo and containing at least one object to be removed from the reference photo; performing the following operations on each of the reference photo and the at least one continuous photo respectively: detecting the at least one object to be removed from the reference photo, dividing the photo into two layers; dividing the photo into tiles; calculating a cost function for each tile; finding a best combination of tiles based on all tiles of all photos by performing a global cost function optimization, and outputting a photo containing the best combination of tiles with a global function minimum value, wherein the output photo is the reference photo with the at least one object removed and replaced by a background from the at least one continuous photo.

Description

Technical Field

[0001] This invention relates to a method for automatically removing objects from images such as photographs or live camera previews, particularly for mobile devices equipped with cameras. The invention also relates to a processing system capable of implementing the method and a computer program product related to the method. Background Technology

[0002] Removing crowds and unwanted objects to improve photographic quality is one of the situations where photo editing or retouching is essential. In particular, many photographs are taken in locations where other objects are present besides the subject being photographed. Often, it's difficult to reach the subject, and it's obscured by other objects such as fences or trees. As a result, for example, fence bars or branches may appear in the photograph and obscure important parts of the scene, reducing the photograph's appeal to the photographer. This often happens in zoos, where iron fences or wire mesh separate animals from visitors, or in museums where exhibits are behind some kind of barrier. In these situations, it's usually impossible to photograph the subject without including certain objects, such as fences that partially obstruct the view of the subject (e.g., animals).

[0003] Therefore, it is well known in the prior art to edit and retouch individual scenes and photographs through inpainting. It is also well known in the prior art to edit and retouch multiple scenes through inpainting. Video inpainting is a field of computer vision that aims to utilize spatial and temporal information from neighboring scenes to remove objects or restore missing or contaminated areas in a video sequence. The primary goal is to generate inpainted areas that are seamlessly integrated into the video. This way, when the video is played back as a sequence, visual coherence is maintained, and distortion in the affected areas is not visible to the human eye.

[0004] There are other known solutions for editing single photos, typically used by professional graphic designers using high-quality tools (such as Photoshop), but these require a significant amount of manual work and a high level of expertise. There are also mass-market applications (such as TouchRetouch) that are much simpler, but of much lower quality and require manual selection of objects to be removed.

[0005] Generally speaking, all known solutions for removing objects or crowds from a single photo require manual intervention from the user.

[0006] Therefore, a high-quality, automated method is still needed for removing crowds or objects from a single photograph. Summary of the Invention

[0007] Based on the aforementioned prior art, the purpose of this invention is to develop a method for automatically removing objects from photographs. This method is firstly applicable to ensuring the automatic removal of unwanted objects, such as crowds, cars, or fences, from photographs without additional manual operation by the user. Secondly, this method can obtain high-quality output photographs.

[0008] The objective is achieved through a method for automatically removing objects from photographs, a processing system including an image processing pipeline based on the present invention, and related computer program products.

[0009] This invention provides crowd removal or object removal as a novel feature of cameras or any device with embedded cameras, which allows for the automatic removal of crowds or objects from photographs using deep neural networks and semantic methods (object detection, semantic segmentation, etc.).

[0010] According to a first aspect of the invention, a computer-implemented method for automatically removing stationary objects from photographs is provided. The method includes providing a reference photograph containing at least one stationary object to be removed from the reference photograph; providing at least one series of photographs taken from another angle relative to the reference photograph, each containing at least one stationary object to be removed from the reference photograph; and performing the following operations on each of the reference photograph and the at least one series of photographs: a) detecting the at least one stationary object to be removed from the reference photograph, wherein the photograph is divided into two layers, wherein the at least one object to be removed is assigned to a first layer, and the second layer includes elements to be output in a final photograph; b) segmenting all the photographs into tiles; c) calculating a cost function for each tile, wherein only tiles with a second layer are processed, and then proceeding to the next step of finding an optimal tile selection based on the tiles in all photographs by optimizing a global cost function, outputting a final photograph containing the selected tiles, the selected tiles being the optimal combination of tiles with the minimum value of the global cost function, the output final photograph being a reference photograph in which at least one stationary object has been removed and the stationary object has been replaced by the background of at least one of the series of photographs.

[0011] Advantageous extensions of the method for automatically removing objects from photographs based on the present invention are specifically described in the dependent claims.

[0012] The method according to the present invention may also have one or more of the following features, alone or in combination:

[0013] Steps a)-c) are executed immediately after each consecutive photograph is provided;

[0014] Use neural networks for semantic object detection to detect at least one immobile object in a reference photograph that needs to be removed.

[0015] After detecting at least one stationary object, an object mask is generated, wherein the object mask includes at least one object to be removed, and the object mask is a first layer, separate from a second layer, wherein the second layer includes elements to be output in the final image;

[0016] Before step a), align each of at least one consecutive photographs with a reference photograph;

[0017] Each photo is divided into at least 2×2 blocks;

[0018] The size of the tiles is adaptive;

[0019] The loopy belief propagation algorithm is used in the optimization step;

[0020] The steps to output a new photo include: blending;

[0021] The step of providing at least one consecutive photo ends automatically;

[0022] The alignment steps include: using a combined edge and corner detector to determine key points;

[0023] In each successive iteration of the global cost function optimization step, a new value of the cost function is calculated for all candidate tiles, which are the neighboring tiles of the tile that was replaced in the reference photo in the previous iteration.

[0024] According to a second aspect of the present invention, a processing system for automatically removing objects from photographs is provided. The system includes at least one memory and a processor configured to implement the steps of the method of the present invention.

[0025] The present invention also relates to a computer program product containing instructions that, when executed by a processor, cause the processor to perform the steps of the method of the present invention.

[0026] This invention allows for the automatic removal of crowds, fences, or other stationary objects from photographs. Because at least two photographs are acquired, information about the objects and the background can be collected in a timely manner.

[0027] Because a specific cost function is computed for each predefined sub-region of each consecutive image, it is possible to evaluate whether each different sub-region taken from other photos can fit into an edited reference photo.

[0028] Due to the global optimization of the context-based cost function, this invention allows for the editing of photographs, thereby ignoring areas that should not change or whose changes are insignificant in the context of a better photograph. Attached Figure Description

[0029] Other advantages and features of the present invention will become apparent from the accompanying drawings, in which:

[0030] Figure 1 The input and output photographs of the method according to the present invention are shown.

[0031] Figure 2 The key points found in each consecutive photograph are shown, which are used to align each consecutive photograph with a reference photograph.

[0032] Figure 3 A flowchart of a method according to an embodiment of the present invention is shown. Detailed Implementation

[0033] The various aspects of the present invention will now be described with reference to the accompanying drawings. In particular, Figure 3 A flowchart of the method, apparatus (system), and computer program product according to the present invention is shown. It will be understood that each block of the flowchart and combinations of blocks in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that the instructions are executed by the processor of that computer or the other programmable data processing apparatus to create means for implementing the functions / behaviors specified in the flowchart.

[0034] The overall idea of ​​this invention is to allow digital camera users not only to take single photos as in traditional camera mode, but also to continuously capture frames as needed, in order to obtain sufficient information for further processing. In a preferred solution, the digital camera user can slightly change position while capturing different frames, thereby obtaining different angles of the image.

[0035] This invention is based on the assumption that when different frames are captured from different angles, the object, or the object to be removed, hides different parts of the object of interest on different frames, allowing the removal of unwanted objects and the creation of an unobstructed image of the object of interest by combining elements from different frames. To some extent, each photograph is divided into two layers: the first layer contains the object to be removed, and the second layer contains the rest of the scene, including the object of interest. Preferably, the distance between the camera and the object to be removed is relatively small. Preferably, the distance between the object to be removed and the object of interest is greater than the distance between the camera and the object to be removed, so we can consider the second layer as a planar layer.

[0036] The system processes captured data online, using a deep neural network to detect people and / or objects and divides the photograph into several layers. The people and / or objects to be removed are assigned to the first layer, and the second layer includes elements to be output in the final photograph. This information allows for the removal of unwanted people and / or stationary objects from a reference photograph.

[0037] Figure 1 This reveals the purpose and result of a typical method for automatically removing objects from a photograph. It can be observed that the input image contains a fence that obscures a building, which is the object of interest. The method according to the invention receives an output image at the end containing only the building, while replacing the fence with a high-quality background. Figure 1 (The elements of the second layer in the left image, located behind and obscured by the fence).

[0038] Now refer to Figure 3 To describe the method of automatically removing objects from a photograph according to the present invention, such as... Figure 3 The flowchart shown illustrates the relevant steps. The method for automatically removing objects from photographs comprises several steps. These steps include: step 100, providing multiple photographs; image alignment step 200, performed in the alignment module; object detection step 300, performed in the object detection module; segmentation step 400, performed in the background modeling module; tile cost function calculation step 500, performed in the calculation module; global cost optimization step 600, performed in the optimization module; blending step 700, performed in the blending module; and final result step 800. All steps will be described in more detail below.

[0039] The method begins with step 100, which provides multiple photographs. Step 100 includes two sub-steps.

[0040] The first sub-step 101 is to provide a "reference photograph" by means of a camera. The "reference photograph" is the first photograph in a series of photographs taken continuously by the camera user. The reference photograph is a photograph in the series of photographs in which at least one stationary object will be removed and replaced with a background obtained from another photograph in the series.

[0041] In a preferred embodiment, the method of the present invention requires taking multiple photographs. In addition to the reference photograph provided in sub-step 101, a series of images are taken sequentially by the camera user from slightly different positions and angles within a predetermined time period in sub-step 102, providing at least one consecutive photograph. This set of images or series of images may include images provided by a standalone digital camera or a digital camera embedded in another digital device (such as a mobile phone). Images may be written to a separate memory before processing or may be taken directly from the camera.

[0042] Preferably, sub-step 102 of providing at least one continuous input image includes directly capturing a series of photographs and their parameters from the camera in dynamic mode. Alternatively, the reference photograph and at least one continuous photograph can be provided by reading it from memory, obtaining image parameters (particularly its size and resolution), and optionally displaying it on a display for user perception.

[0043] In one embodiment, if the method is executed dynamically, the user decides when to stop acquiring consecutive images. In practice, the acquisition process can last from 5 to 20 seconds. In another embodiment, the number of consecutive images to process is automatically determined based on the amount of information acquired; that is, the method automatically stops if the required amount of information has been collected to remove all detected objects.

[0044] The steps for processing each acquired photograph will be described below. Those skilled in the art will understand that these steps can be performed dynamically as each photograph arrives, or they can be performed in another technically reasonable order, provided all photographs are provided.

[0045] The next step in the method for automatically removing objects from a photograph is to perform image alignment step 200 in the alignment module. In this alignment step, each incoming image is aligned to a reference frame of the first image (i.e., the reference photograph) using keypoints and an ORB keypoint detector, which will be described in more detail below. Here, "reference frame" refers to the initial position of the reference photograph.

[0046] First, keypoints are determined in sub-step 201. For example, the Shi-Thomasi feature detector described in "Chris Harris and Mike Stephens. 'A Combined Corner and Edge Detector'. Alvey Vision Conference, 1988" is used for keypoint finding. Using this method, keypoints can be found very quickly, even on mobile devices. Keypoints are used to detect the relative positions between consecutive input images. The results are as follows... Figure 2 As shown.

[0047] Next, image alignment step 200 includes a matching sub-step 202. In this step, each input image from a set of consecutive images is matched using, for example, the RANSAC algorithm described in "MA Fischler, R.C. Bolles. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Comm. Of the ACM, 1981". After knowing the positions of these keypoints in each consecutive input image, each incoming image can be deformed and input into a reference frame of the first image, and then such aligned images can be used in further processing. By performing this alignment step 200, it is ensured that the content of each incoming input image can be objectively compared with the reference photograph (i.e., the first image). This is necessary because the user moves between taking two consecutive images and takes the same object at different coordinates in each photograph. It will be apparent to those skilled in the art that alignment step 200 will result in cutting certain portions of the consecutive images for further processing.

[0048] During image alignment step 200, the position of each frame fed into the alignment module is adjusted to match the previously collected frames. That is, the first image in the image set is adjusted to the reference photograph, the second image to the first image, and so on. This is accomplished by using a GMS matcher to match the keypoints of the image to be aligned, detected by the ORB keypoint detector, with keypoints found in the reference frame (the frame of the previous image). The detected keypoints must be unique and numerous enough to allow for robust matching, thus enabling homography to be found between frames.

[0049] The next step in the method for automatically removing objects from photographs is to perform an object detection step 300 in the semantic object detection module. In this detection step 300, object detection is performed on the reference photograph and each consecutive photograph using a neural network based on an efficient Det architecture to generate an object mask with the same resolution as the reference photograph. Specifically, in one embodiment, the object to be removed is a fence, and the produced mask is a fence mask. This fence is detected on the reference photograph and each consecutive photograph. In another embodiment, other types of objects, such as tree branches, can be detected. In the semantic object detection step 300, the reference photograph and other consecutive photographs are processed to detect objects, particularly portions of fences. In this step, the object detection algorithm implemented using the aforementioned neural network outputs regular rectangular regions of the photographs, and the dataset includes photographs with fence-marked portions. Of course, models can also be trained for other types of objects based on the marked dataset. In this description, such detected regions are called "regions of interest." All captured images and detected regions of interest are provided provisionally. Information about which parts of the photograph contain the detected object is contained in a specific data structure and is later assigned to each corresponding image sub-region extracted in the next step of the method of the present invention. During detection step 300, the detected objects to be removed are assigned to the first layer, while the elements that were not removed and are to be output in the final image are assigned to the second layer.

[0050] The aligned image and object mask (e.g., fence mask) are then forwarded to the background reconstruction module to reconstruct the background portion from the reference photo obscured by the fence.

[0051] The method then proceeds to segmentation step 400, dividing all the photographs taken in sub-steps 101 and 102 into tiles, and these tiles include the objects to be removed. Here, a "tile" refers to a two-dimensional (2D) rectangular sub-region of the received image, partitioned using a mesh of predetermined dimensions. Alternatively, a tile is a three-dimensional (3D) element generated by partitioning a 3D structure, which consists of a set of consecutive images. The first two dimensions of the 3D structure are the physical dimensions of each consecutive image, while the third dimension is time. Essentially, each spatial location in the mesh has multiple tiles collected from different points in time and from different angles. This multidimensional mesh can be represented as a Markov Random Field, where each tile is a node. After collecting multiple tiles from different points in time, each location in the mesh should have at least one object-free (fence-free) tile, which allows for the creation of an unobstructed background image created by tiles from the second layer, without needing to use the fenced tiles, i.e., the tiles from the first layer. If there are not enough unfenced tiles (tiles in the second layer) at each location in the grid, return to step 102, where feedback is sent to send new consecutive frames and then forwarded to the alignment module.

[0052] In general, the size of the tiles in 2D will be generated by a predetermined number of grids, which are the grids into which the image should be segmented in 2D. The number of tiles depends on the computing power of the hardware and can be adaptive. In one embodiment, the method for determining how to segment a photograph into tiles can be automated, for example, based on the hardware capacity of a specific mobile phone type. Preferably, the size and number of tiles are predetermined. Each photograph is divided into at least 2×2 tiles. For example, a photograph is divided into 21×11 tiles. For a photograph with a resolution of 1920×1080 and segmented into 21×11 tiles, the tile size is (1920 / 21)×(1080 / 11). Each tile of the partitioned photograph has adjacent tiles, also called adjacent tiles or contiguous tiles. Details of the relationships between tiles will be described in further detail in the specification.

[0053] Each photograph is divided into the same grid and has a predetermined number of rectangular sub-regions, meaning that each photograph is divided into exactly the same number of tiles. Those skilled in the art will understand that smaller tile sizes allow for more detailed processing. For example, if a fence has a denser grid or more clustered railings, more detailed processing is required. On the other hand, the above processing cannot be time-consuming; preferably, this processing is performed in real time, so the number of tiles determined under different circumstances is dynamically changing.

[0054] Therefore, the structure of step 400, which divides each photograph into tiles, is a photograph composed of regular rectangular mosaics. Tiles of the same size and coordinates appear on the reference photograph and other consecutive photographs. Tiles in consecutive photographs are called candidate tiles. Theoretically, each candidate tile can replace a tile with the same coordinates on the reference photograph. However, only tiles selected from the reference photograph can be replaced by candidate tiles in consecutive photographs; that is, tiles containing the portion of the object to be removed are replaced. Furthermore, each tile is assigned information about the presence of detected objects within its region. Therefore, the information output by the trained neural network model is further stored in the tiles and can also be used to check whether photograph acquisition can be stopped early. That is, if for each tile in the reference photograph in which an object to be removed is detected, and at least one candidate tile in at least one consecutive photograph does not contain the detected object, it means that enough information has been obtained for processing.

[0055] Once the image is segmented into tiles, information about the presence or absence of detected objects within a tile's region is assigned to each tile based on the neural network's output. This will further inform subsequent steps in the method, addressing which tile should be replaced and whether it should be replaced at all.

[0056] For each photograph, including the reference photograph, the method of the present invention performs a tile cost function calculation step 500, in which the cost function of each tile in the second layer is calculated. Each tile of each image is treated as a separate node of a Markov Random Field (MRF). In practice, this step results in receiving a specific value C(x) for each tile in a set of the second layers. p Each value represents the degree to which this tile matches a specific location in the reference photograph. A set of such values ​​is also calculated for tiles within the reference photograph.

[0057] The cost function for allocating candidate tiles can be expressed by the following formula:

[0058]

[0059] The formula defines the candidate patch x p Cost allocated to position p

[0060] The cost include:

[0061] cost Represents the plot x p Does it contain immobile objects?

[0062] Total cost V pq (x p x q) represents the tile x p The pixels and adjacent tiles x q The sum of squared differences between adjacent pixels. The total cost V pq (x p x q ) is calculated based on p and all its adjacent positions q;

[0063] q represents the adjacent positions of a given position p.

[0064] The parameters to consider when calculating the cost function are as follows:

[0065] Does the tile include a specific part of the immobile object detected in semantic detection step 300?

[0066] How well does the tile fit with adjacent tiles (it is more suitable for rectangles that fit each other)?

[0067] The applicant found that processing an input photo (including steps 100, 200, 300, 400, and 500) in dynamic mode takes approximately 500 milliseconds, which allows for real-time calculations and meets the requirements for commercial use.

[0068] The next step in the method of this invention is a global cost optimization step 600, which selects the optimal combination of tiles from all captured images to form a new image. In practice, this step is performed iteratively, selecting the tile combination with the lowest cost. By iteratively optimizing the global cost function, it can be determined whether each tile in the reference photo should be replaced with a tile having the same coordinates as one of the consecutive photos.

[0069] If the combination of the tiles is optimal, then:

[0070] Replace the stationary object with a background that would be visible if the stationary object were not present;

[0071] There are no abrupt transitions between the tiles;

[0072] There was no trace of the described immobile object;

[0073] There are no incomplete objects, such as part of a fence suspended "in the air".

[0074] In the global cost optimization step 600, any algorithm commonly used for optimizing MRFs can be utilized, such as the Loopy Belief Propagation algorithm. The choice of algorithm determines the trade-off between computational speed and the probability of convergence to the optimal configuration. However, in one embodiment, the following optimized greedy algorithm can be used.

[0075] The calculation is initialized using tiles obtained from a reference frame (reference photograph). Each tile is then examined iteratively in a random order. During this examination, all available versions of a particular tile (i.e., all other tiles in the mesh with the same coordinates) are compared to the tile in the reference photograph, and their costs are calculated. Tiles with lower costs replace tiles with higher costs. After a replacement, the neighboring regions of the replaced tile are recalculated. This means that the new values ​​of the cost function for all four neighboring tiles of the replaced tile are calculated.

[0076] In other words, in the first iteration of the optimization algorithm, the global cost function values ​​of all tiles from the reference photograph are known. Furthermore, for each candidate tile, a single cost function value is known, and the position of this candidate tile corresponds to a tile already considered in the reference photograph. In one iteration, theoretically, all individual tiles in the reference photograph are actually replaced by their corresponding candidate tiles in each subsequent photograph. This process is random in terms of the position of the tiles in the reference photographs from which the calculations are performed. However, if a replacement tile with a lower global cost function value is found, it signifies the end of an iteration. Then, another iteration begins, and the optimization step cannot be prematurely stopped until the tile with the lowest global cost function value is found.

[0077] In each successive iteration, all calculations of the global cost function are performed in the same manner. However, in each successive iteration preceding the next, if the previous iteration was computed, the cost functions of the four neighboring candidate tiles of the tile being replaced are calculated. This is because the cost function of a particular tile takes into account factors generated by neighboring tiles. This means that once a neighboring tile of a candidate tile in the reference photograph is replaced, the cost function value of that candidate tile becomes outdated. If the calculated value of the global cost function is low, the value of the global cost function can be correctly determined in the successive iterations only for updated input data, and then the replacement is performed again and the iteration stops, or a calculation is randomly performed on another tile in the reference photograph.

[0078] If the predetermined number of iterations has been performed, or if there are no more tiles in the reference image that would yield a lower global cost function value after replacement (no further replacements will occur), then the optimization algorithm is stopped.

[0079] Considering the possibility of convergence to a suboptimal solution, it is advisable to restart the algorithm multiple times with random initial values. The number of restarts should be proportional to the number of executable computations, which depends on the specifications of the final device implementing the method.

[0080] If there is no progress in the optimization process and the losses are not reduced, the global cost optimization will stop, which can be called premature termination.

[0081] The method then proceeds to blending step 700. This step is performed to output a final photograph in which at least one object is removed and replaced with the background. In one embodiment, once the region (tile) to be reconstructed is calculated in a specific iteration of the optimization step, it is blended into a reference photograph, for example, where real-time display is required. In another embodiment, blending is performed after the optimization step. Blending can be accomplished using any suitable algorithm, such as Poisson blending as described in "Patrick Perez, Michel Gangnet, Andrew Blake, 'Poisson Image Editing' 2014," where the aim is to obtain a very high-quality photograph. The Poisson blending algorithm can improve chromatic aberration and reduce the amount of artifacts. Blending the region (tile) into the reference photograph removes crowds and unwanted objects, generating the final result.

[0082] The processing system (not shown) of the present invention includes a memory and a processor, which is configured (by including multiple software modules) to perform the steps of the method described above. In one embodiment, the processing system is embedded in a mobile phone. According to the present invention, the mobile phone includes a display device (not shown) for displaying the output of the software modules. From the user's perspective, the processing system allows the user to perform the following operations:

[0083] The user begins removing objects and takes the first photo;

[0084] Users continue removing objects to collect data. More photos are collected;

[0085] The user stops removing objects and calculates the final result. The processing system uses the collected data to remove objects, such as fences, from the first image.

[0086] In another embodiment, the processing system may include a module for supporting a stably held camera. The module outputs camera position information, which is displayed by a display device. This increases the chance of obtaining a blur-free photograph of the same quality. In one embodiment, the display device displays a reference photograph with a border around the detected object and a camera position indicator. The user can see when enough information (e.g., based on the border color) has been acquired to remove a specific object. In another embodiment, the method of the present invention automatically stops if sufficient information has been collected to remove all detected objects to be removed.

[0087] Various aspects of the present invention can be implemented by a computer (or a device such as a CPU or MPU) having a system or device that reads and executes a computer program product stored on a storage device and performs the functions of the embodiments described above; and can be implemented by a method in which the steps of the method are performed by a computer having a system or device by reading and executing a program stored on a storage device to perform the functions of the embodiments described above. Therefore, the claimed computer program product can be provided to a computer via a network or various types of recording media as storage devices. The computer program product of the present invention also includes a non-transitory machine-readable medium.

[0088] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for automatically removing stationary objects from a photograph, the method being applied to a computer, characterized in that, include: A reference photograph is provided, which contains at least one stationary object to be removed from the reference photograph; Provide at least one consecutive photograph taken from another angle relative to the reference photograph, the consecutive photograph containing at least one stationary object to be removed from the reference photograph; For each of the reference photograph and at least one of the consecutive photographs, perform the following operations: a) Detect at least one of the stationary objects to be removed from the reference photograph, wherein the reference photograph is divided into two layers, wherein at least one of the stationary objects to be removed is assigned to the first layer, and the second layer includes elements to be output in the final photograph; b) Divide all the photos taken into several image tiles; c) Calculate the cost function for each tile, where only tiles with the second layer are processed; Based on the tiles in the captured photos, the optimal tiles are determined by optimizing the global cost function. The final photograph is output, which includes selected tiles that are the optimal combination of tiles with a minimum global cost function. The output final photograph is a reference photograph in which at least one of the stationary objects has been removed and replaced by the background of at least one of the consecutive photographs.

2. The method according to claim 1, characterized in that, The cost function is calculated according to the following formula: in, It is to set the tile x p The cost of assigning a position p to a given location; It defines whether a block contains a portion of the object to be removed; V pq (x p , x q ) is the sum of squared differences between the pixels of the tile x p and the neighboring pixels of the neighboring tile x q ​ q represents the adjacent positions of a given position p.

3. The method according to claim 1, characterized in that, After each of the consecutive photographs is provided, steps a) through c) are performed sequentially in a loop.

4. The method according to claim 1, characterized in that, The detection of at least one of the stationary objects removed from the reference photograph is performed based on semantic object detection using a neural network.

5. The method according to claim 1, characterized in that, The result of detecting at least one of the stationary objects is the generation of an object mask, wherein the object mask includes at least one object to be removed, the object mask being a first layer separate from a second layer, wherein the second layer includes elements to be output in the final photograph.

6. The method according to claim 1, characterized in that, Before step a), each of at least one of the consecutive photographs is aligned with the reference photograph.

7. The method according to claim 1, characterized in that, Divide each photo into at least 2×2 tiles.

8. The method according to claim 7, characterized in that, The size of the tile is adaptively adjusted.

9. The method according to claim 1, characterized in that, The optimization of the global cost function is based on the cyclic belief propagation algorithm.

10. The method according to claim 1, characterized in that, The steps for outputting a new photo include blending.

11. The method according to claim 1, characterized in that, The step of providing at least one consecutive photo ends automatically.

12. The method according to claim 6, characterized in that, The alignment step includes using a combined corner detector to determine key points.

13. The method according to claim 1, characterized in that, In each successive iteration of the global cost function optimization step, a new value of the cost function is calculated for all candidate tiles, which are adjacent tiles to the tile that was replaced in the reference photograph in the previous iteration.

14. A processing system for automatically removing stationary objects from photographs, characterized in that, include: A processor; as well as A memory connected to the processor stores instructions that, when executed by the processor, cause the processor to perform operations, including: A reference photograph is provided, which contains at least one stationary object to be removed from the reference photograph; Provide at least one consecutive photograph taken from another angle relative to the reference photograph, and the consecutive photograph contains the stationary object to be removed from the reference photograph; Perform the following operations on each of the reference photograph and at least one of the consecutive photographs: a) Detect the stationary object to be removed from the reference photograph, wherein the reference photograph is divided into two layers, wherein at least one object to be removed is assigned to the first layer, and the second layer includes elements to be output in the final photograph; b) Divide all the captured photos into tiles; c) Calculate the cost function for each tile, processing only tiles with the second layer. Next, based on the tiles in the captured photos, the optimal selection of the tiles is found by optimizing the global cost function. Output the final photograph, which contains selected tiles that are the optimal combination of tiles with the minimum global cost function. The output final photograph is a reference photograph in which at least one of the stationary objects has been removed and the stationary objects have been replaced by the background of at least one of the consecutive photographs.

15. The processing system according to claim 14, characterized in that, The cost function is calculated according to the following formula: in, It is to set the tile x p The cost of assigning a position p to a given location; It defines whether a block contains a portion of the object to be removed; V pq (x p x q ) is the tile x p With adjacent tile x q The sum of squared differences between adjacent pixels; q represents the adjacent positions of a given position p.

16. The processing system according to claim 14, characterized in that, In each successive iteration of the global cost function optimization step, a new value of the cost function is calculated for all candidate tiles, which are adjacent tiles to the tile that was replaced in the reference photograph in the previous iteration.

17. The processing system according to claim 14, characterized in that, Before step a), each of at least one of the consecutive photographs is aligned with the reference photograph.

18. A mobile phone, characterized in that, Includes the processing system described in claim 14.

19. A computer program product containing instructions, characterized in that, When the instruction is executed by the processor, it causes the processor to perform operations, including: A reference photograph is provided, which contains at least one stationary object to be removed from the reference photograph; Provide at least one consecutive photograph taken from another angle relative to the reference photograph, the consecutive photograph containing the stationary object to be removed from the reference photograph; Perform the following operations on each of the reference photograph and at least one of the consecutive photographs: a) Detect the stationary object to be removed from the reference photograph, wherein the reference photograph is divided into two layers, wherein at least one object to be removed is assigned to the first layer, and the second layer includes elements to be output in the final photograph; b) Divide all the captured photos into tiles; c) Calculate the cost function for each tile, where only tiles with the second layer are processed; Next, based on all the tiles in the captured photos, the optimal selection of the tiles is found by optimizing the global cost function; The final photograph is output, which includes selected tiles that are the optimal combination of tiles with a minimum global cost function. The output final photograph is a reference photograph in which at least one of the stationary objects has been removed and replaced by the background of at least one of the consecutive photographs.

20. The computer program product according to claim 19, characterized in that, Before step a), each of at least one of the consecutive photographs is aligned with the reference photograph.

Images