Video processing methods, apparatus, equipment, media and program products

Abstract

This disclosure provides a video processing method, apparatus, device, medium, and program product. The video processing method provided by this disclosure, in response to a trigger command, acquires a target image, wherein the target image is the previous video frame of a reference image, and the reference image is the currently acquired video frame by an image sensor. Then, the target area in the reference image containing the target object is filled using the background area in the target image to generate a processed image. This method, which uses only two adjacent frames for filling, ensures both the filling effect and the real-time performance of the filling process. Finally, the processed image is displayed as the current video frame, thus making the processed image an image after at least part of the target object has been removed from the reference image, achieving the effect of the target object disappearing in the video.

Description

Technical Field

[0001] This disclosure relates to the field of video processing technology, and in particular to a video processing method, apparatus, device, medium, and program product. Background Technology

[0002] With the development of communication technology and terminal equipment, various terminal devices such as mobile phones and tablets have become an indispensable part of people's work and life. Moreover, with the increasing popularity of terminal devices, video applications have become a major channel for communication and entertainment.

[0003] In video applications, foreground completion technology has wide applications in the field of digital entertainment special effects. Existing foreground completion technology usually uses deep learning methods for completion. Specifically, it generally uses optical flow networks and generative adversarial networks (GANs) to complete video frames.

[0004] However, the aforementioned deep learning-based completion method involves a large amount of computation during the completion process, and if real-time completion is required, the real-time performance cannot be guaranteed. Summary of the Invention

[0005] This disclosure provides a video processing method, apparatus, device, medium, and program product to solve the problem of difficulty in guaranteeing the real-time performance of foreground completion in current video interactive applications.

[0006] In a first aspect, embodiments of this disclosure provide a video processing method, including:

[0007] In response to a trigger command, a target image is acquired, wherein the target image is the previous video frame of the reference image, and the reference image is the video frame currently acquired by the image sensor.

[0008] The target region in the reference image is filled in using the background region in the target image to generate a processed image, wherein the target region is the area covered by the target object in the reference image;

[0009] The processed image is displayed as the current video frame, and the processed image is the image after at least part of the target object has been removed from the reference image.

[0010] In a second aspect, embodiments of this disclosure provide a video processing apparatus, comprising:

[0011] The image acquisition module is used to acquire a target image in response to a trigger command. The target image is the previous video frame of the reference image, and the reference image is the video frame currently acquired by the image sensor.

[0012] An image processing module is used to fill in the target region in the reference image using the background region in the target image to generate a processed image, wherein the target region is the area covered by the target object in the reference image;

[0013] An image display module is used to display the processed image as the current video frame, wherein the processed image is an image after at least part of the target object has been removed from the reference image.

[0014] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0015] Processor; and

[0016] Memory for storing the computer program of the processor;

[0017] A display for showing the video processed by the processor;

[0018] The processor is configured to implement the video processing method described in the first aspect and various possible designs of the first aspect by executing the computer program.

[0019] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the video processing method described in the first aspect and various possible designs of the first aspect.

[0020] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the video processing method as described in the first aspect and various possible designs of the first aspect.

[0021] This disclosure provides a video processing method, apparatus, device, medium, and program product. In response to a trigger command, a target image is acquired, where the target image is the previous video frame of a reference image, and the reference image is the currently acquired video frame by an image sensor. Then, the background area in the target image is used to fill in the target area of ​​the target object in the reference image to generate a processed image. This method, which uses only two adjacent frames for completion processing, ensures both the completion effect and the real-time performance of the completion process. Finally, the processed image is displayed as the current video frame, thus making the processed image an image after at least part of the target object has been removed from the reference image, achieving the effect of the target object disappearing in the video. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is an application scenario diagram illustrating a video processing method according to an example embodiment of the present disclosure;

[0024] Figure 2 This is a schematic flowchart illustrating a video processing method according to an example embodiment of the present disclosure;

[0025] Figure 3 This is a schematic flowchart illustrating a video processing method according to another exemplary embodiment of the present disclosure;

[0026] Figure 4 This is a schematic diagram showing the deformation of the square containing the pixel in this disclosure;

[0027] Figure 5 This is a schematic diagram of the similarity transformation of a triangle within the grid in this disclosure;

[0028] Figure 6 This is a schematic diagram of the structure of a video processing apparatus according to an exemplary embodiment of the present disclosure;

[0029] Figure 7 This is a schematic diagram of the structure of an electronic device according to an example embodiment of the present disclosure. Detailed Implementation

[0030] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0031] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0032] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0033] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0034] In video applications, foreground completion technology has wide applications in the field of digital entertainment special effects. Existing foreground completion techniques typically employ deep learning methods, specifically using a combination of optical flow networks and Generative Adversarial Networks (GANs) to complete video frames. However, deep learning-based completion methods involve significant computational overhead. Furthermore, existing methods generally optimize across multiple frames simultaneously, further increasing the computational load and making real-time performance even more difficult to guarantee.

[0035] This disclosure aims to provide a video processing method that, in response to a trigger command, acquires a target image, wherein the target image is the previous video frame of a reference image, and the reference image is the video frame currently acquired by an image sensor. Then, the target area in the reference image containing the target object is filled using the background area in the target image to generate a processed image. This method, which uses only two adjacent frames for completion processing, ensures both the completion effect and the real-time performance of the completion processing. Finally, the processed image is displayed as the current video frame, thereby making the processed image an image after at least part of the target object has been removed from the reference image, thus achieving the effect of the target object disappearing in the video.

[0036] Figure 1 This is an application scenario diagram illustrating a video processing method according to an example embodiment of this disclosure. For example... Figure 1 As shown, the video processing method provided in this embodiment can be executed by a terminal device equipped with a camera and a display screen. Specifically, the target object (e.g., vehicle, animal, plant, building, etc.) can be captured on video using a camera on the terminal device (e.g., front camera, rear camera, external camera, etc.).

[0037] Taking a vehicle as an example, in one possible scenario, when using a terminal device to film a video of a target vehicle, the camera on the terminal device is usually pointed at the target vehicle. Understandably, during the framing process, the camera will capture not only the target vehicle but also the target background. At this time, the user can input trigger commands into the terminal device (such as target gesture commands, target voice commands, target facial expression commands, target text commands, and target body commands, etc.) to remove the target object at least partially from the current video image.

[0038] It's worth noting that the trigger command can be a user-inputted command or a trigger command issued by the target object in the video. In this case, the video frame currently acquired by the image sensor is reference image 120. Then, the background area in the previous video frame of reference image 120, i.e., target image 110, is used to fill in the target area in the reference image, generating and displaying the processed image 121. Therefore, during the process of capturing or playing back the target object, a trigger command can be used to remove the target object from the reference image and perform the corresponding completion operation, thus achieving the visual effect of the target object disappearing from the reference image.

[0039] Figure 2 This is a schematic flowchart illustrating a video processing method according to an example embodiment of the present disclosure. Figure 2 As shown, the video processing method provided in this embodiment includes:

[0040] Step 101: In response to the trigger command, acquire the target image.

[0041] In one possible scenario, when using a terminal device to shoot a video of a target object, such as a target plant, the camera on the terminal device is usually pointed at the target object. Understandably, during the framing process, the camera will capture not only the target object but also the target background.

[0042] At this time, the terminal device responds to the trigger command and acquires the target image, where the target image is the previous video frame of the reference image, and the reference image is the video frame currently acquired by the image sensor. During the video capture of the target object, the user triggers the effect of the object disappearing from the image by inputting a target gesture command (e.g., a hand-reaching command). When the terminal device recognizes the target gesture command, it can trigger the effect of removing the target object from the reference image.

[0043] Step 102: Use the background region in the target image to fill in the target region in the reference image to generate the processed image.

[0044] After triggering the effect of removing the target object from the reference image, the background area in the target image is used to fill in the target area in the reference image to generate the processed image. The target area is the area covered by the target object in the reference image.

[0045] During the video recording of a target object, the user triggers the recording by inputting a hand gesture command. This triggers the effect of the target object disappearing from the current frame of the video in response to the user's gesture. Furthermore, it's worth noting that in this embodiment, the target object can be a plant, animal, building, etc., without specific limitations. Additionally, the trigger command can also be a gesture command, voice command, facial expression command, text command, body language command, etc., again without specific limitations.

[0046] Step 103: Display the processed image as the current video frame.

[0047] In this step, the processed image is displayed as the current video frame. The processed image is the image after the target object has been removed from the reference image, thereby achieving the effect of the target object disappearing in the current video frame.

[0048] In this embodiment, in response to a trigger command, a target image is acquired, wherein the target image is the previous video frame of the reference image, and the reference image is the video frame currently acquired by the image sensor. Then, the background area in the target image is used to fill in the target area where the target object is located in the reference image to generate a processed image. Thus, by using only two adjacent frames for completion processing, both the completion effect and the real-time performance of the completion processing can be guaranteed. Finally, the processed image is displayed as the current video frame, thereby making the processed image an image after at least part of the target object has been removed from the reference image, so as to achieve the effect of the target object disappearing in the video.

[0049] Figure 3 This is a schematic flowchart illustrating a video processing method according to another exemplary embodiment of the present disclosure.

[0050] like Figure 3 As shown, the video processing method provided in this embodiment includes:

[0051] Step 201: In response to the trigger command, acquire the target image.

[0052] In one possible scenario, when using a terminal device to capture video of a target object, the camera on the terminal device is usually pointed at the target object. Understandably, during the framing process, the camera will capture not only the target object but also the target background.

[0053] At this time, the terminal device responds to the trigger command and acquires the target image, where the target image is the previous video frame of the reference image, and the reference image is the video frame currently acquired by the image sensor. During the video capture of the target object, the user triggers the effect of the target object disappearing from the image by inputting a target gesture command (e.g., a hand-reaching command). When the terminal device recognizes the target gesture command, it can trigger the effect of removing the target object from the reference image.

[0054] Step 202: Determine the photometric error of each pixel in the target background relative to the corresponding pixel in the reference background, the grid deformation error of the grid vertex corresponding to each pixel in the target background determined by the similarity transformation, and the grid vertex inertia error of the grid vertex displacement regularization corresponding to each pixel in the target background.

[0055] In this step, after acquiring the target image and the reference image, the target foreground corresponding to the target image and the reference foreground corresponding to the reference image can be determined according to the preset foreground segmentation module. The target image includes the target foreground and the target background, and the reference image includes the reference foreground and the reference background. Mesh grids are then constructed for the target image, target foreground, reference image, and reference foreground respectively to generate the target image mesh, target foreground mesh, reference image mesh, and reference foreground mesh. It is worth noting that the aforementioned preset foreground segmentation module can be based on any foreground segmentation algorithm, aiming to estimate the foreground object mask from the target image using computer vision and graphics algorithms to determine the corresponding target foreground.

[0056] Then, the photometric error of each pixel in the target background relative to the corresponding pixel in the reference background is determined, the grid deformation error of the grid vertex corresponding to each pixel in the target background is determined by the similarity transformation, and the grid vertex inertia error of the grid vertex displacement regularization corresponding to each pixel in the target background is determined, where the target background corresponds to the background region in the target image and the reference background corresponds to the background region in the reference image.

[0057] The process of constructing the aforementioned grid can be as follows: image pyramids can be constructed for the target image, target foreground, reference image, and reference foreground respectively to generate target image pyramids, target foreground pyramids, reference image pyramids, and reference foreground pyramids with the same number of layers. Then, grids are constructed for each image layer in the target image pyramid, target foreground pyramid, reference image pyramid, and reference foreground pyramid respectively, wherein the grid shape and scale of corresponding levels in each image pyramid are the same.

[0058] Specifically, this can be done by separately analyzing the target image I. tar Target Prospect F tar Reference Image Iref and reference prospect F ref Construct an L-layer image pyramid, where L is typically set to 3, with layer 0 representing the top layer and layer L-1 representing the bottom layer.

[0059] For each of the image pyramids mentioned above, firstly, a grid for the top-level image is constructed, with each cell set to M×M pixels. Then, a grid is constructed in the target image space according to the cell size, ultimately generating W. grid ×H grid There are 1,000 grid vertices, where M is typically between 30 and 50 pixels. In each subsequent layer, the grid dimensions become twice the dimensions of the grid above it, until all L layers have two-dimensional grids of the same shape but different scales.

[0060] The determination of each pixel in the target background can be achieved by sampling at each layer of the image pyramid. Specifically, this can involve calculating the brightness gradient of each pixel in the first target image layer and traversing each pixel in the first target image layer according to a preset step size to determine the first target sample set. Pixels in the first target sample set belong to the first target background layer, and their corresponding brightness gradients are greater than a preset first threshold. The first target image layer and the first target background layer are located at the same level in the image pyramid. The preset step size can be N pixels, meaning that a pixel is taken every N pixels in the first target image layer. Furthermore, the traversal direction can be either along the horizontal or vertical direction of the first target image layer.

[0061] Then, the photometric error of each pixel on the first target image layer relative to the corresponding pixel in the reference background, the grid deformation error of each pixel on the first target image layer determined by the similarity transformation, and the grid vertex inertia error of each pixel on the first target image layer after displacement regularization are determined.

[0062] Specifically, the brightness gradient of the target image in layer L can be calculated separately. Pixels q in each target image layer are traversed at a step size s pixel intervals, where s is typically an integer between 1 and 5. If q is a background pixel, belonging to the background layer of the target image in layer L, and the gradient magnitude at its location exceeds a threshold t... g If the target image is selected, it is added to the corresponding L-layer target sample set, and each target image layer maintains its own sample set. The L-layer target sample set can be expressed as:

[0063] S L ={q|grad(q)>t g &&q∈Background L}

[0064] grad(q) represents the brightness gradient at pixel q, Background L For the target background of layer L, t g It can be set to 0.1.

[0065] Step 203: Determine the overall cost function based on photometric error, grid deformation error, and grid vertex inertia error.

[0066] To achieve mesh deformation optimization, this embodiment employs three types of overall cost functions, namely, the photometric error f based on sampling points. q Grid deformation error f d and mesh vertex inertia error f V .

[0067] Specifically, regarding the photometric error f g See the following formula:

[0068]

[0069] Among them, I tar Represents the target image, I ref Let q represent the reference image and q represent the target sample set S. L The pixel position in I tar (q) and I ref (q) represents the brightness values ​​at point q for the same target image layer and the reference image layer, respectively. represents the brightness gradient of the target image layer at point q.

[0070] Figure 4 This is a schematic diagram showing the deformation of the square containing the pixel in this disclosure. For example... Figure 4 As shown, let the four vertices of the square containing q be... Therefore, q can be represented as a linear combination of these four vertices (where the coefficient C...). k (Determined by bilinear interpolation), then q can be expressed as:

[0071]

[0072] As for q The displacement vector can then be expressed as:

[0073]

[0074] Continue to refer to Figure 4 Solid-lined quadrilaterals represent the squares before deformation, and dashed-lined quadrilaterals represent the squares after deformation. The deformation of the squares is achieved by vertex displacement. The interpolation coefficient C of q before and after deformation is... kIf the grid remains unchanged, the grid deformation will move q to a more suitable position q', thereby reducing the photometric error.

[0075] Regarding the grid deformation error f d See the following formula:

[0076]

[0077]

[0078] f d One component, used to represent l 12 With l 13 Compared to l' 12 with l' 13 The magnitude of the deformation error; f d Another component is used to represent l 12 With l 23 Compared to l' 12 with l' 23 The magnitude of the deformation error. Where, l 12 l 13 and l 23 It forms one of the triangles after the grid is divided, and l' 12 、l' 13 and l' 23 This forms the corresponding deformed triangle.

[0079] By controlling the grid deformation error, it can be ensured that the deformation of the squares within the grid conforms to the similarity transformation as much as possible, thus making the image filled in the foreground area look more natural and more in line with perspective transformation. Specifically, each square can be divided into two triangles, that is, the grid corresponding to each pixel in the target background is divided into a first triangle and a second triangle. The grid deformation error is then used to transform the first triangle into a deformed third triangle, and the second triangle into a deformed fourth triangle. Figure 5 This is a schematic diagram of the similarity transformation of a triangle within the grid in this disclosure, referencing... Figure 5 However, this can be achieved by minimizing as well as The values ​​are set to ensure that each triangle conforms to the similarity transformation during mesh deformation as much as possible.

[0080] Specifically, {V k {k = 1, 2, 3} and {V} k {l}, k = 1, 2, 3} represent the vertices of the triangle before and after the transformation, respectively. k1k2 ,k1k2=12,13,23} and {l k1k2Let ', k1k2=1, 2, 3} represent the side lengths of the triangle before and after the transformation.

[0081] Regarding the inertial error f V See the following formula:

[0082] f V =||t V ||2

[0083] Among them, t V ∈R 2 This represents the displacement vector of mesh vertex V, which is used to regularize the mesh vertex displacement through inertial error, thereby suppressing excessive mesh vertex displacement.

[0084] Subsequently, based on the aforementioned photometric error, grid deformation error, and inertial error, the overall cost function can be constructed:

[0085]

[0086] Therefore, using the overall cost function described above, the optimization variable obtained is the displacement vector t of each grid vertex. V The optimization algorithm can utilize the Gauss-Newton or Levenberg-Marquardt (LM) algorithm to obtain the displacement vectors of the mesh vertices, thereby achieving mesh deformation. Furthermore, λ1 can take the value 1, λ2 can take the value 0.2-0.5, and λ3 can take the value 0.1.

[0087] Step 204: Determine the displacement vector of the grid vertex corresponding to each pixel in the target background based on the overall cost function and the preset optimization algorithm.

[0088] The displacement vector of the grid vertex corresponding to each pixel in the target background is determined based on the overall cost function and a preset optimization algorithm. The target image is then adjusted according to the displacement vector to generate an optimized target image. In one possible implementation, the corresponding pixels in the first target sample set are subjected to grid deformation based on the displacement vector. The deformed result is then passed to the next target image layer in the target image pyramid, up to the bottom layer, to generate the optimized target image. The first target image layer is the top layer of the target image pyramid, and the direction of passing the deformed result is from the top layer to the bottom layer of the target image pyramid.

[0089] The process of passing the deformed mesh result to the next target image layer in the target image pyramid can be done in a coarse-to-fine manner. That is, starting from the top-level mesh, the deformed mesh vertices are propagated layer by layer to the next layer for mesh vertex position optimization until the bottom-level deformed mesh vertices are obtained. Understandably, when the first target image layer is the first layer of the image pyramid, mesh deformation can be performed on the corresponding pixels in the first target sample set based on the displacement vector, and the deformed mesh result is passed to the second target image layer in the target image pyramid. In other words, the mesh deformation result of the first target image layer is used as the initial mesh of the second target image layer before mesh deformation. Then, the same mesh deformation is performed on the second target image layer, and the deformed mesh result is passed to the third target image layer in the target image pyramid. The deformed mesh result of the second target image layer is used as the initial mesh of the third target image layer before mesh deformation. Finally, the same mesh deformation is performed on the third target image layer, and the deformed mesh result is passed to the fourth target image layer in the target image pyramid. The deformed mesh result of the third target image layer is used as the initial mesh of the fourth target image layer before mesh deformation.

[0090] Furthermore, it is worth noting that when dividing the image layers of the pyramid into grids, the length and width of each grid cell in the next lower layer become twice the length and width of the grid cells in the image above it. Therefore, when transferring the optimized vertex position of the upper layer grid to the lower layer image, a scale-up process is required, i.e., V” = 2·V', where V” is the vertex position of the lower layer grid and V' is the vertex position of the upper layer grid.

[0091] Step 205: Adjust the target image according to the displacement vector to generate an optimized target image.

[0092] After determining the displacement vector of the grid vertex corresponding to each pixel in the target background based on the overall cost function and the preset optimization algorithm, and adjusting the target image according to the displacement vector, the optimization of the lowest-level deformable grid vertex is completed, and the optimized target image can be generated.

[0093] Step 206: Calculate the dense optical flow of the reference image relative to the optimized target image.

[0094] Step 207: Fill in the target region in the reference image according to the dense optical flow.

[0095] After generating the optimized target image, the dense optical flow of the reference image relative to the optimized target image can be calculated. The target area in the reference image is then filled in based on the dense optical flow. During the target video recording process, the user triggers a command, which enables the terminal device to respond to the user's gesture, thereby achieving the effect of the target object disappearing in the current frame of the video.

[0096] Furthermore, in this embodiment, by employing a strategy of incomplete between adjacent frames, both the incomplete effect and the real-time performance of the incomplete processing can be guaranteed. Additionally, grid optimization ensures that the calculated dense optical flow guarantees low photometric error in the overlapping background region between the target image and the reference image, while also ensuring that adjacent pixels share a consistent optical flow vector, thereby effectively predicting the pixel optical flow in the foreground region. Moreover, by constructing an image pyramid and then sequentially propagating the deformation of the grid vertices of each layer—starting from the top layer of the image pyramid and propagating the deformation of the grid vertices from the previous layer to the next, and so on—a multi-scale processing strategy is formed to accelerate optical flow calculation.

[0097] In the video processing method described in the above embodiments, during the real-time foreground completion of the reference image using the target image, the use of bilinear interpolation leads to a gradual blurring of the rendered image quality in the foreground area under static conditions. Therefore, based on the above embodiments, before using the background area in the target image to complete and fill the target area in the reference image, it is necessary to first determine that the image acquisition state is a motion acquisition state. Specifically, this can be achieved by calculating the displacement of each grid vertex in the target image relative to its corresponding grid vertex in the target image through steps 201 to 204 in the above embodiments, and then calculating the average displacement. If the average displacement is greater than or equal to a preset displacement threshold, the image acquisition state is determined to be a motion acquisition state. The displacement threshold can be set to a length of 0.3-0.5 pixels.

[0098] If the determined average displacement is less than the preset displacement threshold, then the image acquisition state is a stationary acquisition state, and the target image I is... tar Set as background image I bg Target prospect F tar Set as background foreground F bg For newly added video frames, if the image acquisition state is static, the background image is used for completion; otherwise, a moving foreground completion strategy is adopted. Figure 2 or Figure 3 The steps in the illustrated embodiment.

[0099] Based on background image I bg The completion algorithm is as follows:

[0100] The first step is to update the background image. This can be done every 5-10 frames, updating the background image I using the background region of the reference image. bg and background foreground F bg .

[0101] The second step is to complete the image using a reference image. Use the background image I... bg The background area is filled with the foreground area of ​​the reference image, and the foreground template of the reference image is updated at the same time.

[0102] In the above embodiments, the image acquisition state is determined by judging the relationship between the average displacement of the grid vertices and the preset displacement threshold; then, the corresponding foreground completion strategy is selected according to the determined image acquisition state, which can ensure the effect and real-time performance of foreground completion in the motion acquisition state, and also ensure the effect of foreground completion in the static acquisition state.

[0103] Figure 6 This is a schematic diagram of the structure of a video processing apparatus according to an example embodiment of the present disclosure. Figure 6 As shown, the video processing apparatus 300 provided in this embodiment includes:

[0104] The image acquisition module 301 is used to acquire a target image in response to a trigger command. The target image is the previous video frame of the reference image, and the reference image is the video frame currently acquired by the image sensor.

[0105] Image processing module 302 is used to fill in the target region in the reference image using the background region in the target image to generate a processed image, wherein the target region is the area covered by the target object in the reference image;

[0106] Display 303 is used to display the processed image as the current video frame, wherein the processed image is an image after at least part of the target object has been removed from the reference image.

[0107] According to one or more embodiments of this disclosure, the image processing module 302 is specifically used for:

[0108] The photometric error of each pixel in the target background relative to the corresponding pixel in the reference background is determined, the grid deformation error of the grid vertex corresponding to each pixel in the target background is determined by similarity transformation, and the grid vertex inertia error of the grid vertex corresponding to each pixel in the target background is determined by displacement regularization, wherein the target background corresponds to the background region in the target image, and the reference background corresponds to the background region in the reference image;

[0109] The overall cost function is determined based on the photometric error, the grid deformation error, and the grid vertex inertia error.

[0110] The displacement vector of each grid vertex corresponding to each pixel in the target background is determined according to the overall cost function and the preset optimization algorithm, so as to generate an optimized target image after adjusting the target image according to the displacement vector;

[0111] Calculate the dense optical flow of the reference image relative to the optimized target image, so as to fill in the target region in the reference image according to the dense optical flow.

[0112] According to one or more embodiments of this disclosure, the image processing module 302 is further configured to:

[0113] Determine the target foreground corresponding to the target image and the reference foreground corresponding to the reference image, wherein the target image includes the target foreground and the target background, and the reference image includes the reference foreground and the reference background;

[0114] Mesh grids are constructed for the target image, the target foreground, the reference image, and the reference foreground, respectively, to generate a target image mesh, a target foreground mesh, a reference image mesh, and a reference foreground mesh.

[0115] According to one or more embodiments of this disclosure, the image processing module 302 is specifically used for:

[0116] Image pyramids are constructed for the target image, the target foreground, the reference image, and the reference foreground, respectively, to generate target image pyramids, target foreground pyramids, reference image pyramids, and reference foreground pyramids with the same number of layers;

[0117] Mesh grids are constructed for each image layer in the target image pyramid, the target foreground pyramid, the reference image pyramid, and the reference foreground pyramid, respectively, wherein the mesh shape and scale of the corresponding layer in each image pyramid are the same.

[0118] According to one or more embodiments of this disclosure, the image processing module 302 is specifically used for:

[0119] Calculate the brightness gradient of each pixel on the first target image layer, and traverse each pixel on the first target image layer according to a preset step size to determine the first target sample set. The pixels in the first target sample set belong to the first target background layer and the corresponding brightness gradient is greater than a preset first threshold. The first target image layer and the first target background layer are located at the same level in the image pyramid.

[0120] The photometric error of each pixel on the first target image layer relative to the corresponding pixel in the reference background is determined, the grid deformation error of the grid vertices corresponding to each pixel on the first target image layer is determined by similarity transformation, and the grid vertex inertia error of the grid vertex displacement regularization of each pixel on the first target image layer is determined.

[0121] According to one or more embodiments of this disclosure, the image processing module 302 is specifically used for:

[0122] The corresponding pixels in the first target sample set are subjected to mesh deformation according to the displacement vector, and the mesh deformation result is passed to the next target image layer in the target image pyramid, until the bottom layer of the target image pyramid, to generate an optimized target image, wherein the first target image layer is the top layer of the target image pyramid.

[0123] According to one or more embodiments of this disclosure, the image processing module 302 is specifically used for:

[0124] The grid corresponding to each pixel in the target background is divided into a first triangle and a second triangle;

[0125] Based on the grid deformation error, the first triangle is transformed into a deformed third triangle, and the second triangle is transformed into a deformed fourth triangle.

[0126] According to one or more embodiments of the present disclosure, the image processing module 302 is further configured to: determine that the image acquisition state is a motion acquisition state.

[0127] According to one or more embodiments of this disclosure, the image processing module 302 is specifically used for:

[0128] Calculate the average displacement of each grid vertex in the optimized target image relative to the corresponding grid vertex in the target image;

[0129] If the average displacement is greater than or equal to a preset displacement threshold, then the image acquisition state is determined to be the motion acquisition state.

[0130] It is worth noting that Figure 6 The video processing apparatus provided in the illustrated embodiment can be used to execute the method steps provided in any of the above method embodiments. The specific implementation and technical effects are similar, and will not be described again here.

[0131] Figure 7 This is a schematic diagram illustrating the structure of an electronic device according to an example embodiment of the present disclosure. Figure 7The diagram illustrates a structural schematic of an electronic device 400 suitable for implementing embodiments of the present disclosure. The terminal devices in these embodiments may include, but are not limited to, mobile terminals with image acquisition capabilities such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, and fixed terminals with external image acquisition devices such as digital TVs, desktop computers, and smart home devices. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0132] like Figure 7 As shown, the electronic device 400 may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a memory 408 into a random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the electronic device 400. The processor 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404. The memory is used to store programs for executing the video processing methods described in the various method embodiments above; the processor is configured to execute the programs stored in the memory.

[0133] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows electronic device 400 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 400 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0134] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer-readable storage medium comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the video processing method shown in the flowcharts of embodiments of this disclosure. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 409, or installed from a storage device 408, or installed from a ROM 402. When the computer program is executed by the processor 401, the video processing functions defined in the methods of embodiments of this disclosure are performed.

[0135] It should be noted that the computer-readable storage medium described in this disclosure can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0136] The aforementioned computer-readable storage medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0137] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: in response to a trigger command, acquire a target image, wherein the target image is a previous video frame of a reference image, and the reference image is a video frame currently acquired by an image sensor; fill in the target region in the reference image using the background region in the target image to generate a processed image, wherein the target region is the area covered by the target object in the reference image; and display the processed image as the current video frame, wherein the processed image is an image after at least part of the target object has been removed from the reference image.

[0138] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0140] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0141] The modules described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a module does not necessarily limit the unit itself; for example, a display module can also be described as "a unit that displays the object face and a sequence of face masks".

[0142] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0143] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0144] In a first aspect, according to one or more embodiments of this disclosure, a video processing method is provided, comprising:

[0145] In response to a trigger command, a target image is acquired, wherein the target image is the previous video frame of the reference image, and the reference image is the video frame currently acquired by the image sensor.

[0146] The target region in the reference image is filled in using the background region in the target image to generate a processed image, wherein the target region is the area covered by the target object in the reference image;

[0147] The processed image is displayed as the current video frame, and the processed image is the image after at least part of the target object has been removed from the reference image.

[0148] According to one or more embodiments of this disclosure, the step of filling in the target region in the reference image using the background region in the target image includes:

[0149] The photometric error of each pixel in the target background relative to the corresponding pixel in the reference background is determined, the grid deformation error of the grid vertex corresponding to each pixel in the target background is determined by similarity transformation, and the grid vertex inertia error of the grid vertex corresponding to each pixel in the target background is determined by displacement regularization, wherein the target background corresponds to the background region in the target image, and the reference background corresponds to the background region in the reference image;

[0150] The overall cost function is determined based on the photometric error, the grid deformation error, and the grid vertex inertia error.

[0151] The displacement vector of each grid vertex corresponding to each pixel in the target background is determined according to the overall cost function and the preset optimization algorithm, so as to generate an optimized target image after adjusting the target image according to the displacement vector;

[0152] Calculate the dense optical flow of the reference image relative to the optimized target image;

[0153] The target region in the reference image is filled in according to the dense optical flow.

[0154] According to one or more embodiments of this disclosure, before determining the photometric error of each pixel in the target background relative to the corresponding pixel in the reference background, the grid deformation error of the grid vertices corresponding to each pixel in the target background determined by similarity transformation, and the grid vertex inertia error of the grid vertex displacement regularization corresponding to each pixel in the target background, the method further includes:

[0155] Determine the target foreground corresponding to the target image and the reference foreground corresponding to the reference image, wherein the target image includes the target foreground and the target background, and the reference image includes the reference foreground and the reference background;

[0156] Mesh grids are constructed for the target image, the target foreground, the reference image, and the reference foreground, respectively, to generate a target image mesh, a target foreground mesh, a reference image mesh, and a reference foreground mesh.

[0157] According to one or more embodiments of this disclosure, constructing meshes for the target image, the target foreground, the reference image, and the reference foreground to generate a target image mesh, a target foreground mesh, a reference image mesh, and a reference foreground mesh includes:

[0158] Image pyramids are constructed for the target image, the target foreground, the reference image, and the reference foreground, respectively, to generate target image pyramids, target foreground pyramids, reference image pyramids, and reference foreground pyramids with the same number of layers;

[0159] Mesh grids are constructed for each image layer in the target image pyramid, the target foreground pyramid, the reference image pyramid, and the reference foreground pyramid, respectively, wherein the mesh shape and scale of the corresponding layer in each image pyramid are the same.

[0160] According to one or more embodiments of this disclosure, determining the photometric error of each pixel in the target background relative to a corresponding pixel in the reference background, the grid deformation error of the grid vertices corresponding to each pixel in the target background determined by similarity transformation, and the grid vertex inertia error of the grid vertex displacement regularized for each pixel in the target background, includes:

[0161] Calculate the brightness gradient of each pixel on the first target image layer, and traverse each pixel on the first target image layer according to a preset step size to determine the first target sample set. The pixels in the first target sample set belong to the first target background layer and the corresponding brightness gradient is greater than a preset first threshold. The first target image layer and the first target background layer are located at the same level in the image pyramid.

[0162] The photometric error of each pixel on the first target image layer relative to the corresponding pixel in the reference background is determined, the grid deformation error of the grid vertices corresponding to each pixel on the first target image layer is determined by similarity transformation, and the grid vertex inertia error of the grid vertex displacement regularization of each pixel on the first target image layer is determined.

[0163] According to one or more embodiments of this disclosure, the step of determining the displacement vector of the grid vertex corresponding to each pixel in the target background based on the overall cost function and a preset optimization algorithm, so as to adjust the target image according to the displacement vector and generate an optimized target image, includes:

[0164] The corresponding pixels in the first target sample set are subjected to mesh deformation according to the displacement vector, and the mesh deformation result is passed to the next target image layer in the target image pyramid, until the bottom layer of the target image pyramid, to generate an optimized target image, wherein the first target image layer is the top layer of the target image pyramid.

[0165] According to one or more embodiments of this disclosure, after determining the grid deformation error of each pixel in the target background corresponding to the grid vertex according to the similarity transformation, the method further includes:

[0166] The grid corresponding to each pixel in the target background is divided into a first triangle and a second triangle;

[0167] Based on the grid deformation error, the first triangle is transformed into a deformed third triangle, and the second triangle is transformed into a deformed fourth triangle.

[0168] According to one or more embodiments of this disclosure, before filling the target region in the reference image using the background region in the target image, the method further includes:

[0169] The image acquisition state is determined to be a motion acquisition state.

[0170] According to one or more embodiments of this disclosure, determining the image acquisition state as a motion acquisition state includes:

[0171] Calculate the average displacement of each grid vertex in the optimized target image relative to the corresponding grid vertex in the target image;

[0172] If the average displacement is greater than or equal to a preset displacement threshold, then the image acquisition state is determined to be the motion acquisition state.

[0173] Secondly, according to one or more embodiments of the present disclosure, a video processing apparatus is provided, comprising:

[0174] The image acquisition module is used to acquire a target image in response to a trigger command. The target image is the previous video frame of the reference image, and the reference image is the video frame currently acquired by the image sensor.

[0175] An image processing module is used to fill in the target region in the reference image using the background region in the target image to generate a processed image, wherein the target region is the area covered by the target object in the reference image;

[0176] A display for displaying the processed image as the current video frame, wherein the processed image is an image after at least part of the target object has been removed from the reference image.

[0177] According to one or more embodiments of this disclosure, the image processing module is specifically used for:

[0178] The photometric error of each pixel in the target background relative to the corresponding pixel in the reference background is determined, the grid deformation error of the grid vertex corresponding to each pixel in the target background is determined by similarity transformation, and the grid vertex inertia error of the grid vertex corresponding to each pixel in the target background is determined by displacement regularization, wherein the target background corresponds to the background region in the target image, and the reference background corresponds to the background region in the reference image;

[0179] The overall cost function is determined based on the photometric error, the grid deformation error, and the grid vertex inertia error.

[0180] The displacement vector of each grid vertex corresponding to each pixel in the target background is determined according to the overall cost function and the preset optimization algorithm, so as to generate an optimized target image after adjusting the target image according to the displacement vector;

[0181] Calculate the dense optical flow of the reference image relative to the optimized target image, so as to fill in the target region in the reference image according to the dense optical flow.

[0182] According to one or more embodiments of this disclosure, the image processing module is further configured to:

[0183] Determine the target foreground corresponding to the target image and the reference foreground corresponding to the reference image, wherein the target image includes the target foreground and the target background, and the reference image includes the reference foreground and the reference background;

[0184] Mesh grids are constructed for the target image, the target foreground, the reference image, and the reference foreground, respectively, to generate a target image mesh, a target foreground mesh, a reference image mesh, and a reference foreground mesh.

[0185] According to one or more embodiments of this disclosure, the image processing module is specifically used for:

[0186] Image pyramids are constructed for the target image, the target foreground, the reference image, and the reference foreground, respectively, to generate target image pyramids, target foreground pyramids, reference image pyramids, and reference foreground pyramids with the same number of layers;

[0187] Mesh grids are constructed for each image layer in the target image pyramid, the target foreground pyramid, the reference image pyramid, and the reference foreground pyramid, respectively, wherein the mesh shape and scale of the corresponding layer in each image pyramid are the same.

[0188] According to one or more embodiments of this disclosure, the image processing module is specifically used for:

[0189] Calculate the brightness gradient of each pixel on the first target image layer, and traverse each pixel on the first target image layer according to a preset step size to determine the first target sample set. The pixels in the first target sample set belong to the first target background layer and the corresponding brightness gradient is greater than a preset first threshold. The first target image layer and the first target background layer are located at the same level in the image pyramid.

[0190] The photometric error of each pixel on the first target image layer relative to the corresponding pixel in the reference background is determined, the grid deformation error of the grid vertices corresponding to each pixel on the first target image layer is determined by similarity transformation, and the grid vertex inertia error of the grid vertex displacement regularization of each pixel on the first target image layer is determined.

[0191] According to one or more embodiments of this disclosure, the image processing module is specifically used for:

[0192] The corresponding pixels in the first target sample set are subjected to mesh deformation according to the displacement vector, and the mesh deformation result is passed to the next target image layer in the target image pyramid, until the bottom layer of the target image pyramid, to generate an optimized target image, wherein the first target image layer is the top layer of the target image pyramid.

[0193] According to one or more embodiments of this disclosure, the image processing module is specifically used for:

[0194] The grid corresponding to each pixel in the target background is divided into a first triangle and a second triangle;

[0195] Based on the grid deformation error, the first triangle is transformed into a deformed third triangle, and the second triangle is transformed into a deformed fourth triangle.

[0196] According to one or more embodiments of this disclosure, the image processing module is further configured to: determine that the image acquisition state is a motion acquisition state.

[0197] According to one or more embodiments of this disclosure, the image processing module is specifically used for:

[0198] Calculate the average displacement of each grid vertex in the optimized target image relative to the corresponding grid vertex in the target image;

[0199] If the average displacement is greater than or equal to a preset displacement threshold, then the image acquisition state is determined to be the motion acquisition state.

[0200] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0201] Processor; and

[0202] Memory for storing the computer program of the processor;

[0203] A display for showing the video processed by the processor;

[0204] The processor is configured to implement the video processing method described in the first aspect and various possible designs of the first aspect by executing the computer program.

[0205] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the video processing method described in the first aspect and various possible designs of the first aspect.

[0206] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the video processing method as described in the first aspect and various possible designs of the first aspect.

[0207] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0208] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0209] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method of video processing, the method comprising: The method comprises the following steps: In response to a trigger instruction, a target image is acquired, the target image being a previous video frame of a reference image, the reference image being a current video frame acquired by an image sensor; A target region in the reference image is filled and completed by using a background region in the target image, the target region being a region covered by a target object in the reference image; The processed image is displayed as a current video frame, the processed image being an image in which the target object is at least partially removed from the reference image; The filling and completion of the target region in the reference image by using the background region in the target image comprises the following steps: A photometric error of each pixel in a target background with respect to a corresponding pixel in a reference background is determined, a grid deformation error of a grid vertex corresponding to each pixel in the target background according to a similarity transformation is determined, and a grid vertex inertia error of the grid vertex corresponding to each pixel in the target background is regularized by displacement, wherein the target background corresponds to the background region in the target image, and the reference background corresponds to the background region in the reference image; A total cost function is determined according to the photometric error, the grid deformation error, and the grid vertex inertia error; A displacement vector of the grid vertex corresponding to each pixel in the target background is determined according to the total cost function and a preset optimization algorithm, so that the target image is adjusted according to the displacement vector to generate an optimized target image; A dense optical flow of the reference image with respect to the optimized target image is calculated; The target region in the reference image is filled and completed according to the dense optical flow.

2. The video processing method of claim 1, wherein, Before the determination of the photometric error of each pixel in the target background with respect to the corresponding pixel in the reference background, the grid deformation error of the grid vertex corresponding to each pixel in the target background according to the similarity transformation, and the grid vertex inertia error of the grid vertex corresponding to each pixel in the target background regularized by displacement, the method further comprises the following steps: A target foreground corresponding to the target image and a reference foreground corresponding to the reference image are determined, the target image comprising the target foreground and the target background, and the reference image comprising the reference foreground and the reference background; Grids are respectively constructed for the target image, the target foreground, the reference image, and the reference foreground to generate a target image grid, a target foreground grid, a reference image grid, and a reference foreground grid.

3. The video processing method of claim 2, wherein, The construction of the grids for the target image, the target foreground, the reference image, and the reference foreground to generate the target image grid, the target foreground grid, the reference image grid, and the reference foreground grid comprises the following steps: Image pyramids are respectively constructed for the target image, the target foreground, the reference image, and the reference foreground to generate target image pyramids, target foreground pyramids, reference image pyramids, and reference foreground pyramids with the same number of layers. Grids are respectively constructed for each image layer in the target image pyramid, the target foreground pyramid, the reference image pyramid and the reference foreground pyramid, wherein the grids of the corresponding layers in each image pyramid have the same shape and scale.

4. The video processing method of claim 3, wherein, The luminance error of each pixel in the target background relative to the corresponding pixel in the reference background, the grid deformation error of the corresponding grid vertex of each pixel in the target background determined according to the similarity transformation, and the grid vertex inertia error of the corresponding grid vertex of each pixel in the target background regularized by the displacement, include: The luminance gradient of each pixel on the first target image layer is calculated, and each pixel on the first target image layer is traversed according to a preset step size to determine a first target sample set, wherein the pixels in the first target sample set belong to the first target background layer, and the corresponding luminance gradient is greater than a preset first threshold, and the first target image layer and the first target background layer are located at the same level in the image pyramid. The luminance error of each pixel on the first target image layer relative to the corresponding pixel in the reference background, the grid deformation error of the corresponding grid vertex of each pixel on the first target image layer determined according to the similarity transformation, and the grid vertex inertia error of the corresponding grid vertex of each pixel on the first target image layer regularized by the displacement.

5. The video processing method of claim 4, wherein, The displacement vector of the corresponding grid vertex of each pixel in the target background is determined according to the total cost function and a preset optimization algorithm, so as to generate an optimized target image by adjusting the target image according to the displacement vector. According to the displacement vector, the corresponding pixels in the first target sample set are deformed by grid, and the result of the grid deformation is transmitted to the next target image layer in the target image pyramid until the bottom layer of the target image pyramid, so as to generate an optimized target image, wherein the first target image layer is the top layer of the target image pyramid.

6. The video processing method of any of claims 2-5, wherein, After determining the grid deformation error of the corresponding grid vertex of each pixel in the target background according to the similarity transformation, it further includes: The grid corresponding to each pixel in the target background is divided into a first triangle and a second triangle. According to the grid deformation error, the first triangle is similarity transformed into a deformed third triangle, and the second triangle is similarity transformed into a deformed fourth triangle.

7. The video processing method of any of claims 2-5, wherein, Before the background area in the target image is used to fill and complete the target area in the reference image, it further includes: The image acquisition state is determined as a motion acquisition state.

8. The video processing method of claim 7, wherein, The image acquisition state is determined as the motion acquisition state, including: The average displacement of each grid vertex in the optimized target image relative to the corresponding each grid vertex in the target image is calculated; If the average displacement is greater than or equal to a preset displacement threshold, the image acquisition state is determined as the motion acquisition state.

9. A video processing apparatus, comprising: It includes: An image acquisition module is configured to acquire a target image in response to a trigger instruction, wherein the target image is a previous video frame of a reference image, and the reference image is a current video frame acquired by an image sensor. An image processing module is configured to fill in a target region in the reference image using a background region in the target image to generate a processed image, the target region being a region covered by a target object in the reference image; A display is configured to display the processed image as a current video frame, the processed image being an image in which the target object is at least partially removed from the reference image; The image processing module is specifically configured to determine a photometric error of each pixel in a target background relative to a corresponding pixel in a reference background, a lattice vertex deformation error of a lattice vertex corresponding to each pixel in the target background according to a lattice deformation determined by a similarity transformation, and a lattice vertex inertia error of the lattice vertex corresponding to each pixel in the target background regularized by a displacement of the lattice vertex, wherein the target background corresponds to a background region in the target image, and the reference background corresponds to a background region in the reference image; determine a total cost function according to the photometric error, the lattice vertex deformation error, and the lattice vertex inertia error; determine a displacement vector of the lattice vertex corresponding to each pixel in the target background according to the total cost function and a preset optimization algorithm, so as to generate an optimized target image by adjusting the target image according to the displacement vector; calculate a dense optical flow of the reference image relative to the optimized target image; and fill in the target region in the reference image according to the dense optical flow.

10. An electronic device, comprising: Comprise: a processor; and a memory for storing a computer program; wherein the processor is configured to implement the video processing method of any one of claims 1 to 8 by executing the computer program; and a display for displaying a video processed by the processor.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, the video processing method of any one of claims 1 to 8 is implemented.

12. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the video processing method of any one of claims 1 to 8. The computer program is executed by the processor to implement the video processing method of any one of claims 1 to 8.

Images