Image processing method and related device

Abstract

Embodiments of the present application disclose an image processing method and related equipment. The method can be applied to a scene of removing other subjects irrelevant to a first subject in an image or a video. The method comprises: acquiring a first image and a second image; determining a first region of the first image and a second region of the second image; constructing a Poisson equation based on a target region, the target region comprising the first region and the second region; and repairing the first region and the second region based on the Poisson equation to obtain a first target image and a second target image, the first target image and the second target image not comprising a second subject. Other subjects irrelevant to the first subject in multiple images can be removed to obtain a target image with the first subject and a clean background. Moreover, multiple repaired target images are obtained by repairing target regions based on the Poisson equation constructed based on the target regions. Compared with the prior art which needs to construct a Poisson equation for each image, the image repair rate can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image processing method and related equipment. Background Technology

[0002] With the rapid iteration and popularization of smartphones and tablets, the built-in camera functions for taking photos and videos have become increasingly rich and powerful, gaining popularity among consumers. Meanwhile, in the internet age, sharing text, pictures, and videos on social media platforms has become a daily habit. Along with rising living standards and the accompanying rise of consumerism, thanks to the convenience of transportation and accommodation, more ordinary people are choosing to travel, using the camera functions of smartphones and other mobile devices to take photos anytime, anywhere and share them on social media platforms such as WeChat and Weibo. Typically, when taking photos at scenic spots or popular photo spots, there are many tourists, especially during holidays. When taking photos / videos to commemorate the occasion, in addition to the main subject, the background of the photos / videos often contains many passersby and objects. These passersby not only obscure the scenery but also make the background appear cluttered, seriously affecting the final product.

[0003] Currently, the traditional method to obtain a photo / video without passersby is to remove passersby / objects through offline post-processing, such as manually retouching photos using image editing tools (e.g., Photoshop / Snapseed) or manually retouching videos using video editing and design tools (e.g., After Effects).

[0004] However, the methods mentioned above are time-consuming, labor-intensive, and require a high level of expertise from the user. How to efficiently remove parts unrelated to the subject when taking photos is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] This application provides an image processing method and related equipment, which can not only remove other subjects unrelated to the first subject from an image or video, but also improve the image or video restoration rate.

[0006] The first aspect of this application provides an image processing method. This method can be applied to scenes involving the removal of other subjects unrelated to a first subject from an image or video. The method can be executed by an image processing device (e.g., a terminal device or a server) or by a component of the image processing device (e.g., a processor, chip, or chip system). The image processing device can be a terminal device or a cloud device (e.g., a server). The method includes: acquiring a first image and a second image, wherein the first image includes a first subject, a second subject, and a first background, and the first background is the image content in the first image excluding the first and second subjects; the second image includes the first subject, the second subject, and a second background, and the second background is the image content in the second image excluding the first and second subjects, and the second background and the first background correspond to different perspectives of the same scene; determining a first region of the first image and a second region of the second image, wherein the first region includes the area occupied by the second subject in the first image, and the second region includes the area occupied by the second subject in the second image; and constructing a Poisson equation based on a target region, wherein the target region includes the first region and the second region, and the Poisson equation is used to represent... The relationship between the target pixel value to be determined in the target region and the color gradient in the first image / second image is shown. The target pixel value is used to cover the pixel values ​​of the first region and the second region. The first region and the second region are repaired based on the Poisson equation to obtain the first target image and the second target image. The first target image and the second target image do not include the second subject. The first target image includes the first subject and the first target background. The first target background is the image content in the first target image other than the first subject. The second target image includes the first subject and the second target background. The second target background is the image content in the second target image other than the first subject. The first background is a sub-region of the first target background and the second background is a sub-region of the second target background.

[0007] The first and second images can be understood as being captured from different perspectives of the same scene. The first subject, also known as the photographed object, can be the person or object occupying the largest pixel area in the first image, or it can be a person or object closer to the terminal device. The first subject can also be a person or object determined in the first image by the user's actions (or understood as the subject the user wants to highlight in the image). Besides the methods mentioned above, there are other possible ways to determine the first subject, which are not limited here. Furthermore, the second subject can also be understood as another subject unrelated to the first subject, or a subject the user wants to remove from the first image, such as passersby or cluttered objects that obstruct the user's view.

[0008] Furthermore, if the position of the first region in the first image is the same as the position of the second region in the second image, it can be directly repaired using the method described above for constructing the Poisson equation. If the position of the first region in the first image is different from the position of the second region in the second image, either the first or second region can be filled first (e.g., using an image inpainting network), and then repaired using the method described above for constructing the Poisson equation.

[0009] In this embodiment, on one hand, other subjects unrelated to the first subject can be removed from the first and second images to obtain a first target image and a second target image with the first subject and a clean background. On the other hand, the first and second target images are obtained by repairing the first and second regions based on the Poisson equation constructed from the union of the first and second regions (i.e., the target region). Compared to the prior art which requires constructing a Poisson equation for each image, this application can improve the image repair rate by using the union region to construct the Poisson equation.

[0010] Optionally, in one possible implementation of the first aspect, the above steps: acquiring the first image and the second image include: receiving the first image and the second image sent by the terminal device; the method further includes: sending the first target image and the second target image to the terminal device.

[0011] In this possible implementation, the repair process takes place on a cloud server, which can acquire multiple repaired target images while saving computing power on terminal devices.

[0012] Optionally, in one possible implementation of the first aspect, the above steps further include: receiving a first mask and a second mask sent by a terminal device, wherein the first mask is used to indicate a first region in a first image and the second mask is used to indicate a second region in a second image; determining the first region of the first image and the second region of the second image, including: determining the first region based on the first mask and the first image; and determining the second region based on the second mask and the second image.

[0013] In this possible implementation, the area to be repaired is determined by the mask and image sent by the terminal device, thereby enabling batch repair of the areas to be repaired in multiple images to obtain multiple target images.

[0014] Optionally, in one possible implementation of the first aspect, the above steps further include: receiving a first mask sent by a terminal device, the first mask being used to indicate a first region in a first image; determining the first region of the first image and a second region of the second image, including: determining the second region based on the first image, the second image, and the first mask.

[0015] In this possible implementation, a mask and multiple images sent by the terminal device are used to determine the repair areas of multiple images, thereby enabling batch repair of the repair areas of multiple images to obtain multiple target images.

[0016] Optionally, in one possible implementation of the first aspect, the above steps: constructing a Poisson equation based on the target region, include: determining optical flow information based on a first image and a second image, whereby the optical flow information represents the change in the value of each pixel in the first image or the first region; determining a first color gradient of the first region based on the optical flow information and the color gradient of a third region, where the third region is the region in the second image corresponding to the first region; determining a second color gradient of the second region based on the optical flow information and the color gradient of a fourth region, where the fourth region is the region in the first image corresponding to the second region; and constructing a Poisson equation based on the first color gradient and the second color gradient. The third region can also be understood as the image content in the second image corresponding to the first region that is occluded in the first image. The fourth region can also be understood as the image content in the first image corresponding to the second region that is occluded in the second image.

[0017] In this possible implementation, the gradient of the first region can be solved based on the optical flow information and the gradient of the region corresponding to the first region in the known image. That is, the gradient can be propagated using optical flow to obtain the filling content of the occluded region with higher temporal consistency.

[0018] Alternatively, in one possible implementation of the first aspect, the Poisson equation described above is as follows:

[0019] Ax = b;

[0020] Where A is the coefficient matrix of the target region, b is the divergence of the first color gradient or the divergence of the second color gradient, and x is the target pixel.

[0021] In this possible implementation, batch image restoration can be achieved by constructing a Poisson equation based on multiple regions to be restored.

[0022] Optionally, in one possible implementation of the first aspect, the number of pixels in the target region is N, the coefficient matrix is ​​an N*N matrix, and b is an N*1 dimensional vector. A and b are determined as follows:

[0023]

[0024]

[0025] Among them, A i,j Let G be the element in the i-th row and j-th column of matrix A; pixel j is the neighboring pixel of pixel i; i,j f represents the gradient in the direction from pixel i to pixel j;j Let f represent the color of pixel j. If the color of pixel j is unknown, then f... j =0.

[0026] In this possible implementation, the first target image and the second target image are obtained by repairing the first region and the second region based on the Poisson equation constructed from the union of the first region and the second region (i.e., the target region). That is, multiple images are repaired through a single A matrix. Compared with the prior art, which requires constructing a Poisson equation for each image, this application can improve the image repair speed by using the union region to construct the Poisson equation.

[0027] Optionally, in one possible implementation of the first aspect, the above steps: repairing the first region and the second region based on the Poisson equation include: replacing the pixel values ​​of the first region and the second region with the target pixel values ​​to obtain the first target image and the second target image.

[0028] In this possible implementation, the target pixel value of the target region is solved by an A matrix, and the original pixel value of the target region is replaced, thereby realizing the restoration of multiple images.

[0029] A second aspect of this application provides an image processing method. This method can be applied to a scene where other subjects unrelated to a first subject are removed from an image or video. The method can be executed by an image processing device (e.g., a terminal device or a server) or by a component of the image processing device (e.g., a processor, a chip, or a chip system). The image processing device can be a terminal device. The method includes: acquiring a first image and a second image, wherein the first image includes a first subject, a second subject, and a first background, and the first background is the image content in the first image excluding the first and second subjects; the second image includes the first subject, the second subject, and a second background, and the second background is the image content in the second image excluding the first and second subjects, and the second background and the first background correspond to different perspectives of the same scene; determining a first region of the first image and a second region of the second image, wherein the first region includes the region occupied by the second subject in the first image, and the second region includes the region occupied by the second subject in the second image; and sending the first image, a first mask, and the second image to a server, wherein the first mask is used to indicate the first region in the first image. The domain, a first image, a first mask, and a second image are used to determine the target region, which includes the first region and the second region; the first target image and the second target image sent by the server are received. The first target image and the second target image are obtained by solving the Poisson equation based on the target region. The first target image and the second target image do not include the second subject. The first target image includes the first subject and the first target background. The first target background is the image content in the first target image other than the first subject. The second target image includes the first subject and the second target background. The second target background is the image content in the second target image other than the first subject. The first background is a sub-region of the first target background, and the second background is a sub-region of the second target background.

[0030] In this embodiment, on the one hand, other subjects unrelated to the first subject can be removed from the first and second images to obtain a first target image and a second target image with the first subject and a clean background. On the other hand, the server can construct a Poisson equation based on the target region and repair the first and second regions based on the Poisson equation to obtain the first target image and the second target image. Alternatively, it can be understood that the repair of multiple regions can be achieved by calculating the A matrix once. Compared to the prior art, which requires constructing a Poisson equation for each image, this application can improve the image repair speed by using a union region to construct the Poisson equation. Furthermore, by moving complex computing power to the cloud (i.e., the server) for processing, the computing power of the terminal device can be saved.

[0031] Optionally, in one possible implementation of the second aspect, the above step of acquiring the first image and the second image includes: acquiring the first image and the second image based on a first operation of the user, wherein the first operation includes at least one of the user clicking to take a picture, selecting an image from a database, and an operation of the user's mobile terminal device.

[0032] In this possible implementation, the image to be repaired can be determined through user operation, thus meeting the user's needs for image repair and improving the user experience.

[0033] Optionally, in one possible implementation of the second aspect, the above steps: determining the first region of the first image and the second region of the second image include: displaying the first image and the second image to the user; determining the first region in the first image and the second region in the second image based on the user's second operation; or, determining the first region and the second region according to a preset rule.

[0034] In this possible implementation, the repair area in multiple images is determined by user operation or preset rules, which can satisfy the user's need to remove other subjects (i.e., second subjects) that are unrelated to the first subject, and obtain a target image with the first subject and a clean background.

[0035] Optionally, in one possible implementation of the second aspect, the above steps: determining the first region of the first image and the second region of the second image include: determining the first region based on a third user operation or a preset rule; and determining the second region based on the first image, the first region, and the second image.

[0036] In this possible implementation, the region to be repaired in an image can be determined first based on user operation or preset rules, and then the regions to be repaired in other images (i.e., the second region) can be determined based on multiple images and the region to be repaired (i.e., the first region), thus saving the transmission overhead caused by transmitting the second mask.

[0037] Optionally, in one possible implementation of the second aspect, the above steps further include: sending a second mask to the server, the second mask being used to indicate a second region in the second image.

[0038] In one possible implementation, the second mask can be directly sent to the server, allowing the server to determine the second region based on the second mask and the second image. Compared to predicting the second region using the first region, the first image, and the second image, this method allows for faster determination of the second region.

[0039] Optionally, in one possible implementation of the second aspect, the above steps further include: displaying the first target image and the second target image.

[0040] In this possible implementation, after repairing multiple images, multiple target images that meet the user's repair needs can be displayed to the user.

[0041] A third aspect of this application provides an image processing apparatus that can be applied to a scenario involving the removal of other subjects unrelated to a first subject from an image or video. The image processing apparatus can be a terminal device or a cloud device (e.g., a server). The image processing apparatus includes: an acquisition unit for acquiring a first image and a second image, wherein the first image includes a first subject, a second subject, and a first background, and the first background is image content in the first image excluding the first and second subjects; the second image includes a first subject, a second subject, and a second background, and the second background is image content in the second image excluding the first and second subjects, and the second background and the first background correspond to different perspectives of the same scene; a determination unit for determining a first region of the first image and a second region of the second image, wherein the first region includes the region occupied by the second subject in the first image, and the second region includes the region occupied by the second subject in the second image; and a construction unit for constructing a Poisson equation based on a target region, wherein the target region includes the first region and the second region. The domain, where the Poisson equation is used to represent the relationship between the target pixel value to be determined in the target region and the color gradient in the first image / second image, and the target pixel value is used to cover the pixel values ​​of the first region and the second region; the repair unit is used to repair the first region and the second region based on the Poisson equation to obtain the first target image and the second target image. The first target image and the second target image do not include the second subject. The first target image includes the first subject and the first target background. The first target background is the image content in the first target image other than the first subject. The second target image includes the first subject and the second target background. The second target background is the image content in the second target image other than the first subject. The first background is a sub-region of the first target background, and the second background is a sub-region of the second target background.

[0042] Optionally, in one possible implementation of the third aspect, the aforementioned acquisition unit is specifically used to receive the first image and the second image sent by the terminal device; the image processing device further includes: a sending unit, used to send the first target image and the second target image to the terminal device.

[0043] Optionally, in one possible implementation of the third aspect, the acquisition unit described above is further configured to receive a first mask and a second mask sent by the terminal device, wherein the first mask is used to indicate a first region in the first image and the second mask is used to indicate a second region in the second image; the determination unit is specifically configured to determine the first region based on the first mask and the first image; the determination unit is specifically configured to determine the second region based on the second mask and the second image.

[0044] Optionally, in one possible implementation of the third aspect, the acquisition unit described above is further configured to receive a first mask sent by the terminal device, the first mask being used to indicate a first region in the first image; the determination unit is specifically configured to determine a second region based on the first image, the second image, and the first mask.

[0045] Optionally, in one possible implementation of the third aspect, the aforementioned construction unit is specifically used to determine optical flow information based on the first image and the second image, wherein the optical flow information is used to represent the change in the value of each pixel in the first image or the first region; the construction unit is specifically used to determine the first color gradient of the first region based on the optical flow information and the color gradient of the third region, wherein the third region is the region in the second image corresponding to the first region; the construction unit is specifically used to determine the second color gradient of the second region based on the optical flow information and the color gradient of the fourth region, wherein the fourth region is the region in the first image corresponding to the second region; and the construction unit is specifically used to construct the Poisson equation based on the first color gradient and the second color gradient.

[0046] Alternatively, in one possible implementation of the third aspect, the Poisson equation described above is as follows:

[0047] Ax = b;

[0048] Where A is the coefficient matrix of the target region, b is the divergence of the first color gradient or the divergence of the second color gradient, and x is the target pixel.

[0049] Alternatively, in one possible implementation of the third aspect, the number of pixels in the target region is N, the coefficient matrix is ​​an N*N matrix, and b is an N*1 dimensional vector. A and b are determined as follows:

[0050]

[0051]

[0052] Among them, A i,j Let G be the element in the i-th row and j-th column of matrix A; pixel j is the neighboring pixel of pixel i; i,j f represents the gradient in the direction from pixel i to pixel j; j Let f represent the color of pixel j. If the color of pixel j is unknown, then f... j =0.

[0053] Optionally, in one possible implementation of the third aspect, the aforementioned repair unit is specifically used to replace the pixel values ​​of the first region and the second region with the target pixel values ​​to obtain the first target image and the second target image.

[0054] A fourth aspect of this application provides an image processing apparatus that can be applied to a scene where other subjects unrelated to a first subject are removed from an image or video. The image processing apparatus can be a terminal device. The image processing apparatus includes: an acquisition unit for acquiring a first image and a second image, wherein the first image includes a first subject, a second subject, and a first background, and the first background is image content in the first image excluding the first and second subjects; the second image includes a first subject, a second subject, and a second background, and the second background is image content in the second image excluding the first and second subjects, and the second background and the first background correspond to different perspectives of the same scene; a determination unit for determining a first region of the first image and a second region of the second image, wherein the first region includes the region occupied by the second subject in the first image, and the second region includes the region occupied by the second subject in the second image; and a sending unit for sending the first image, a first mask, and the second image to a server, wherein the first mask is used for indication. The first image, the first image, the first mask, and the second image are used to determine the target region, which includes the first region and the second region; the receiving unit is used to receive the first target image and the second target image sent by the server. The first target image and the second target image are obtained by solving the Poisson equation based on the target region. The first target image and the second target image do not include the second subject. The first target image includes the first subject and the first target background. The first target background is the image content in the first target image other than the first subject. The second target image includes the first subject and the second target background. The second target background is the image content in the second target image other than the first subject. The first background is a sub-region of the first target background, and the second background is a sub-region of the second target background.

[0055] Optionally, in one possible implementation of the fourth aspect, the aforementioned acquisition unit is specifically used to acquire the first image and the second image based on a user's first operation, the first operation including at least one of the user clicking to take a picture, selecting an image from a database, and the user's mobile terminal device.

[0056] Optionally, in one possible implementation of the fourth aspect, the determining unit described above is specifically used to display the first image and the second image to the user; the determining unit is specifically used to determine the first region in the first image and the second region in the second image based on the user's second operation; or, the determining unit is specifically used to determine the first region and the second region according to a preset rule.

[0057] Optionally, in one possible implementation of the fourth aspect, the aforementioned determining unit is specifically used to determine the first region based on the user's third operation or preset rules; the determining unit is specifically used to determine the second region based on the first image, the first region, and the second image.

[0058] Alternatively, in one possible implementation of the fourth aspect, the aforementioned sending unit is further configured to send a second mask to the server, the second mask being used to indicate a second region in the second image.

[0059] Optionally, in one possible implementation of the fourth aspect, the image processing device described above further includes: a display unit for displaying the first target image and the second target image.

[0060] The fifth aspect of this application provides an image processing apparatus that performs the method in the first aspect or any possible implementation thereof, or performs the method in the second aspect or any possible implementation thereof.

[0061] A sixth aspect of this application provides an image processing apparatus, comprising: a processor coupled to a memory for storing a program or instructions, wherein when the program or instructions are executed by the processor, the image processing apparatus implements the method of the first aspect or any possible implementation thereof, or implements the method of the second aspect or any possible implementation thereof.

[0062] The seventh aspect of this application provides a computer-readable medium having a computer program or instructions stored thereon, which, when run on a computer, cause the computer to perform the methods of the first aspect or any possible implementation thereof, or cause the computer to perform the methods of the second aspect or any possible implementation thereof.

[0063] The eighth aspect of this application provides a computer program product that, when executed on a computer, causes the computer to perform the methods of the first aspect or any possible implementation thereof, or causes the computer to perform the methods of the second aspect or any possible implementation thereof.

[0064] The technical effects of the third, fifth, sixth, seventh, and eighth aspects or any of their possible implementations can be found in the first aspect or the technical effects of different possible implementations of the first aspect, and will not be repeated here.

[0065] The technical effects of the fourth, fifth, sixth, seventh, and eighth aspects, or any one of their possible implementations, can be found in the second aspect or the technical effects of different possible implementations of the second aspect, and will not be repeated here.

[0066] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: On the one hand, other subjects unrelated to the first subject can be removed from the first image and the second image, resulting in a first target image and a first target image with a clean background containing the first subject. On the other hand, since the first target image and the second target image are obtained by repairing the first region and the second region based on the Poisson equation constructed from the union of the first region and the second region (i.e., the target region), compared to the prior art which requires constructing a Poisson equation for each image, this application can improve the image repair rate by using the union region to construct the Poisson equation. Attached Figure Description

[0067] Figure 1 A schematic diagram of a communication system provided in an embodiment of the present invention;

[0068] Figure 2 A schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0069] Figure 3 A flowchart illustrating the image processing method provided in this application embodiment;

[0070] Figure 4 An example diagram of a user interface for determining a first image and a second image provided in an embodiment of this application;

[0071] Figure 5 Example diagrams of the first and second images provided for embodiments of this application;

[0072] Figure 6 Another example user interface diagram for determining the first image and the second image provided in the embodiments of this application;

[0073] Figure 7 Another example user interface diagram for determining the first image and the second image provided in the embodiments of this application;

[0074] Figure 8 An example diagram of a first region of a first image and a second region of a second image provided in an embodiment of this application;

[0075] Figure 9 Another example diagram of the first region of the first image and the second region of the second image provided in the embodiments of this application;

[0076] Figure 10 An example diagram of the first mask and the second mask provided in the embodiments of this application;

[0077] Figure 11 An example diagram of the target area provided in an embodiment of this application;

[0078] Figure 12A Example diagrams of the first target image and the second target image provided in the embodiments of this application;

[0079] Figure 12B This is an example diagram showing the restoration progress during the image restoration process provided in the embodiments of this application;

[0080] Figure 13 This is another schematic flowchart of the image processing method provided in the embodiments of this application;

[0081] Figures 14 to 16 Several structural schematic diagrams of image processing devices provided in the embodiments of this application are shown. Detailed Implementation

[0082] This application provides an image processing method and related equipment, which can not only remove other subjects unrelated to the first subject from an image or video, but also improve the image or video restoration rate.

[0083] The embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Those skilled in the art will understand that with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the term "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0084] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0085] To facilitate understanding of the technical solutions provided in this application, some concepts will be introduced below.

[0086] 1. Image gradient

[0087] Image gradient refers to the directional change in image intensity or color. Generally, image gradient includes horizontal and vertical gradients. The horizontal gradient is the difference between two horizontally adjacent pixels, and the vertical gradient is the difference between two vertically adjacent pixels. In physics, the gradient is the vector sum of the first-order partial derivatives of a multivariate function, representing the direction and magnitude of the change in the multivariate function.

[0088] 2. Divergence

[0089] Divergence is the algebraic sum of the second-order partial derivatives of a multivariate function, representing the strength of the divergence of the gradient of the multivariate function. Divergence can also be understood as calculating the degree of divergence; a positive value indicates that the vector field is diverging outwards, while a negative value indicates that the vector field is converging inwards.

[0090] 3. Optical flow

[0091] Optical flow refers to the displacement of pixel positions in two images. For two images with temporal correlation, the optical flow of a pixel in one image points to the corresponding pixel position in the next image. Optical flow can be understood as a method that uses the correlation between images to find the correspondence between pixels, thereby calculating the motion information of objects between the two images.

[0092] 4. Pixel value

[0093] A pixel value in an image can be a red-green-blue (RGB) color value, which can be a long integer representing the color. For example, a pixel value of 256*Red+100*Green+76*Blue, where Blue represents the blue component, Green represents the green component, and Red represents the red component. Within each color component, the smaller the value, the lower the brightness; the larger the value, the higher the brightness. For grayscale images, the pixel value can be a grayscale value.

[0094] Please see Figure 1 This application provides a communication system, which includes a terminal device 101 and a server 102.

[0095] The terminal device 101 can be a device with photo and / or video recording capabilities, such as a camera, mobile phone, tablet, personal computer (PC), television, smart bracelet, smartwatch, or other terminal with photo or video recording functions. It is understood that... Figure 1 The terminal device 101 mentioned is just an example, and no specific limit is specified here.

[0096] Server 102, also known as a cloud device or cloud server, receives data sent by terminal device 101, processes the received data to obtain a processing result, and then sends the processing result back to terminal device 101. Furthermore, terminal device 101 can display the processing result to the user. Alternatively, terminal device 102 can further process the processing result and display the further processed result to the user.

[0097] Optionally, the terminal device 101 and the server 102 can interact through a communication network using any communication mechanism / standard. The communication network can be a wide area network (WAN), a local area network (LAN), a point-to-point connection, or any combination thereof. Specifically, the communication network can include a wireless network, a wired network, or a combination of wireless and wired networks. The wireless network includes, but is not limited to, any one or more combinations of: 5th-Generation (5G) systems, Long Term Evolution (LTE) systems, Global System for Mobile Communication (GSM) or Code Division Multiple Access (CDMA) networks, Wideband Code Division Multiple Access (WCDMA) networks, Wireless Fidelity (WiFi), Bluetooth, Zigbee, Radio Frequency Identification (RFID), long-range (Lora) wireless communication, and near-field communication (NFC). The wired network can include fiber optic communication networks or networks composed of coaxial cables.

[0098] Taking mobile phones as an example, Figure 2 The diagram shown is a partial structural representation of a mobile phone, a terminal device provided in an embodiment of this application. (Reference) Figure 2 The mobile phone includes components such as a radio frequency (RF) circuit 210, a memory 220, an input unit 230, a display unit 240, a sensor 251, a camera 252, an audio circuit 260, a wireless fidelity (WiFi) module 270, a processor 280, and a power supply 290. Those skilled in the art will understand that... Figure 2The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0099] The following is combined with Figure 2 A detailed introduction to each component of a mobile phone:

[0100] RF circuit 210 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 280; additionally, it transmits uplink data to the base station. Typically, RF circuit 210 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, RF circuit 210 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0101] The memory 220 can be used to store software programs and modules. The processor 280 executes various functions and data processing of the mobile phone by running the software programs and modules stored in the memory 220. The memory 220 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 220 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0102] The input unit 230 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 230 may include a touch panel 231 and other input devices 232. The touch panel 231, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 231), and drive the corresponding connection devices according to a pre-set program. Optionally, the touch panel 231 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 280, and can also receive and execute commands sent by the processor 280. In addition, the touch panel 231 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 231, the input unit 230 may also include other input devices 232. Specifically, other input devices 232 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0103] The display unit 240 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 240 may include a display panel 241, which may optionally be configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar display. Furthermore, a touch panel 231 may cover the display panel 241. When the touch panel 231 detects a touch operation on or near it, it transmits the information to the processor 280 to determine the type of touch event. Subsequently, the processor 280 provides corresponding visual output on the display panel 241 based on the type of touch event. Although in Figure 2 In this embodiment, the touch panel 231 and the display panel 241 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 231 and the display panel 241 can be integrated to realize the input and output functions of the mobile phone.

[0104] The mobile phone may also include at least one sensor 251, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 241 according to the ambient light level, and the proximity sensor can turn off the display panel 241 and / or backlight when the phone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, tapping), etc. Other sensors that may be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, inertial measurement units (IMUs), and simultaneous localization and mapping (SLAM) sensors, will not be described in detail here.

[0105] Camera 252 is used to capture images and / or videos and transmit them to display panel 241 so that display panel 241 can display the images and / or videos to the user. Furthermore, camera 252 can be a monocular or binocular camera, located at the front (i.e., front-facing camera) or rear (i.e., rear-facing camera) of the phone's main body casing. Additionally, camera 252 can be an ultra-wide-angle camera, a wide-angle camera, or a telephoto camera, etc., and the specific type is not limited here.

[0106] Audio circuit 260, speaker 261, and microphone 262 provide an audio interface between the user and the mobile phone. Audio circuit 260 converts received audio data into electrical signals and transmits them to speaker 261, where speaker 261 converts them into sound signals for output. On the other hand, microphone 262 converts collected sound signals into electrical signals, which are received by audio circuit 260, converted into audio data, and then processed by processor 280 before being transmitted via RF circuit 210 to, for example, another mobile phone, or the audio data can be output to memory 220 for further processing.

[0107] WiFi is a short-range wireless transmission technology. Mobile phones, through their WiFi module 270, can help users send and receive emails, browse web pages, and access streaming media, providing wireless broadband internet access. Although Figure 2 WiFi module 270 is shown, but it is understood that it is not an essential component of a mobile phone.

[0108] The processor 280 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It executes software programs and / or modules stored in the memory 220, and calls data stored in the memory 220 to perform various functions and process data, thereby providing overall monitoring of the phone. Optionally, the processor 280 may include one or more processing units; preferably, the processor 280 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 280.

[0109] The mobile phone also includes a power supply 290 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 280 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0110] Although not shown, mobile phones may also include Bluetooth modules, etc., which will not be elaborated here.

[0111] In this embodiment of the application, the processor 280 included in the mobile phone can perform the functions of the terminal device in this embodiment of the application, which will not be described in detail here.

[0112] Currently, the traditional method for obtaining a photo / video without passersby is through offline post-processing to remove them. This includes manual photo retouching using image editing tools (such as Photoshop / Snapseed) or manual video retouching using video editing and design tools (such as After Effects). However, these methods are time-consuming, labor-intensive, and require a high level of expertise from the user. How to efficiently remove irrelevant parts from the subject when taking a photo is a pressing technical problem that needs to be solved.

[0113] To address the aforementioned issues, this application provides an image processing method that can remove other subjects unrelated to the first subject from a first image and a second image, resulting in a first target image and a second target image with the first subject and a clean background, thereby improving the image restoration rate.

[0114] The image processing method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings. For ease of description, a mobile phone is used as an example of the terminal device.

[0115] The image processing method provided in this application can be executed by an image processing device or by a component of the image processing device (such as a processor, chip, or chip system). The image processing device can be a server or a terminal device; the following description assumes the image processing device is a terminal device. Of course, the image processing method can also be executed jointly by the terminal device and the server; this is not limited here.

[0116] Please see Figure 3 One embodiment of the image processing method in this application includes steps 301 to 304.

[0117] Step 301: Obtain a first image and a second image. The first image includes a first subject, a second subject, and a first background. The second image includes a first subject, a second subject, and a first background.

[0118] The first image in this embodiment includes a first subject, a second subject, and a first background, wherein the first background is the image content in the first image excluding the first and second subjects. The second image includes a first subject, a second subject, and a second background, wherein the second background is the image content in the second image excluding the first and second subjects. The first background and the second background correspond to different perspectives of the same scene. The subject (including the first and second subjects) can refer to a person or object, etc., and is not specifically limited here.

[0119] In this context, the first subject, also known as the photographed subject, can be a person or object occupying the largest pixel area in the first image. The first subject can also be a person or object located near the terminal device (or camera). The first subject can also be a person or object located in the central area of ​​the image (e.g., a regular or irregular area), which can be understood as the first subject being closer to the center of the image than the second subject. The first subject can also be a person or object determined in the first image through user operations (clicking, dragging, etc.) (or understood as a subject the user wants to highlight in the image). Besides the methods mentioned above, other possible ways to determine the first subject are not limited here. Furthermore, the other subjects mentioned above can also be understood as subjects unrelated to the first subject, or subjects the user wants to remove from the first image, such as passersby, cluttered objects affecting the user's vision, etc. In this embodiment, the first subject can be understood as the photographed object or person the user wants to photograph, etc., without further limitation. This embodiment only describes the first subject as a person.

[0120] The first and second backgrounds mentioned above correspond to the same scene from different perspectives, which can be understood as satisfying at least one of the following properties:

[0121] 1. The same scene can refer to the same content in two backgrounds (e.g., the first background and the second background). For example, the overlapping content (or area, area) of the first background and the second background is greater than or equal to 30%.

[0122] 2. "Same scene" can refer to the following: the distance between the first position of the terminal device when acquiring the first image (corresponding to the first background) and the second position of the terminal device when acquiring the second image (corresponding to the second background) is less than a certain threshold (e.g., if the distance between the first and second positions is 1 meter and the threshold is 2 meters, then if the distance is less than the threshold, the first and second backgrounds can be determined to be the same scene); and / or the overlap angle of the field of view of the first and second images is greater than a certain threshold (e.g., the overlap angle of the field of view of the first and second images is greater than 30 degrees); and / or the difference in rotation angle between the terminal devices acquiring the first and second images is less than a certain threshold. The rotation angle can be the horizontal rotation angle of the terminal device or the downward rotation angle of the terminal device.

[0123] The location mentioned above can be a relative location or a geographic location. If the location is a relative location, the relative location between the first location and the second location can be determined by establishing a scene model. If the location is a geographic location, it can be the location of the terminal device determined based on the Global Positioning System (GPS) or the BeiDou Navigation Satellite System.

[0124] 3. The same scene can also be judged based on light intensity. For example, the similarity between the weather type when the first image was captured and the weather type when the second image was captured can be used to determine if the first background in the first image and the second background in the second image belong to the same scene. For instance, if both the first and second images were captured on sunny days, then the first and second backgrounds are considered to be in the same scene. If both images were captured on sunny days and rainy days, then the first and second backgrounds are not considered to be in the same scene. Generally, this method needs to be combined with other methods mentioned above for judgment.

[0125] 4. The same scene can also refer to the texture similarity between the first background and the second background being greater than or equal to a certain threshold. Generally, this method needs to be combined with the other methods mentioned above for judgment.

[0126] It is understandable that the above method of determining whether the first background and the second background are the same scene is just an example. In actual applications, there may be other methods, which are not limited here.

[0127] Optionally, the first image and the second image in the embodiments of this application can be two adjacent frames in a video, or two images in a video that are n frames apart, where n can be 1 or 2, etc., and is not specifically limited here.

[0128] Optionally, the terminal device may also acquire more images or videos, which may include the first image and the second image.

[0129] In this application embodiment, the terminal device can acquire the first image and the second image in various ways. It can be acquired through the user's first operation, or by receiving the first image and the second image sent by another device, or other ways of acquiring the first image and the second image, etc. The specific method is not limited here.

[0130] The following description uses the first method (i.e., the terminal device acquires the first and second images through the user's first operation) as an example:

[0131] The terminal device acquires a first image and a second image based on a user's first operation, the first operation including at least one of the following: the user clicking to take a picture, the user selecting an image from a database, and the user moving the terminal device.

[0132] Optionally, the terminal device can acquire the first and second images by having the user click the capture button. Alternatively, the user can select images from a gallery (i.e., a database) to acquire the first and second images (or, in other words, the user selects the first and second images to be repaired from the gallery). The terminal device can also acquire the second image after the user moves the terminal device to a preset location following the acquisition of the first image by clicking the capture button. This preset location can be a location indicated by the terminal device or a location selected by the user based on actual needs. In practical applications, the preset location and the method of acquiring the first and second images based on the first operation can have other variations, which are not limited here.

[0133] The following examples illustrate various scenarios in which the aforementioned terminal device acquires the first and second images based on the user's first operation.

[0134] Example 1, such as Figure 4 As shown in (a), the terminal device displays to the user as follows: Figure 4The user interface shown in (a) or (b) may include a shooting interface and a shooting button. The shooting interface displays the image captured by the camera of the current terminal device, and the shooting button triggers the terminal device to capture an image. Optionally, the user interface may also include a preview interface, a gallery, and / or shooting options. The preview interface displays a real-time preview of the repaired image. The gallery stores multiple images. Shooting options may include front / rear shooting options, aperture, night scene, portrait, photo, video, professional, and more options, etc., without specific limitations here. Furthermore, the terminal device can acquire a first image based on the user clicking the shooting button 401. Similarly, the terminal device can acquire a second image based on the user clicking the shooting button 402. The first image is as follows... Figure 5 As shown in (a), the second image is as follows Figure 5 As shown in (b). Furthermore, after determining the first and second images, the preview interface can display the target image after subsequent repair steps. Or, to put it another way, Figure 4 In (a) and (b), the image processing device can display the real-time repaired image in the upper right corner of the user interface so that the user can view the image repair process in real time.

[0135] Example 2, such as Figure 6 As shown in (a), the terminal device displays to the user as follows: Figure 6 The user interface shown in (a) may include multiple images stored on the terminal device. Optionally, the user interface may also include time, discovery, etc., which are not limited here. Figure 6 The user interface shown in (a) can also be triggered by the user clicking on the gallery, thus displaying itself on the terminal device. Furthermore, as... Figure 6 As shown in (b), a user can select a first image and a second image by clicking among multiple images. For example, the terminal device selects the first image through user click operation 601, and the terminal device selects the second image through user click operation 602. The first image is as follows: Figure 5 As shown in (a), the second image is as follows Figure 5 As shown in (b).

[0136] Example 3, such as Figure 7 As shown in (a), after the user acquires the first image through Example 1, Example 2, or other means described above, the terminal device indicates to the user the direction of movement of the terminal device, which is used by the terminal device to acquire the second image. Figure 7 As shown in (b), the user moves according to the direction indicated by the terminal device, and the terminal device displays a preview image. Further, as... Figure 7 As shown in (c), the terminal device 701 captures a second image based on the user's click of the capture button. The first image is as follows. Figure 5 As shown in (a), the second image is as follows Figure 5 As shown in (b). It is understandable that... Figure 7 (a), (b), and (c) can also be related to the aforementioned Figure 4 The descriptions of the preview interface and the shooting interface in (a) and (b) are similar. That is, the image processing device can display the real-time repaired image in the upper right corner of the user interface, so that the user can view the image repair process in real time.

[0137] Step 302: Determine the first region of the first image and the second region of the second image. The first region is the region occupied by the second subject in the first image, and the second region is the region occupied by the second subject in the second image.

[0138] In this embodiment, the first region can also be understood as the occlusion region of the first image, and the second region can be understood as the occlusion region of the second image. Furthermore, there are multiple ways for the terminal device to determine the first region of the first image and the second region of the second image, which are described below:

[0139] The first method determines the first and second regions based on the user's second action.

[0140] In this method, the terminal device can display a first image to the user and determine a first area of ​​the first image based on the user's second operation. The second operation can be a click, drag, and / or swipe, etc., and is not limited here.

[0141] The second method involves determining the first and second regions based on preset rules.

[0142] The preset rule can be that the circumscribed polygonal region (e.g., the circumscribed rectangular region) of the second subject in the first image is the first region, and the circumscribed polygonal region (e.g., the circumscribed rectangular region) of the second subject in the second image is the second region. Alternatively, the preset rule can be that the region extending outward from the circumscribed polygonal region of the second subject in the first image by a preset value is the first region, and the region extending outward from the circumscribed polygonal region of the second subject in the second image by a preset value is the second region. This preset value is set according to actual needs; for example, the preset values ​​for the outward extension of adjacent sides of the circumscribed polygonal region of the second subject can be different or the same, and are not specifically limited here.

[0143] For example, such as Figure 8 As shown in (a), the bounding rectangular region of the second subject in the first image is defined as the first region. Figure 8 As shown in (b), the bounding rectangular region of the second subject in the second image is defined as the second region. Figure 9 As shown in (a), the region extending outward by a preset value from the circumscribed polygonal region of the second subject in the first image is defined as the first region. Figure 9 As shown in (b), the area extending outward from the circumscribed polygonal region of the second subject in the second image by a preset value is defined as the second region.

[0144] The third method involves determining the first region based on the user's third operation or preset rules, and then determining the second region based on the first image, the second image, and the first region.

[0145] In this case, the first region in the first image can be determined based on the user's third operation or preset rules. The third operation is similar to the second operation in the first case mentioned above, and the preset rules are similar to the preset rules in the second case mentioned above, which will not be repeated here.

[0146] After determining the first region, the second region can be determined based on the first image, the second image, and the first region.

[0147] Optionally, a first mask corresponding to the first region in the first image is obtained. The first image, the first mask, and the second image are then input into a prediction network to obtain a second mask. This prediction network is used to propagate regions in one frame of the video to regions in other frames. This process can be understood as video instance segmentation. Alternatively, by inputting the video and the instance segmentation mask (i.e., the first mask) of one frame, the segmentation mask (i.e., the second mask) for the corresponding instance in other frames can be predicted using information from the video.

[0148] For example, continuing with the above example, to Figure 8 The first region shown in (a) is... Figure 8 Taking the second region shown in (b) as an example, the first mask is as follows: Figure 10 As shown in (a), the second mask is as follows Figure 10 As shown in (b).

[0149] It is understandable that the above methods for determining the first and second regions are just examples. In practical applications, there may be other ways to determine the first and second regions, which are not limited here.

[0150] Furthermore, if the position of the first region in the first image is the same as the position of the second region in the second image, it can be directly repaired using the method described above for constructing the Poisson equation. If the position of the first region in the first image is different from the position of the second region in the second image, either the first or second region can be filled first (e.g., using an image inpainting network), and then repaired using the method described above for constructing the Poisson equation. Here, "same position" can be understood as the first and second regions overlapping, and "different position" can be understood as the first and second regions partially overlapping or not overlapping at all.

[0151] Step 303: Construct the Poisson equation based on the target region, which includes the first region and the second region.

[0152] Optionally, optical flow information is first determined based on the first and second images. This optical flow information represents the changes in the values ​​of each pixel in the first image or a first region. A first color gradient of the first region is determined based on the optical flow information and the color gradient of a third region, where the third region is the region in the second image corresponding to the first region in the first image. A second color gradient of the second region is determined based on the optical flow information and the color gradient of a fourth region, where the fourth region is the region in the first image corresponding to the second region in the second image. After obtaining the optical flow information, the first color gradient, and the second color gradient, a Poisson equation is constructed based on the first and second color gradients.

[0153] The above can be understood as follows: if the first region in the first image is occluded (i.e., the second subject in the first image occludes the scene content), the color gradient in the second image that corresponds to the first region and is not occluded (i.e., the color gradient of the third region) is found through optical flow information and used as the first color gradient of the first region.

[0154] Optionally, if the third region corresponding to the first region in the second image is also occluded, the occluded region in the first or second image can be filled by an image inpainting network to achieve gradient propagation across different images and thus determine the gradient of each image.

[0155] After the gradients of the first and second images are filled, a Poisson equation is constructed based on the target region. Specifically, the Poisson equation can be constructed based on the color gradient of the target region and the known pixel values ​​of the target region's edges. The target region edges can be understood as pixels located at a preset threshold (e.g., a preset number of pixels or a preset distance) from the target region.

[0156] Alternatively, the Poisson equation is as follows:

[0157] Ax = b;

[0158] Where A is the coefficient matrix of the target region, b is the divergence of the first color gradient or the divergence of the second color gradient, and x is the target pixel to be determined in the target region.

[0159] Optionally, if the number of pixels in the target region is N, then A is an N*N matrix, and A and b are determined as follows:

[0160]

[0161]

[0162] Among them, A i,jLet G be the element in the i-th row and j-th column of matrix A; pixel j is the neighboring pixel of pixel i; i,j f represents the gradient in the direction from pixel i to pixel j; j Let f represent the color of pixel j. If the color of pixel j is unknown, then f... j =0. If pixel i is at the center of the target area, then the color of pixel j is unknown. If pixel i is at the border of the target area, then pixel j can be understood as a pixel value known at the edge of the target area.

[0163] For example, continuing with the above example, the mask corresponding to the target area can be as follows: Figure 11 As shown. To facilitate understanding of the construction of the Poisson equation, taking a target region consisting of 3*3 pixels as an example, please refer to Table 1 and Table 2. Table 1 shows the labels of each pixel in the target region, and Table 2 shows the values ​​of the coefficient matrix.

[0164] Table 1

[0165] 1 2 3 4 5 6 7 8 9

[0166] The numbers 1 to 9 simply indicate that there are 9 locations in the target area, and do not involve specific pixel values.

[0167] Table 2

[0168] With 1 With 2 With 3 With 4 With 5 With 6 With 7 With 8 With 9 The first one -2 1 0 1 0 0 0 0 0 The second one 1 -3 1 0 1 0 0 0 0 The 3rd one 0 1 -2 0 0 1 0 0 0 The 4th 1 0 0 -3 1 0 1 0 0 The 5th 0 1 0 1 -4 1 0 1 0 The 6th 0 0 1 0 1 -3 0 0 1 The 7th 0 0 0 1 0 0 -2 1 0 The 8th 0 0 0 0 1 0 1 -3 1 The 9th 0 0 0 0 0 1 0 1 -2

[0169] Taking the first row of Table 2 as an example, A 1,1 The value is -2, meaning the first pixel has two adjacent pixels (i.e., 1 is adjacent to 2, and 1 is adjacent to 4), therefore K = 2. A 1,2 A value of 1 indicates that the first pixel is adjacent to the second pixel. A 1,3 A value of 0 indicates that the first and third pixels are not adjacent. A 1,4 A value of 1 means the first pixel is adjacent to the fourth pixel. Similarly, A... 1,5 A 1,6 A 1,7 A 1,8 A 1,9 The value is 0. Similarly, the values ​​for the other rows can refer to the rules for the first row, which will not be repeated here.

[0170] Step 304: Repair the first region and the second region based on the Poisson equation to obtain the first target image and the second target image. The first target image and the second target image do not include the second subject.

[0171] After determining the Poisson equation, the first and second regions can be repaired based on the Poisson equation to obtain the first and second target images, respectively. The first and second target images do not include the second subject. Alternatively, it can be understood that the first region in the first image is repaired to obtain the first target image, and the second region in the second image is repaired to obtain the second target image.

[0172] Alternatively, if the Poisson equation is Ax = b, then x = A -1 *b. Where A -1 Let b be the inverse matrix of A, and let b be determined as described in step 303 above. Therefore, we can use the formula: x = A -1 *b calculates the target pixel values ​​in the target region, thereby obtaining the first target image and the second target image.

[0173] It is understood that this embodiment only uses two images as an example for illustration. In practical applications, multiple images can also be acquired. These multiple images can be different images of the same scene or several frames of a video of the same scene. The specifics are not limited here.

[0174] For example, the first target image is as follows: Figure 12A As shown in (a), the second target image is as follows Figure 12A As shown in (b) of the diagram.

[0175] Furthermore, during the restoration process, the restoration progress can be displayed to the user, and a preview area can show the restoration progress, allowing the user to intuitively understand the current restoration progress of the image. For example, as described above... Figure 4 For example, in (b) of the above, such as Figure 12B As shown, the preview interface can display the repair process (e.g., showing an image that is half-repaired). Figure 12B The interface can also include a repair progress icon, which is used to indicate to the user the current progress of the repair, such as 50%.

[0176] In one possible implementation, steps 301 to 304 occur during the shooting preview stream process, where the first and second images are preview images, and the first and second target images are the images taken after the shutter button is pressed. This approach can be applied to image processing scenarios where the user is shooting in real time. Furthermore, in this scenario, the image processing device can display the real-time repaired image in the upper right corner of the user interface, allowing the user to view the image repair process in real time.

[0177] In another possible implementation, steps 301 to 304 occur after the shooting preview stream, which can also be understood as the first image and the second image being the images after the shutter button is pressed. This method can be applied to offline image processing scenarios.

[0178] In this embodiment, on one hand, other subjects unrelated to the first subject can be removed from the first and second images to obtain a first target image and a second target image with the first subject and a clean background. On the other hand, the first and second target images are obtained by repairing the first and second regions based on the Poisson equation constructed from the union of the first and second regions (i.e., the target region). Compared to the prior art which requires constructing a Poisson equation for each image, this application can improve the image repair rate by using the union region to construct the Poisson equation. Furthermore, the gradient of the first region can be solved based on optical flow information and the gradient of the region corresponding to the first region in the known image. That is, the gradient can be propagated using optical flow to obtain the filling content of the occluded region with higher temporal consistency.

[0179] Please see Figure 13 Another embodiment of the image processing method in this application is that the image processing method can be jointly executed by a terminal device and a server, that is, the image processing device includes a terminal device and a server. This embodiment includes steps 1301 to 1311.

[0180] Step 1301: The terminal device acquires a first image and a second image. The first image includes a first subject, a second subject, and a first background. The second image includes a first subject, a second subject, and a first background.

[0181] Step 1302: The terminal device determines the first region of the first image and the second region of the second image.

[0182] Steps 1301 and 1302 are the same as those mentioned above. Figure 3 Steps 301 and 302 in the illustrated embodiment are similar and will not be repeated here.

[0183] Step 1303: The terminal device generates a first mask based on the first image and the first region.

[0184] The first mask in step 1303 is the same as described above. Figure 3 The description of the first mask in the third case of step 302 of the illustrated embodiment is similar and will not be repeated here.

[0185] Step 1304: The terminal device generates a second mask based on the second image and the second region. This step is optional.

[0186] Optionally, the terminal device may also generate a second mask based on the second image and the second region, and send the second mask to the server.

[0187] The second mask in step 1304 is the same as described above. Figure 3 The description of the second mask in the third case of step 302 of the illustrated embodiment is similar and will not be repeated here.

[0188] In step 1305, the terminal device sends a first image, a second image, and a first mask to the server. Correspondingly, the server receives the first image, the second image, and the first mask sent by the terminal device.

[0189] Optionally, the terminal device can send a first image, a second image, and a first mask to the server. The server can determine the second mask based on the first image, the second image, and the first mask. Alternatively, the terminal device can send the first image, the second image, the first mask, and the second mask to the server. Specifically, the terminal device can send the first image, the second image, the first mask, and the second mask to the server all at once. It can also send the first image and the first mask to the server first, and then send the second image and the second mask to the server. Of course, it can also send the first image and the second image to the server first, and then send the first mask and the second mask to the server.

[0190] Step 1306: The server determines the first region based on the first image and the first mask.

[0191] After receiving the first image and the first mask sent by the terminal device, the server can determine the first region in the first image based on the first image and the first mask.

[0192] Step 1307: The server determines the second region based on the second image and the second mask. This step is optional.

[0193] Optionally, if the terminal sends a second mask to the server in step 1305, the server can determine the second region in the second image based on the second image and the second mask after receiving the second image and the second mask sent by the terminal device.

[0194] Of course, if the terminal does not send the second mask to the server in step 1305, the server can determine the second mask based on the first image, the second image, and the first mask, and then determine the second region based on the second mask and the second image. Alternatively, it can be understood that the server can determine the second region based on the first region, the first image, and the second image.

[0195] Step 1308: The server constructs the Poisson equation based on the target region.

[0196] Step 1309: The server repairs the first region and the second region based on the Poisson equation to obtain the first target image and the second target image.

[0197] Steps 1308 and 1309 are the same as those mentioned above. Figure 3 Steps 303 and 304 in the illustrated embodiment are similar and will not be repeated here.

[0198] Step 1310: The server sends the first target image and the second target image to the terminal device. Correspondingly, the terminal device receives the first target image and the second target image sent by the server.

[0199] After the server obtains the first target image and the second target image, it can send the first target image and the second target image to the terminal device.

[0200] The relevant descriptions of the first target image and the second target image can be found in the foregoing. Figure 3 The descriptions in the illustrated embodiments will not be repeated here.

[0201] Step 1311: The terminal device displays the first target image and the second target image. This step is optional.

[0202] Optionally, after receiving the first target image and the second target image sent by the server, the terminal device can display the first target image and the second target image to the user.

[0203] It is understood that steps 1303 and 1304 are not sequentially related; that is, step 1304 can be after or before step 1303. Similarly, steps 1306 and 1307 are not sequentially related; that is, step 1307 can be after or before step 1306. Furthermore, this embodiment may include steps 1301 to 1310, or it may include steps 1301 to 1311. Steps 1304, 1307, and 1311 are optional steps.

[0204] In this embodiment, on one hand, the server can construct a Poisson equation based on the target region and repair the first and second regions based on the Poisson equation to obtain the first target image and the second target image. Alternatively, it can be understood that multiple regions can be repaired by calculating the A matrix once. Compared to the prior art, which requires constructing a Poisson equation for each image, this application can improve the image repair speed by using a union region to construct the Poisson equation. On the other hand, the server can solve the gradient of the first region based on optical flow information and the gradient of the region corresponding to the first region in the known image. That is, the gradient can be propagated using optical flow to obtain the filling content of the occluded region with higher temporal consistency. On the other hand, by moving the complex computing power to the cloud (i.e., the server) for processing, the computing power of the terminal device can be saved. On the other hand, other subjects unrelated to the first subject can be removed from the first image and the second image to obtain the first target image and the first target image with the first subject and a clean background.

[0205] The image processing method in the embodiments of this application has been described above. The image processing device in the embodiments of this application is described below. Please refer to [link / reference]. Figure 14 One embodiment of the image processing device (e.g., a terminal device or a server) in this application includes:

[0206] The acquisition unit 1401 is used to acquire a first image and a second image. The first image includes a first subject, a second subject, and a first background. The first background is the image content in the first image other than the first subject and the second subject. The second image includes the first subject, the second subject, and a second background. The second background is the image content in the second image other than the first subject and the second background. The second background and the first background correspond to different perspectives of the same scene.

[0207] The determining unit 1402 is used to determine a first region of the first image and a second region of the second image, wherein the first region includes the region occupied by the second subject in the first image, and the second region includes the region occupied by the second subject in the second image;

[0208] The construction unit 1403 is used to construct a Poisson equation based on a target region, wherein the target region includes the first region and the second region, and the Poisson equation is used to represent the relationship between the target pixel value to be determined in the target region and the color gradient in the first image / second image, and the target pixel value is used to cover the pixel values ​​of the first region and the second region;

[0209] Repair unit 1404 is used to repair the first region and the second region based on the Poisson equation to obtain a first target image and a second target image. The first target image and the second target image do not include the second subject. The first target image includes the first subject and a first target background. The first target background is the image content in the first target image other than the first subject. The second target image includes the first subject and a second target background. The second target background is the image content in the second target image other than the first subject. The first background is a sub-region of the first target background, and the second background is a sub-region of the second target background.

[0210] Optionally, the image processing device in this embodiment may further include: a sending unit 1405, used to send the first target image and the second target image to the terminal device.

[0211] In this embodiment, the operations performed by each unit in the image processing device are the same as described above. Figures 3 to 13 The embodiments shown are similar and will not be repeated here.

[0212] In this embodiment, on the one hand, other subjects unrelated to the first subject can be removed from the first image and the second image to obtain a first target image and a second target image with the first subject and a clean background. On the other hand, since the repair unit 1404 repairs the first region and the second region based on the Poisson equation constructed from the union of the first region and the second region (i.e., the target region) to obtain the first target image and the second target image. Compared with the prior art, which requires constructing a Poisson equation for each image, this application can improve the image repair rate by using the union region to construct the Poisson equation.

[0213] Please see Figure 15 Another embodiment of the image processing device (e.g., a terminal device) in this application includes:

[0214] The acquisition unit 1501 is used to acquire a first image and a second image. The first image includes a first subject, a second subject, and a first background. The first background is the image content in the first image other than the first subject and the second subject. The second image includes the first subject, the second subject, and a second background. The second background is the image content in the second image other than the first subject and the second background. The second background and the first background correspond to different perspectives of the same scene.

[0215] The determining unit 1502 is used to determine a first region of the first image and a second region of the second image, wherein the first region includes the region occupied by the second subject in the first image, and the second region includes the region occupied by the second subject in the second image;

[0216] The sending unit 1503 is used to send the first image, the first mask, and the second image to the server. The first mask is used to indicate the first region in the first image. The first image, the first mask, and the second image are used to determine a target region, which includes the first region and the second region.

[0217] The receiving unit 1504 is used to receive a first target image and a second target image sent by the server. The first target image and the second target image are obtained by solving the Poisson equation based on the target region. The first target image and the second target image do not include the second subject. The first target image includes the first subject and a first target background. The first target background is the image content in the first target image other than the first subject. The second target image includes the first subject and a second target background. The second target background is the image content in the second target image other than the first subject. The first background is a sub-region of the first target background, and the second background is a sub-region of the second target background.

[0218] Optionally, the image processing device in this embodiment may further include a display unit 1505 for displaying the first target image and the second target image.

[0219] In this embodiment, the operations performed by each unit in the image processing device are the same as described above. Figures 3 to 13 The embodiments shown are similar and will not be repeated here.

[0220] In this embodiment, on the one hand, other subjects unrelated to the first subject can be removed from the first image and the second image to obtain a first target image and a second target image with the first subject and a clean background. On the other hand, the receiving unit 1504 can receive the first target image and the second target image obtained by repairing the first region and the second region based on the Poisson equation constructed by the server based on the target region, or it can be understood that the repair of multiple regions can be achieved by calculating the A matrix once. Compared with the prior art, which requires the construction of a Poisson equation for each image, this application can improve the image repair speed by using the union region to construct the Poisson equation. On the other hand, by moving the complex computing power to the cloud (i.e., the server) for processing, the computing power of the terminal device can be saved.

[0221] See Figure 16 This application provides a schematic diagram of another image processing device. The image processing device may include a processor 1601, a memory 1602, and a communication interface 1603. The processor 1601, memory 1602, and communication interface 1603 are interconnected via lines. The memory 1602 stores program instructions and data.

[0222] The aforementioned are stored in memory 1602 Figures 3 to 13 In the corresponding implementation shown, the program instructions and data corresponding to the steps executed by the device are described.

[0223] Processor 1601, for executing the aforementioned Figures 3 to 13The steps performed by the device are shown in any of the embodiments illustrated.

[0224] Communication interface 1603 can be used to receive and send data, and to perform the aforementioned tasks. Figures 3 to 13 The steps related to acquiring, sending, and receiving in any of the embodiments shown.

[0225] In one implementation, the image processing device may include, relative to Figure 16 More or fewer components are merely illustrative in this application and are not intended to limit the scope of the application.

[0226] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0227] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0228] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented wholly or partially through software, hardware, firmware, or any combination thereof.

[0229] When the integrated unit is implemented using software, it can be implemented wholly or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0230] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

Claims

1. An image processing method, characterized by, The method is applied to an image processing device, and the method includes: Acquire a first image and a second image. The first image includes a first subject, a second subject, and a first background. The first background is the image content in the first image other than the first subject and the second subject. The second image includes the first subject, the second subject, and a second background. The second background is the image content in the second image other than the first subject and the second background. The second background and the first background correspond to different perspectives of the same scene. A first region of the first image and a second region of the second image are determined, wherein the first region includes the region occupied by the second subject in the first image, and the second region includes the region occupied by the second subject in the second image; A Poisson equation is constructed based on a target region, the target region including the first region and the second region. The Poisson equation is used to represent the relationship between the target pixel value to be determined in the target region and the color gradient in the first image / second image. The target pixel value is used to cover the pixel values ​​of the first region and the second region. Based on the Poisson equation, the first region and the second region are repaired to obtain a first target image and a second target image. The first target image and the second target image do not include the second subject. The first target image includes the first subject and a first target background. The first target background is the image content in the first target image other than the first subject. The second target image includes the first subject and a second target background. The second target background is the image content in the second target image other than the first subject. The first background is a sub-region of the first target background, and the second background is a sub-region of the second target background.

2. The method of claim 1, wherein, The acquisition of the first image and the second image includes: Receive the first image and the second image sent by the terminal device; The method further includes: The first target image and the second target image are sent to the terminal device.

3. The method of claim 1, wherein, The method further includes: The receiving terminal device sends a first mask and a second mask, wherein the first mask is used to indicate the first region in the first image, and the second mask is used to indicate the second region in the second image; Determining the first region of the first image and the second region of the second image includes: The first region is determined based on the first mask and the first image; The second region is determined based on the second mask and the second image.

4. The method of claim 1, wherein, The method further includes: The receiving terminal device sends a first mask, which is used to indicate the first region in the first image; Determining the first region of the first image and the second region of the second image includes: The second region is determined based on the first image, the second image, and the first mask.

5. The method according to any one of claims 1 to 4, characterized in that, The construction of the Poisson equation based on the target region includes: determining optical flow information based on the first image and the second image, the optical flow information being used to represent a change of a value of each pixel in the first image or the first region; determining a first color gradient of the first region based on the optical flow information and a color gradient of a third region, the third region being a region corresponding to the first region in the second image; determining a second color gradient of the second region based on the optical flow information and a color gradient of a fourth region, the fourth region being a region corresponding to the second region in the first image; constructing the Poisson equation based on the first color gradient and the second color gradient.

6. The method of claim 5, wherein, The Poisson equation is as follows: Ax = b; wherein A is a coefficient matrix of the target region, b is a divergence of the first color gradient or a divergence of the second color gradient, and x is a target pixel.

7. The method of claim 6, wherein, The number of pixels of the target region is N, the coefficient matrix is an N*N matrix, b is an N*1 vector, and A and b are determined as follows: where A i,j is the element of matrix A in the i-th row and j-th column; pixel j is a neighboring pixel of pixel i; G i,j represents the gradient from pixel i to pixel j; f j represents the color of pixel j, and if the color of pixel j is unknown, then f j = 0.

8. The method according to any one of claims 1 to 4, characterized in that, repairing the first region and the second region based on the Poisson equation, comprising: replacing the pixel values of the first region and the second region with the target pixel value to obtain the first target image and the second target image.

9. An image processing method characterized by, The method is applied to an image processing device, and the method comprises: obtaining a first image and a second image, the first image comprising a first subject, a second subject and a first background, the first background being image content in the first image other than the first subject and the second subject; the second image comprising the first subject, the second subject and a second background, the second background being image content in the second image other than the first subject and the second subject, the second background and the first background corresponding to different perspectives of the same scene; determining a first region of the first image and a second region of the second image, the first region comprising a region occupied by the second subject in the first image, and the second region comprising a region occupied by the second subject in the second image; sending the first image, a first mask and the second image to a server, the first mask being used to indicate the first region in the first image, the first image, the first mask and the second image being used to determine a target region, the target region comprising the first region and the second region; receive the first target image and the second target image sent by the server, the first target image and the second target image are obtained based on a Poisson equation constructed according to the target region, the first target image and the second target image do not include the second subject, the first target image includes the first subject and a first target background, the first target background is image content in the first target image except the first subject, the second target image includes the first subject and a second target background, the second target background is image content in the second target image except the first subject, the first background is a sub-region of the first target background, and the second background is a sub-region of the second target background.

10. The method of claim 9, wherein, The method further includes: obtaining the first image and the second image based on a first operation of a user, the first operation including at least one of an operation of the user clicking a shooting, an operation of selecting an image from a database, and an operation of the user moving a terminal device.

11. The method according to claim 9 or 10, characterized in that, The method further includes: displaying the first image and the second image to the user; determining the first region in the first image and the second region in the second image based on a second operation of the user; or determining the first region and the second region according to a preset rule.

12. The method according to claim 9 or 10, characterized in that, The method further includes: determining the first region based on a third operation of the user or a preset rule; determining the second region based on the first image, the first region, and the second image.

13. The method of claim 9 or 10, wherein, The method further includes: sending a second mask to the server, the second mask being used to indicate the second region in the second image.

14. The method of claim 9 or 10, wherein, The method further includes: displaying the first target image and the second target image.

15. An image processing apparatus characterized by comprising: The image processing device includes: an obtaining unit, configured to obtain a first image and a second image, the first image including a first subject, a second subject, and a first background, the first background being image content in the first image except the first subject and the second subject, the second image including the first subject, the second subject, and a second background, the second background being image content in the second image except the first subject and the second subject, the second background and the first background corresponding to different perspectives of a same scene; a determining unit, configured to determine a first region of the first image and a second region of the second image, the first region including a region occupied by the second subject in the first image, and the second region including a region occupied by the second subject in the second image. constructing a Poisson equation based on a target region, the target region comprising the first region and the second region, the Poisson equation being used to represent a relationship between target pixel values to be solved in the target region and color gradients in the first image / second image, the target pixel values being used to cover pixel values of the first region and the second region; repairing the first region and the second region based on the Poisson equation to obtain a first target image and a second target image, the first target image and the second target image not comprising the second subject, the first target image comprising the first subject and a first target background, the first target background being image content in the first target image other than the first subject, the second target image comprising the first subject and a second target background, the second target background being image content in the second target image other than the first subject, the first background being a sub-region of the first target background, and the second background being a sub-region of the second target background.

16. The apparatus of claim 15, wherein, The obtaining unit is specifically configured to receive the first image and the second image sent by the terminal device. The image processing device further comprises: The sending unit is configured to send the first target image and the second target image to the terminal device.

17. The apparatus of claim 15 or 16, wherein, The obtaining unit is further configured to receive a first mask and a second mask sent by the terminal device, the first mask being used to indicate the first region in the first image, and the second mask being used to indicate the second region in the second image. The determining unit is specifically configured to determine the first region based on the first mask and the first image. The determining unit is specifically configured to determine the second region based on the second mask and the second image.

18. The apparatus of claim 15 or 16, wherein, The obtaining unit is further configured to receive a first mask sent by the terminal device, the first mask being used to indicate the first region in the first image. The determining unit is specifically configured to determine the second region based on the first image, the second image, and the first mask.

19. The apparatus of claim 15 or 16, wherein, The constructing unit is specifically configured to determine optical flow information based on the first image and the second image, the optical flow information being used to represent changes in values of each pixel point in the first image or the first region. The constructing unit is specifically configured to determine a first color gradient of the first region based on the optical flow information and a color gradient of a third region, the third region being a region in the second image corresponding to the first region. The constructing unit is specifically configured to determine a second color gradient of the second region based on the optical flow information and a color gradient of a fourth region, the fourth region being a region in the first image corresponding to the second region. The constructing unit is specifically configured to construct the Poisson equation based on the first color gradient and the second color gradient.

20. The apparatus of claim 19, wherein, The Poisson equation is as follows: Ax = b; wherein A is a coefficient matrix of the target region, b is a divergence of the first color gradient or a divergence of the second color gradient, and x is a target pixel point.

21. The apparatus of claim 20, wherein, The target region has a pixel number N, the coefficient matrix is an N*N matrix, b is an N*1 vector, and A and b are determined as follows: where A i,j is the element of matrix A in the i-th row and j-th column; pixel j is a neighboring pixel of pixel i; G i,j represents the gradient of pixel i in the direction of pixel j; f j represents the color of pixel j, and if the color of pixel j is unknown, then f j = 0.

22. The apparatus of claim 15 or 16, wherein, The repair unit is specifically configured to replace the target pixel value with the pixel values of the first region and the second region to obtain the first target image and the second target image.

23. An image processing apparatus characterized by comprising: The image processing device comprises: An acquisition unit is configured to acquire a first image and a second image, the first image comprising a first subject, a second subject and a first background, the first background being image content in the first image other than the first subject and the second subject; the second image comprising the first subject, the second subject and a second background, the second background being image content in the second image other than the first subject and the second subject, the second background and the first background corresponding to different perspectives of the same scene; A determination unit is configured to determine a first region of the first image and a second region of the second image, the first region comprising a region occupied by the second subject in the first image, and the second region comprising a region occupied by the second subject in the second image; A sending unit is configured to send the first image, a first mask and the second image to a server, the first mask being used to indicate the first region in the first image, the first image, the first mask and the second image being used to determine a target region, the target region comprising the first region and the second region; A receiving unit is configured to receive a first target image and a second target image sent by the server, the first target image and the second target image being obtained by solving a Poisson equation constructed based on the target region, the first target image and the second target image not comprising the second subject, the first target image comprising the first subject and a first target background, the first target background being image content in the first target image other than the first subject, the second target image comprising the first subject and a second target background, the second target background being image content in the second target image other than the first subject, the first background being a sub-region of the first target background, and the second background being a sub-region of the second target background.

24. The apparatus of claim 23, wherein, The acquisition unit is specifically configured to acquire the first image and the second image based on a first operation of a user, the first operation comprising at least one of an operation of the user clicking to take a picture, an operation of selecting an image from a database and an operation of the user moving a terminal device.

25. The apparatus of claim 23 or 24, wherein, The determination unit is specifically configured to display the first image and the second image to a user. The determination unit is specifically configured to determine the first region in the first image and the second region in the second image based on a second operation of the user. Alternatively, The determination unit is specifically configured to determine the first region and the second region according to a preset rule.

26. The apparatus of claim 23 or 24, wherein, The determination unit is specifically configured to determine the first region based on a third operation of a user or a preset rule. The determining unit is specifically configured to determine the second region based on the first image, the first region, and the second image.

27. The apparatus of claim 23 or 24, wherein, The sending unit is further configured to send a second mask to the server, the second mask being used to indicate the second region in the second image.

28. The apparatus of claim 23 or 24, wherein, The image processing device further includes: a display unit configured to display the first target image and the second target image.

29. An image processing apparatus characterized by comprising: The image processing device further includes: a processor coupled to a memory, the memory being configured to store a program or instructions, when the program or instructions are executed by the processor, the image processing device is caused to perform the method in any one of claims 1 to 8, or the image processing device is caused to perform the method in any one of claims 9 to 14.

30. A computer-readable storage medium, characterized in that, The computer readable storage medium has instructions stored therein, when the instructions are executed on a computer, the computer is caused to perform the method in any one of claims 1 to 8, or the computer is caused to perform the method in any one of claims 9 to 14.

31. A computer program product, characterised in that, The computer program product, when executed on a computer, causes the computer to perform the method in any one of claims 1 to 8, or causes the computer to perform the method in any one of claims 9 to 14.

Images