An image editing method based on Markov random fields and three-dimensional implicit representation

Abstract

This invention discloses an image editing method based on Markov random fields and 3D implicit representation, belonging to the field of 3D reconstruction and image processing technology. The method includes: preprocessing the image data in the original image dataset S to divide it into a training set and images to be edited; transforming the pixel coordinates of the images constituting S into 3D world coordinates; processing the image data using an image segmentation algorithm based on Markov random fields to obtain an initial object mask; optimizing the initial object mask using a neural radiation field; and completing the missing positions in the image to generate the final image editing result. This invention fully utilizes the spatial continuity in the image sequence, improving the accuracy of image segmentation and enhancing the 3D perceptual characteristics of the segmentation results; it preserves the details of local image regions, enabling the generation of more realistic and delicate completion effects visually, thus improving the visual quality of image editing.

Description

Technical Field

[0001] This invention belongs to the field of 3D reconstruction and image processing technology, specifically relating to an image editing method based on Markov random fields and 3D implicit representation. Background Technology

[0002] With the rise of social media and digital media, image editing technology has become an important part of the information technology field. Image editing, especially object removal and masking repair tasks, plays a crucial role in privacy protection, digital content creation, and cultural heritage preservation. Object removal relies on image segmentation, which involves removing unwanted areas from an image and masking those areas. Masking repair, on the other hand, involves patching up the blank spaces left by object removal. This not only helps protect personal privacy and prevent the leakage of sensitive information, but also removes distracting elements in content creation, enhancing the visual appeal of the work.

[0003] While traditional image processing software (such as Adobe Photoshop) offers features like background removal and healing brushes, these typically require extensive professional knowledge and time-consuming manual operation. Therefore, researchers have been exploring how to utilize computer vision and deep learning technologies to automate image editing. However, these studies suffer from two main shortcomings: (1) In deep learning-based image editing methods, convolutional neural networks (CNN) and generative adversarial networks (GAN) do not have three-dimensional perception characteristics and are only suitable for processing two-dimensional images from a single perspective. When processing two-dimensional images from multiple perspectives, it is difficult to maintain consistency in the changes of images from different perspectives.

[0004] (2) Existing image editing methods rely on simple texture synthesis or cloning techniques, which are difficult to capture complex image textures, resulting in the repaired area being inconsistent with the surrounding background and the image having an unrealistic visual effect. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the present invention provides an image editing method based on Markov random fields and three-dimensional implicit representation.

[0006] The technical solution of this invention is: An image editing method based on Markov random fields and three-dimensional implicit representations includes the following steps: Step 1: Obtain the original image dataset, which contains a collection of image data. and each image The corresponding camera parameters; among which Represents the image elements in set S. Indicates the index of the image. Representing a real-world scenario Different random perspectives Frame image; the camera parameters include the camera extrinsic matrix M and the camera intrinsic matrix C; Step 2: Select any frame from S. As the image to be edited, select an object to be removed on the image and mark it. By identifying positive and negative pixels, we obtain the labeled image. ; Step 3: From the image The training set is divided from the remaining portion of the original image dataset containing the corresponding camera parameters. ; Step 4: Convert the pixel coordinates of each image that makes up S into three-dimensional world coordinates; Step 5: Use an image segmentation algorithm based on Markov random fields to segment the image respectively. Images and training set The image is segmented to generate an image. initial object mask and The initial object mask of the image; Step 6: Apply neural radiation fields to the images obtained in Step 5. initial object mask and The initial object mask of the image is optimized to obtain the final masked image. and Image with a mask; Step 7: Combining Image with mask on image with mask Fill in the missing positions to complete the image editing.

[0007] Furthermore, according to the image editing method based on Markov random fields and three-dimensional implicit representation, the generated image The method for initializing the object mask includes the following steps: Step 5-1-1: Initialize the image based on the markings made in Step 2. Each pixel in the image is labeled to represent a different segmentation region, where a positive example pixel is labeled 1 and a negative example pixel is labeled 0. Step 5-1-2: Based on the principles of Markov random fields, construct the first energy function, which consists of the sum of the data term and the smoothing term, as follows: (5) Among them, data items Used to measure pixels in image segmentation results i Assigned to label The probability, The smaller this accumulated value, the higher the probability that each pixel belongs to its assigned label; smoothing term Measuring adjacent pixels in image segmentation results i and j Each assigned label and Similarity, The smaller this accumulated value, the more similar the labels assigned to adjacent pixels; It is the balance coefficient; Step 5-1-3: Set the first energy function Minimize, to obtain Lowest value tag configuration As an image The initial object mask.

[0008] Furthermore, according to the image editing method based on Markov random fields and three-dimensional implicit representation, the generation of the training set... The method for initial object masking of an image includes the following steps: Step 5-2-1: For the training set Perform a difference operation on the pixel label values ​​of adjacent frames in the image; Step 5-2-2: Based on the principles of Markov random fields, construct the second energy function, which consists of three parts: a data term, a smoothing term, and a continuity penalty term, as follows: (7) Among them, data items Used to measure pixels in image segmentation results i Assigned to label The probability, The smaller this accumulated value, the higher the probability that each pixel belongs to its assigned label; continuity penalty term This measure indicates the degree to which the label values ​​at the same pixel location change in adjacent frames during image segmentation. A smaller value indicates that the segmentation results of adjacent frames are more similar; Smoothing term. Used to measure adjacent pixels in image segmentation results i and j Each assigned label and Similarity, The smaller this accumulated value, the more similar the labels assigned to adjacent pixels; and It is the balance coefficient; Step 5-2-3: Make the second energy function Minimize to obtain the second energy function Lowest value tag configuration As a training set The initial object mask of the image.

[0009] Furthermore, according to the image editing method based on Markov random fields and three-dimensional implicit representation, step 6 includes the following steps: Step 6-1: Based on the principle of nerve radiation fields, for For each pixel in each image with an initial object mask, generate a camera ray emanating from that pixel in world coordinates, and perform discrete point sampling on the camera ray. The number of sampling points is represented as... ; Step 6-2: Construct a neural radiation field and utilize the training set The image in the image, the sampling points obtained in step 6-1, and the image initial object mask and The initial object mask of the image is used to train the neural radiation field to obtain a trained neural radiation field. Step 6-3: Calculate the image using the trained neural radiation field. and training set The predicted probability that each pixel in the image will cover the object to be removed is calculated. For all pixels in each image whose predicted value is greater than a threshold k, a mask is applied to obtain the final masked image. and An image with a mask.

[0010] Furthermore, according to the image editing method based on Markov random fields and three-dimensional implicit representation, step 6-2 includes the following steps: Step 6-2-1: Define the three-dimensional implicit representation: Model the coordinates x of the sampling point on the camera ray, the direction d of the camera ray, and the volume density of the sampling point on the camera ray based on the principle of neural radiation field. RGB colors of the camera ray upsampling points The confidence level that the upsampling point of the camera ray is located inside the object to be removed, i.e., the region of interest. The interdependence between them; Step 6-2-2: Based on the principle of neural radiation fields and three-dimensional implicit representation, construct the network structure of neural radiation fields; Step 6-2-3: Perform volume rendering based on the principle of neural radiation fields to obtain the predicted RGB color values ​​of the pixels traversed by the camera rays. ; Step 6-2-4: Calculate the predicted probability that the object to be removed is covered by pixels through which the camera ray passes, using the following formula. (15) That This indicates the confidence level that the sampling point is located in the region of interest; This represents the cumulative value of the volume density corresponding to the sampling point; ; Indicates the distance between adjacent sampling points; Step 6-2-5: Calculate the loss caused by step 6-2-3. This loss is achieved by analyzing the true RGB color values ​​of the pixels traversed by the camera rays. Compared with the RGB color prediction values ​​obtained in step 6-2-3 The mean square error between them is used for calculation; Step 6-2-6: Calculate the loss caused by step 6-2-4 using cross-entropy. : Step 6-2-7: [Regarding...] and Sum to calculate the total loss; Step 6-2-8: Repeat steps 6-2-3 to 6-2-7 until the preset maximum number of iterations is reached, save the network parameters, and obtain the trained neural radiation field.

[0011] Furthermore, according to the image editing method based on Markov random fields and three-dimensional implicit representation, the network structure of the neural radiation field is as follows: the neural radiation field network consists of multiple fully connected layers. First, some fully connected layers are used to process the coordinates x of the sampling points, and the volume density of the sampling points is output. Confidence that the sampling point is located in the region of interest And a 256-dimensional feature vector; then the 256-dimensional feature vector is concatenated with the camera ray direction d, and then passed through a fully connected layer to output the RGB color value c of the sampling point.

[0012] Furthermore, according to the image editing method based on Markov random fields and three-dimensional implicit representation, step 7 includes the following steps: Step 7-1: Using an existing image inpainting model based on Markov random fields, inpaint the masked image obtained in Step 6. and The image with the mask is completed, and the RGB color values ​​of the pixels in the completed image are defined as RGB color reference values. Step 7-2: Using the image completed in Step 7-1 as a supervision signal, the neural radiation field trained in Step 6 is used to reconstruct the new scene after the object to be removed is completed, thereby improving the masked image. and Repairing images with masks; Step 7-3: Perform volume rendering based on the principle of neural radiation fields to obtain a masked image. and The predicted RGB color values ​​of pixels within the area to be filled in on an image with a mask. Step 7-4: Calculate the image mask completion loss, including color prediction loss and depth smoothing loss; Step 7-5: Repeat steps 7-2 and 7-4 until the preset maximum number of iterations is reached, save the network parameters, and obtain the trained neural radiation field for mask repair; Steps 7-6: Calculate the image using the trained neural radiation field for mask restoration. All areas to be completed The RGB color values ​​of the pixels within the image Fill in the missing positions to complete the image editing.

[0013] Furthermore, according to the image editing method based on Markov random fields and three-dimensional implicit representation, the color prediction loss is calculated by the mean square error between the RGB color reference value of the completed pixel obtained in step 7-1 and the RGB color prediction value of the pixel in the area to be completed in the image obtained in step 7-3; the formula for calculating the depth smoothing loss is as follows: = (twenty one) in This indicates that the camera ray r passes through coordinates ( i,j The depth corresponding to the pixel; This indicates that the camera ray r passes through coordinates ( i +1 ,j The depth corresponding to the pixel; This indicates that the camera ray r passes through coordinates ( i,j The depth corresponding to the pixel with +1); This is the set of all camera rays that pass through the pixels of the region to be filled.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. The method provided by this invention uses a multi-view image set for training, which can effectively process the objects to be removed in consecutive image frames, obtain the initial object mask of consecutive frames, make full use of the spatial continuity in the image sequence, and improve the accuracy of segmentation.

[0015] 2. This invention extends the image segmentation problem from two-dimensional images to three-dimensional scenes, and uses three-dimensional implicit representations to reconstruct the scenes described by images from different perspectives, which is used to optimize the initial object mask and further enhance the three-dimensional perception characteristics of the segmentation results.

[0016] 3. This invention models the interdependencies between pixels based on the principle of Markov random fields. Since Markov random fields are very sensitive to image texture correlation, they maintain the details of local areas of the image and improve the visual quality of image editing.

[0017] 4. This invention uses neural radiation fields as a specific implementation of three-dimensional implicit representation. The neural radiation fields predict the color and volume density of each sampling point in the three-dimensional scene. Since color and volume density depend on lighting changes, occlusion relationships, scene depth, object material, and object geometry, the method provided by this invention can generate more realistic and delicate completion effects visually. Attached Figure Description

[0018] Figure 1 This is a flowchart of the image editing method based on Markov random fields and three-dimensional implicit representation in this embodiment; Figure 2 This is a rough diagram showing the dataset used in this embodiment. Figure 3 This is a schematic diagram of the network structure of the neural radiation field in this embodiment; Figure 4 This is a flowchart of step 5 in this embodiment; Figure 5 This is a flowchart of step 6 in this embodiment; Figure 6 This is a flowchart of step 7 in this embodiment; Figure 7 This is a comparison chart of the image editing effects of the method of the present invention with those of traditional GAN ​​and CNN methods. Detailed Implementation

[0019] To facilitate understanding of this application, a more comprehensive description of this application will be provided below with reference to the accompanying drawings.

[0020] Figure 1 This is a flowchart of the image editing method based on Markov random fields and three-dimensional implicit representation in this implementation. Figure 1 As shown, the image editing method based on Markov random fields and three-dimensional implicit representation includes the following steps: Step 1: Obtain the original image dataset, which contains a collection of image data. And the camera parameters corresponding to each image; among which Represents the image elements in set S. Indicates the index of the image. Represents different random perspectives of a real scene Frame image; the camera parameters include the camera extrinsic matrix M and the camera intrinsic matrix G.

[0021] Those skilled in the art will readily understand that original image datasets can be obtained either manually or through publicly available channels. In this embodiment, the publicly available real-world scene dataset NeRF_LLFF, obtained from the internet, is used as the original image dataset. This dataset is a multi-view dataset obtained by capturing the same real-world scene using a camera. The dataset contains a collection of image data and the camera parameters corresponding to each image. This dataset contains two types of data: the first type is the collection of image data. ,in, Represents the image elements in the set. Indicates the index of the image. This indicates that for a real-world scene, footage was taken from different angles. The first category is the frame image; the second category is the camera parameters corresponding to each image, which include the camera extrinsic matrix M (which describes the camera's position and orientation) and the camera intrinsic matrix G (which describes the camera's focal length and optical center).

[0022] Step 2: Select any frame from the image data set S. As the image to be edited, arbitrarily select an object to be removed from the image and mark positive and negative pixels to obtain the marked image. ; In this embodiment, any frame of image is selected from the image data set S. As the image to be edited, any frame of the image data set S is used. Processing: First, from the image data set Any frame of the image Arbitrarily select an object to be removed. The selection method involves randomly selecting a small number of pixels and marking them to roughly identify different regions in the image, such as... Figure 2 As shown, a portion of these marked pixels are used as positive examples (e.g.) Figure 2 The pixels marked with solid dots are used to indicate the approximate location of the object to be removed, while the other part serves as a negative example (e.g., ...). Figure 2 The pixels marked by the inner ring (i.e., interference points) are used to indicate locations unrelated to the object to be removed, ultimately resulting in the marked image. .

[0023] Step 3: From the image The training set is divided from the remaining portion of the original image dataset containing the corresponding camera parameters. ; In this embodiment, except for the image to be edited The remaining portion of the original image dataset, along with its corresponding camera parameters, is divided into a training set and a test set in a 7:1 ratio. Specifically, the images in the remaining portion of the original image dataset are arranged in ascending or descending order of their file numbers. Every seven images are selected as the image data in the test set, and the remaining images are used as the image data in the training set. The camera parameters corresponding to these images are also randomly assigned to either the training set or the test set. The set of image data in the training set is called the training set. ,in, Represents the image elements in the set. Represents the index of an image; there are a total of [number] in the training set. Zhang images; the image data set in the test set is called... ,in, Represents the image elements in the set. Represents the index of the image; there are a total of [number] in the test set. Zhang image.

[0024] Step 4: Using existing camera calibration principles, transform the pixel coordinates of each image constituting the image data set S from pixel coordinates on the two-dimensional imaging plane to three-dimensional world coordinates; The pixel coordinates of each image in the image data set S are the pixel coordinates on the two-dimensional imaging plane. In this embodiment, as... Figure 4 As shown, the pixel coordinates on the two-dimensional imaging plane Through camera intrinsic parameter matrix G Mapped to camera coordinates The following formula can be used to solve this problem: (1) in G This refers to the camera intrinsic parameter matrix; specifically, the camera intrinsic parameter matrix. G Represented as a 3x4 matrix: (2) in , The focal length of the camera on the x and y axes. The coordinates are the center point coordinates of the two-dimensional imaging plane.

[0025] Then, the camera coordinates Mapping to world coordinates via camera extrinsic matrix M The following formula can be used to solve this problem: (3) Where M is the camera extrinsic parameter matrix, specifically, the camera extrinsic parameter matrix M is represented as a 4x4 matrix: (4) Where V is a 3x3 camera rotation matrix. , , These represent the cosine values ​​of the angles between the x-axis of the camera coordinate system and the x-axis, y-axis, and z-axis of the world coordinate system, respectively. , , These represent the cosine values ​​of the angles between the y-axis of the camera coordinate system and the x-axis, y-axis, and z-axis of the world coordinate system, respectively. , , Let represent the cosines of the angles between the z-axis of the camera coordinate system and the x-axis, y-axis, and z-axis of the world coordinate system, respectively. W is the camera's translation vector. This represents the coordinates of the camera coordinate system origin in the world coordinate system.

[0026] Step 5: As Figure 5 As shown, an image segmentation algorithm based on Markov random fields is used to segment the image respectively. Images and training set The images in the image are segmented to obtain the initial object masks of these images; Step 5-1: Generate image The initial object mask includes the following steps: Step 5-1-1: Initialize the image based on the markings made in Step 2. The label for each pixel in the image.

[0027] In this embodiment, the image is initialized based on the markings made in step 2. Each pixel in the image is labeled, and these labels represent different segmentation regions. Pixels that are positive examples are labeled with a label of 1, and pixels that are negative examples are labeled with a label of 0.

[0028] Step 5-1-2: Construct the first energy function based on the principle of Markov random fields; The energy function describes the "quality" of a certain label configuration state of pixels in an image. Its core idea is to find the optimal image segmentation scheme by minimizing the energy function. Specifically, in this embodiment, the segmentation goal is to find a label configuration L (i.e., the image...) The set of assigned labels for all pixels in the image, making the entire image... The energy is minimum. First energy function. It consists of two parts: the data term and the smoothing term. The calculation formula is as follows: (5) Among them, data items The pixels in the image segmentation results were measured. i Assigned to label The probability, The smaller this accumulated value, the higher the probability that each pixel belongs to its assigned label; smoothing term Measuring neighboring pixels in the image segmentation result i and j Each assigned label and Similarity, The smaller this accumulated value, the more similar the labels assigned to adjacent pixels, meaning the image segmentation is smoother and more natural; It is the balance coefficient.

[0029] Step 5-1-3: Set the first energy function Minimize to obtain the first energy function Lowest value tag configuration As an image The initial object mask.

[0030] Specifically, this implementation solves the energy function minimization problem using the existing simulated annealing algorithm. The process of minimizing the energy function is to iteratively adjust the set of labels assigned to pixels until a value is found that minimizes the energy function. Lowest value tag configuration Thus completing the image The segmentation. Output the final pixel label configuration as the image. The initial object mask.

[0031] Step 5-2: Generate training set The initial object mask for the image includes the following steps: Step 5-2-1: For the training set The pixel label values ​​of adjacent frames in the image are differentially calculated.

[0032] training set Adjacent frame images in the image are considered as images taken at time q and time q-1. The pixels in the image at time q The tag, The pixels in the image at time q-1 The labels are randomly initialized. The difference between the pixel label values ​​of two frames is calculated using the following formula: (6) Step 5-2-2: Construct the second energy function based on the principle of Markov random fields.

[0033] Second energy function It consists of three parts: the data term, the smoothing term, and the continuity penalty term, and the calculation formula is as follows: (7) Among them, data items The pixels in the image segmentation results were measured. i Assigned to label The probability, The smaller this accumulated value, the higher the probability that each pixel belongs to its assigned label; continuity penalty term This measures the degree to which the label values ​​at the same pixel location change in adjacent frames during image segmentation. A smaller value indicates that the segmentation results of adjacent frames are more similar; the smoothing term... Measuring neighboring pixels in the image segmentation result i and j Each assigned label and Similarity, The smaller this accumulated value, the more similar the labels assigned to adjacent pixels, meaning the image segmentation is smoother and more natural; and It is the balance coefficient.

[0034] Step 5-2-3: Make the second energy function Minimize to obtain the second energy function Lowest value tag configuration As a training set The initial object mask of the image.

[0035] Specifically, this implementation solves the problem of minimizing the second energy function using the existing simulated annealing algorithm. The process of minimizing the second energy function is to iteratively adjust the set of labels assigned to pixels until a value is found that minimizes the second energy function. Minimum tag configuration Thus, the training set is completed. Image segmentation. Output the final pixel label configuration as the training set. The initial object mask of the image.

[0036] Step 6: As Figure 6 As shown, the image obtained in step 5 is processed using a neural radiation field. initial object mask and The initial object mask of the image is optimized to obtain the final masked image. and Image with a mask; Since the images in the NeRF_LLFF open-source dataset are not continuous video frames but images taken from random viewpoints, the initial object mask obtained in step 5 is inaccurate and needs optimization. Neural radiation fields are an existing method for implicitly representing three-dimensional scenes, reconstructing 3D scenes from 2D images taken from different viewpoints. Step 6-1: Based on the principle of nerve radiation fields, for For each pixel in each image with an initial object mask, generate a camera ray emanating from that pixel in world coordinates, and perform discrete point sampling on the camera ray.

[0037] Specifically, for For each pixel in each image with an initial object mask, generate a camera ray emanating from that pixel in world coordinates. : (8) Where o is the origin of the camera ray. d represents the direction of the camera ray. The camera ray originates at the camera position, passes through every pixel on the image plane, and extends into the pre-reconstructed 3D scene. Random, uniform sampling is performed along the camera ray, using the following formula: (9) in Indicates the first ray on the camera ray. The location of each sampling point Indicates the location of the near-end sampling point on the camera ray. Indicates the location of the far-end sampling point on the camera ray. This indicates the number of sampling points.

[0038] Step 6-2: Construct a neural radiation field and utilize the training set The image in the image, the sampling points obtained in step 6-1, and the image initial object mask and The initial object mask of the image is used to train the neural radiation field, resulting in a trained neural radiation field. ; Step 6-2-1: Define the three-dimensional implicit representation: Model the coordinates x of the sampling point on the camera ray, the direction d of the camera ray, and the volume density of the sampling point based on the principle of neural radiation field. RGB colors of sampling points Confidence that the sampling point is located inside the object to be removed, i.e., the region of interest. The interdependence between them; The neural radiation field will train the set The image in the middle and the image obtained in step 5 initial object mask and The initial object mask of the image is used as input to model the coordinates x of the sampling points on the camera ray, the direction d of the camera ray, and the volume density of the sampling points. RGB colors of sampling points Confidence that the sampling point is located inside the object to be removed, i.e., the region of interest. The interdependence between them. This interdependence is defined as a function mapping F: F (10) Step 6-2-2: Based on the principle of neural radiation fields and three-dimensional implicit representation, construct the specific network structure of neural radiation fields.

[0039] like Figure 3 As shown, the neural radiation field network consists of multiple fully connected layers. Since the volume density value of the sampling point and whether the sampling point is inside the object to be removed are independent of the camera ray direction and only depend on the position of the sampling point, the network first uses 8 fully connected layers to process the x-coordinate of the sampling point and outputs the volume density of the sampling point. Confidence that the sampling point is located in the region of interest The input sample point coordinates (x) and a 256-dimensional feature vector are then concatenated with the camera ray direction (d), and finally passed through a fully connected layer to output the RGB color value (c) of the sample point. Specifically, the input sample point coordinates (x) and camera ray direction (d) are respectively position-encoded. , The input data is mapped to a high-dimensional feature space. The calculation process is as follows: (11) (12) Step 6-2-3: Perform volume rendering based on the principle of neural radiation fields to obtain the predicted RGB color values ​​of the pixels traversed by the camera rays. ; Volume rendering refers to the process of combining the color and opacity of each sampling point along the camera ray to obtain the RGB color values ​​of the pixels passed through by the camera ray.

[0040] First, calculate the cumulative value of the volume density corresponding to each sampling point on the ray. It is used to describe the transmittance at sampling point t on the camera ray, that is, the percentage of the camera ray that is not absorbed by matter in the scene before reaching sampling point t: (13) Where exp is the natural exponential function. Indicates camera ray The volume density of the sampling point at position s.

[0041] Then, the camera ray along the d direction r The RGB color value corresponding to each sampling point t According to the cumulative value of the volume density corresponding to the sampling point Bulk density Integrate to obtain the RGB color prediction value of the pixel passed through by the camera ray. : (14) Camera ray r Discretize the sampling points and approximate the integral formula by summing the results to obtain the color value of the pixel traversed by the camera ray. : (15) in This represents the cumulative value of the volume density corresponding to the sampling point. , This indicates the distance between adjacent sampling points.

[0042] Step 6-2-4: Calculate the predicted probability that the object to be removed will be covered by pixels through which the camera ray passes.

[0043] Similar to the formula for calculating the predicted color value of a pixel in volume rendering, a weighted value is defined to describe the color of the pixel along the camera ray passing through it. arrive The probability that all sampling points within the range are within the object to be removed. Indicates the location of the near-end sampling point on the camera ray. The location of the far-end sampling point on the camera ray is represented in the following form: (16) in This indicates the probability that the object to be removed is covered by pixels traversed by the camera ray; This indicates the confidence level that the sampling point is located inside the object to be removed, i.e., the region of interest. This represents the cumulative value of the volume density corresponding to the sampling point. , This indicates the distance between adjacent sampling points.

[0044] Step 6-2-5: Calculate the loss caused by step 6-2-3. .

[0045] This loss is achieved by analyzing the true RGB color values ​​of the pixels traversed by the camera rays. Compared with the RGB color prediction values ​​obtained in step 6-2-3 The mean square error between them is used for calculation, and the formula is as follows: (17) in This is the set of all camera rays that pass through the pixel.

[0046] Step 6-2-6: Calculate the loss caused by step 6-2-4 .

[0047] The cross-entropy calculation is performed using the following formula: (18) in This is the actual label of the pixel through which the camera ray r passes, and its value is either 0 or 1. This value is used when the camera ray r passes through pixels covered by the object to be removed. The value is 1, otherwise The value is 0. It is the predicted probability value of the object to be removed described in step 6-2-4 being covered by pixels through which the camera ray passes. This is the set of all camera rays that pass through the pixels of the images in the training set. It represents the total number of camera rays.

[0048] Step 6-2-7: Calculate the total loss, using the following formula: = (19) in is the weight, which is a hyperparameter. It should be noted that the method of this invention sets a gradient stopping strategy, which is for... Don't Loss transmitted to The gradient update is guided by the loss, because if the total loss is calculated... At that time, The loss is also added, and an untrained neural radiation field can lead to the generation of incorrect scene geometry, resulting in worse image masking optimization results.

[0049] Step 6-2-8: Repeat steps 6-2-3 to 6-2-7 until the maximum number of iterations is reached, save the network parameters, and obtain the trained neural radiation field.

[0050] Step 6-3: Calculate the image using the trained neural radiation field based on the currently saved network parameters. and training set The predicted probability that each pixel in the image will cover the object to be removed. For each image, all predicted values Pixels larger than a threshold k are used as masks (k is a hyperparameter) to obtain the final masked image. and Image with a mask; The "masking" operation refers to adding a mask layer to the original image: in this implementation, the predicted value... Pixels larger than k are filled with black.

[0051] Step 7: Combining Image with mask on image with mask Fill in the missing positions to complete the image editing; like Figure 7 As shown, this step includes the following steps: Step 7-1: Using an existing image inpainting model based on Markov random fields, inpaint the masked image obtained in Step 6. and The masked image is used for completion, and the completed image is used as a supervision signal to supervise the reconstruction of the new scene after the object to be removed is completed by the neural radiation field.

[0052] In this embodiment, the image with a mask and The images with masks are all divided into two parts: the known region D and the region to be completed. The Criminisi digital image inpainting model based on Markov random field matching criteria is used to inpaint the region. The RGB color values ​​of each pixel within the area are calculated, and the calculated RGB color values ​​are used to fill in the area. Each pixel within the image is filled with color, creating a masked image. and Regions to be filled in on a masked image After all the filling is completed, a set of completed images is generated. . It is the first The completed image contains a total of [number missing]. The image is completed. The RGB color values ​​of the pixels in the completed image are defined as... This value is used as a reference value to supervise the reconstruction of the new scene after the object to be removed is replaced in subsequent steps.

[0053] Step 7-2: Use the neural radiation field trained in Step 6 to reconstruct the new scene after the object to be removed is completed, thereby reconstructing the masked image. and The image with a mask is repaired.

[0054] The neural radiation field saved in step 6 is used to reconstruct the new scene after the object to be removed is completed, and a depth map is obtained. The depth calculation formula is as follows: (20) in Indicates sampling point Corresponding bulk density The cumulative value, This indicates the distance between adjacent sampling points.

[0055] Step 7-3: Perform volume rendering based on the principle of neural radiation fields to obtain a masked image. and Regions to be filled in on a masked image RGB color prediction values ​​of pixels within .

[0056] The color and opacity of each sampling point on the camera ray are combined to obtain the RGB color value of the pixel point through which the camera ray passes. The calculation method of the RGB color value is the same as that in step 6-2-3.

[0057] Step 7-4: Calculate the image mask completion loss, which consists of color prediction loss and depth smoothing loss.

[0058] Step 7-4-1: Calculate the color prediction loss caused by step 7-3.

[0059] The color prediction loss is obtained by using the RGB color reference value of the completed pixel obtained in step 7-1. Compared with the RGB values ​​predicted in step 7-3 The mean square error between them is used for calculation, and the formula is as follows: (twenty one) in This is the set of all camera rays that pass through the pixels of the region to be filled.

[0060] Step 7-4-2: Calculate the depth smoothing loss caused by step 7-2. The purpose of calculating the depth smoothing loss is to minimize the depth distance between every pair of adjacent pixels in the image, ensuring the smoothness of the scene's geometry as reflected in the completed image. The formula is as follows: = (twenty two) in This indicates that the camera ray r passes through coordinates ( i,j The depth corresponding to the pixel; This indicates that the camera ray r passes through coordinates ( i +1,j The depth corresponding to the pixel; This indicates that the camera ray r passes through coordinates ( i,j The depth corresponding to the pixel with +1); This is the set of all camera rays that pass through the pixels of the region to be filled.

[0061] Step 7-4-3: Calculate the total loss as the completion loss.

[0062] The formula is as follows: = (twenty three) in , where is the weight, is a hyperparameter.

[0063] Step 7-5: Repeat steps 7-2 and 7-4 until the maximum number of iterations is reached. Save the network parameters to obtain the trained neural radiation field for mask restoration. .

[0064] Step 7-6: Calculate the image based on the currently saved network parameters. All areas to be completed The RGB color values ​​of the pixels within the image The missing positions were filled in, completing the image editing.

[0065] Figure 7 The method of this invention is compared with traditional GAN ​​and CNN methods on the test set. The comparison images above show the effects. Image (a) is the original image, image (b) is the labeled image, image (c) is the image editing result obtained using the method of this invention, image (d) is the image editing result based on the GAN method, and image (e) is the image editing result based on the CNN method. Figure 7 It is evident that the image editing results obtained using the method of this invention are significantly better than those based on traditional GAN ​​and CNN methods.

[0066] It should be understood that, inspired by the technical concept of this invention, those skilled in the art can make various improvements or modifications based on the above content without departing from the scope of this invention, and these modifications still fall within the protection scope of this invention.

Claims

1. An image editing method based on Markov random fields and three-dimensional implicit representation, characterized in that, The method includes the following steps: Step 1: Obtain the original image dataset, which contains a collection of image data. And the camera parameters corresponding to each image; among which Represents the image elements in set S. Indicates the index of the image. Represents different random perspectives of a real scene Frame image; the camera parameters include the camera extrinsic matrix M and the camera intrinsic matrix C; Step 2: Select any frame from S. As the image to be edited, select an object to be removed on the image and mark positive and negative pixels to obtain the marked image. ; Step 3: From the image The training set is divided from the remaining portion of the original image dataset containing the corresponding camera parameters. ; Step 4: Convert the pixel coordinates of each image that makes up S into three-dimensional world coordinates; Step 5: Use an image segmentation algorithm based on Markov random fields to segment the image respectively. Images and training set The image is segmented to generate an image. initial object mask and The initial object mask of the image; Step 6: Apply neural radiation fields to the images obtained in Step 5. initial object mask and The initial object mask of the image is optimized to obtain the final masked image. and Image with a mask; Step 7: Combining Image with mask on image with mask Fill in the missing positions to complete the image editing.

2. The image editing method based on Markov random fields and three-dimensional implicit representation according to claim 1, characterized in that, The generated image The method for initializing the object mask includes the following steps: Step 5-1-1: Initialize the image based on the markings made in Step 2. Each pixel in the image is labeled to represent a different segmentation region, where a positive example pixel is labeled 1 and a negative example pixel is labeled 0. Step 5-1-2: Based on the principles of Markov random fields, construct the first energy function, which consists of the sum of the data term and the smoothing term, as follows: ； Among them, data items Used to measure how pixel i is assigned to a label in the image segmentation result. The probability, The smaller this accumulated value, the higher the probability that each pixel belongs to its assigned label; smoothing term The label assigned to each of the adjacent pixels i and j in the image segmentation result. and Similarity, The smaller this accumulated value, the more similar the labels assigned to adjacent pixels; It is the balance coefficient; Step 5-1-3: Set the first energy function Minimize, to obtain Lowest value tag configuration As an image The initial object mask.

3. The image editing method based on Markov random fields and three-dimensional implicit representation according to claim 1, characterized in that, The generation of training sets The method for initial object masking of an image includes the following steps: Step 5-2-1: For the training set Perform a difference operation on the pixel label values ​​of adjacent frames in the image; Step 5-2-2: Based on the principles of Markov random fields, construct the second energy function, which consists of three parts: a data term, a smoothing term, and a continuity penalty term, as follows: ； Among them, data items Used to measure how pixel i is assigned to a label in the image segmentation result. The probability, The smaller this accumulated value, the higher the probability that each pixel belongs to its assigned label; continuity penalty term This measure indicates the degree to which the label values ​​at the same pixel location change in adjacent frames during image segmentation. A smaller value indicates that the segmentation results of adjacent frames are more similar; Smoothing term. Used to measure the labels assigned to adjacent pixels i and j in the image segmentation result. and Similarity, The smaller this accumulated value, the more similar the labels assigned to adjacent pixels; and It is the balance coefficient; Step 5-2-3: Make the second energy function Minimize to obtain the second energy function Lowest value tag configuration As a training set The initial object mask of the image.

4. The image editing method based on Markov random fields and three-dimensional implicit representation according to claim 1, characterized in that, Step 6 includes the following steps: Step 6-1: Based on the principle of nerve radiation fields, for For each pixel in each image with an initial object mask, generate a camera ray emanating from that pixel in world coordinates, and perform discrete point sampling on the camera ray. The number of sampling points is represented as... ; Step 6-2: Construct a neural radiation field and utilize the training set The image in the image, the sampling points obtained in step 6-1, and the image initial object mask and The initial object mask of the image is used to train the neural radiation field to obtain a trained neural radiation field. Step 6-3: Calculate the image using the trained neural radiation field. and training set The predicted probability that each pixel in the image will cover the object to be removed is calculated. For all pixels in each image whose predicted value is greater than a threshold k, a mask is applied to obtain the final masked image. and An image with a mask.

5. The image editing method based on Markov random fields and three-dimensional implicit representation according to claim 4, characterized in that, Step 6-2 includes the following steps: Step 6-2-1: Define the three-dimensional implicit representation: Model the coordinates x of the sampling point on the camera ray, the direction d of the camera ray, and the volume density of the sampling point on the camera ray based on the principle of neural radiation field. RGB colors of the camera ray upsampling points The confidence level that the upsampling point of the camera ray is located inside the object to be removed, i.e., the region of interest. The interdependence between them; Step 6-2-2: Based on the principle of neural radiation fields and three-dimensional implicit representation, construct the network structure of neural radiation fields; Step 6-2-3: Perform volume rendering based on the principle of neural radiation fields to obtain the predicted RGB color values ​​of the pixels traversed by the camera rays. ; Step 6-2-4: Calculate the predicted probability that the object to be removed is covered by pixels through which the camera ray passes, using the following formula. ； That This indicates the confidence level that the sampling point is located in the region of interest; This represents the cumulative value of the volume density corresponding to the sampling point; ; Indicates the distance between adjacent sampling points; Step 6-2-5: Calculate the loss caused by step 6-2-3. This loss is achieved by analyzing the true RGB color values ​​of the pixels traversed by the camera rays. Compared with the RGB color prediction values ​​obtained in step 6-2-3 The mean square error between them is used for calculation; Step 6-2-6: Calculate the loss caused by step 6-2-4 using cross-entropy. : Step 6-2-7: [Regarding...] and Sum to calculate the total loss; Step 6-2-8: Repeat steps 6-2-3 to 6-2-7 until the preset maximum number of iterations is reached, save the network parameters, and obtain the trained neural radiation field.

6. The image editing method based on Markov random fields and three-dimensional implicit representation according to claim 5, characterized in that, The network structure of the neural radiation field is as follows: the neural radiation field network consists of multiple fully connected layers. First, some fully connected layers are used to process the coordinate x of the sampling points, and then the volume density of the sampling points is output. Confidence that the sampling point is located in the region of interest And a 256-dimensional feature vector; then the 256-dimensional feature vector is concatenated with the camera ray direction d, and then passed through a fully connected layer to output the RGB color value c of the sampling point.

7. The image editing method based on Markov random fields and three-dimensional implicit representation according to claim 1, characterized in that, Step 7 includes the following steps: Step 7-1: Using an existing image inpainting model based on Markov random fields, inpaint the masked image obtained in Step 6. and The image with the mask is completed, and the RGB color values ​​of the pixels in the completed image are defined as RGB color reference values. Step 7-2: Using the image completed in Step 7-1 as a supervision signal, the neural radiation field trained in Step 6 is used to reconstruct the new scene after the object to be removed is completed, thereby improving the masked image. and Repairing images with masks; Step 7-3: Perform volume rendering based on the principle of neural radiation fields to obtain a masked image. and The predicted RGB color values ​​of pixels within the area to be filled in on an image with a mask. Step 7-4: Calculate the image mask completion loss, including color prediction loss and depth smoothing loss; Step 7-5: Repeat steps 7-2 and 7-4 until the preset maximum number of iterations is reached, save the network parameters, and obtain the trained neural radiation field for mask repair; Steps 7-6: Calculate the image using the trained neural radiation field for mask restoration. All areas to be completed The RGB color values ​​of the pixels within the image Fill in the missing positions to complete the image editing.

8. The image editing method based on Markov random fields and three-dimensional implicit representation according to claim 7, characterized in that, The color prediction loss is calculated using the mean square error between the RGB color reference value of the completed pixel obtained in step 7-1 and the RGB color prediction value of the pixel in the area to be completed in the image obtained in step 7-3; the formula for calculating the depth smoothing loss is as follows: ； in This represents the depth through which the camera ray r passes the pixel with coordinates (i,j); This represents the depth through which the camera ray r passes the pixel with coordinates (i+1,j); This represents the depth through which the camera ray r passes the pixel with coordinates (i,j+1); This is the set of all camera rays that pass through the pixels of the region to be filled.

Images