Image editing method and device based on neural radiation field, equipment and storage medium

Abstract

The invention provides an image editing method and device based on a neural radiation field, equipment and a storage medium, and the method comprises the steps: obtaining a plurality of to-be-processed images, and carrying out the sampling processing of the to-be-processed images, so as to obtain the pixel information of a plurality of sampling points; pixel information of the sampling points is input into a preset neural radiation field optimization model, feature information corresponding to the sampling points is output through the preset neural radiation field optimization model, and radiation rotation invariant constraints are added into the preset neural radiation field optimization model; generating a target neural renderer according to the feature information corresponding to the sampling point; generating a three-dimensional neural point cloud segmentation mask according to the target neural renderer; in response to an editing request of a user, generating an edited image according to the three-dimensional neural point cloud segmentation mask; the technical effect of improving the image editing efficiency and editing controllability based on the neural radiation field is achieved.

Description

Technical Field

[0001] The present application relates to the field of image processing technologies, and in particular, to an image editing method, apparatus, device, and storage medium based on a neural radiance field. Background Art

[0002] Modeling and rendering to generate realistic novel views of a real scene from multi-view images is a core problem in computer vision and graphics. Neural implicit fields (NeRFs) have recently shown remarkable results in novel view synthesis and have become a promising solution to replace traditional explicit 3D representations such as meshes or volumes. A NeRF is an implicit function parameterized by a neural network for representing geometric or physical properties (such as shape, color, density, etc.) in space. However, these implicit modeling methods are extremely complex for users to edit scenes and lack fine-grained and efficient editing capabilities, thus limiting the generation of diverse 3D content. Therefore, developing a more efficient and high-fidelity image editing method based on neural radiance fields has become an application-oriented direction.

[0003] In the prior art, image editing methods based on neural radiance fields mainly achieve image editing of neural radiance fields by combining explicit mesh representations, point-based implicit representations, or by means of large models for text instruction editing.

[0004] However, in the prior art, due to the lack of efficient fine-grained editing capabilities, it is difficult to achieve fast and controllable interactive editing while maintaining high-quality rendering, and there are technical problems of low efficiency and poor controllability in image editing based on neural radiance fields. Summary of the Invention

[0005] The image editing method, apparatus, device, and storage medium based on a neural radiance field provided by the present application are used to achieve the technical effect of improving the efficiency and controllability of image editing based on a neural radiance field.

[0006] In a first aspect, the present application provides an image editing method based on a neural radiance field, including:

[0007] Obtain a plurality of images to be processed, and perform sampling processing on the images to be processed to obtain pixel information of a plurality of sampling points;

[0008] Input the pixel information of the sampling points into a preset neural radiance field optimization model to output feature information corresponding to the sampling points through the preset neural radiance field optimization model, where a radiance rotation invariance constraint is added to the preset neural radiance field optimization model;

[0009] Generate a target neural renderer according to the feature information corresponding to the sampling points;

[0010] Generate a 3D neural point cloud segmentation mask according to the target neural renderer;

[0011] In response to a user's editing request, generate an edited image according to the 3D neural point cloud segmentation mask.

[0012] In a possible implementation manner, generating a target neural renderer according to the feature information corresponding to the sampling points includes:

[0013] Determine multiple point-based scene representations for specific scenes according to the feature information corresponding to the sampling points;

[0014] Obtain the original neural renderer;

[0015] Train the original neural renderer according to the multiple point-based scene representations for specific scenes to obtain the target neural renderer.

[0016] In a possible implementation manner, generating a 3D neural point cloud segmentation mask according to the target neural renderer includes:

[0017] Generate a segmentation mask and a log probability map according to the target neural renderer;

[0018] Construct a real-time mapping function;

[0019] Generate a 3D neural point cloud segmentation mask according to the real-time mapping function, the segmentation mask, and the log probability map.

[0020] In a possible implementation manner, after generating a 3D neural point cloud segmentation mask according to the target neural renderer, it further includes:

[0021] Determine a neural mask region growth and pruning strategy according to the spatial Euclidean distance and feature cosine distance between neural point clouds;

[0022] Optimize the 3D neural point cloud segmentation mask according to the neural mask region growth and pruning strategy to obtain an optimized 3D neural point cloud segmentation mask;

[0023] Correspondingly, in response to a user's editing request, generating an edited image according to the 3D neural point cloud segmentation mask includes:

[0024] In response to a user's editing request, generate an edited image according to the optimized 3D neural point cloud segmentation mask.

[0025] In a possible implementation manner, the editing request includes a click operation by the user on the image to be processed;

[0026] Correspondingly, in response to a user's editing request, generating an edited image according to the 3D neural point cloud segmentation mask includes:

[0027] In response to a click operation, determine, according to a three-dimensional neural point cloud segmentation mask, the neural point cloud on the image to be processed by the user, where the neural point cloud is used to select a target element;

[0028] Perform a fine editing operation on the target element according to the click operation to obtain a fine edited image.

[0029] In a possible implementation manner, in response to a user's editing request, generate an edited image according to a three-dimensional neural point cloud segmentation mask, further including:

[0030] In response to a click operation, determine, according to a three-dimensional neural point cloud segmentation mask, the synthesis target of the user for different specific scenarios;

[0031] Perform a synthesis operation on the synthesis target according to the click operation to obtain a cross-scene synthesis image.

[0032] In a possible implementation manner, before inputting the pixel information of the sampling points into a preset neural radiance field optimization model to output the feature information corresponding to the sampling points through the preset neural radiance field optimization model, further including:

[0033] Generate a neural embedding function according to the radiation rotation invariance constraint;

[0034] Based on the neural embedding function, establish a rotation-invariant neural inverse distance weighted interpolation module;

[0035] Embed the rotation-invariant neural inverse distance weighted interpolation module into the original neural radiance field optimization model to obtain a preset neural radiance field optimization model.

[0036] In a second aspect, the present application provides an image editing device based on a neural radiance field, including:

[0037] An image processing module, configured to obtain a plurality of images to be processed and perform sampling processing on the images to be processed to obtain the pixel information of a plurality of sampling points;

[0038] A feature processing module, configured to input the pixel information of the sampling points into a preset neural radiance field optimization model to output the feature information corresponding to the sampling points through the preset neural radiance field optimization model, where a radiation rotation invariance constraint is added to the preset neural radiance field optimization model;

[0039] A rendering processing module, configured to generate a target neural renderer according to the feature information corresponding to the sampling points;

[0040] A mask processing module, configured to generate a three-dimensional neural point cloud segmentation mask according to the target neural renderer;

[0041] An editing module, configured to generate an edited image according to a three-dimensional neural point cloud segmentation mask in response to a user's editing request.

[0042] In a possible implementation manner, the rendering processing module is further configured to:

[0043] Determine multiple point-based scene representations of specific scenes according to the feature information corresponding to the sampling points;

[0044] Obtain an original neural renderer;

[0045] Train the original neural renderer according to the multiple point-based scene representations of specific scenes to obtain a target neural renderer.

[0046] In a possible implementation manner, the mask processing module is further configured to:

[0047] Generate a segmentation mask and a log probability map according to the target neural renderer;

[0048] Construct a real-time mapping function;

[0049] Generate a three-dimensional neural point cloud segmentation mask according to the real-time mapping function, the segmentation mask, and the log probability map.

[0050] In a possible implementation manner, the mask processing module is further configured to:

[0051] Determine a neural mask region growth and pruning strategy according to the spatial Euclidean distance and the feature cosine distance between neural point clouds;

[0052] Optimize the three-dimensional neural point cloud segmentation mask according to the neural mask region growth and pruning strategy to obtain an optimized three-dimensional neural point cloud segmentation mask;

[0053] Correspondingly, the editing module is further configured to:

[0054] Generate an edited image according to the optimized three-dimensional neural point cloud segmentation mask in response to a user's editing request.

[0055] In a possible implementation manner, the editing request includes a click operation of the user on the image to be processed. Correspondingly, the editing module is further configured to:

[0056] In response to the click operation, determine the neural point cloud of the user on the image to be processed according to the three-dimensional neural point cloud segmentation mask, where the neural point cloud is used to select a target element;

[0057] Perform a fine editing operation on the target element according to the click operation to obtain a finely edited image.

[0058] In a possible implementation manner, the editing module is further configured to:

[0059] In response to a click operation, determine a synthesis target for different specific scenarios according to the three-dimensional neural point cloud segmentation mask;

[0060] Perform a synthesis operation on the synthesis target according to the click operation to obtain a cross-scene synthesis image.

[0061] In a possible implementation manner, the feature processing module is further configured to:

[0062] Generate a neural embedding function according to the radiation rotation invariance constraint;

[0063] Based on the neural embedding function, establish a rotation-invariant neural inverse distance weighted interpolation module;

[0064] Embed the rotation-invariant neural inverse distance weighted interpolation module into the original neural radiance field optimization model to obtain a preset neural radiance field optimization model.

[0065] In a third aspect, the present application provides an image editing device based on a neural radiance field, including: a memory, a processor;

[0066] The memory stores computer execution instructions;

[0067] The processor executes the computer execution instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementation manners of the first aspect.

[0068] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer execution instructions are stored, and when the computer execution instructions are executed by a processor, they are used to implement the above first aspect and / or various possible implementation manners of the first aspect.

[0069] In a fifth aspect, the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the above first aspect and / or various possible implementation manners of the first aspect.

[0070] A method, device, equipment, and storage medium for image editing based on neural radiance fields. On the one hand, by sampling the image to be processed, key pixel information in the image is extracted, providing high-quality input data for subsequent optimization of the neural radiance field, ensuring the accuracy of editing and the retention of details. On the other hand, by adding a radiation rotation invariance constraint, the optimization model can better process images under different perspectives and lighting conditions, enhancing the robustness and consistency of feature extraction, providing stable feature information for subsequent editing. At the same time, according to the feature information corresponding to the sampling points, a target neural renderer is generated. The generated target neural renderer can accurately represent the geometry and appearance information of the three-dimensional scene, providing a high-quality rendering basis for subsequent three-dimensional point cloud segmentation and editing. In addition, by generating a three-dimensional neural point cloud segmentation mask, different objects or components in the scene can be accurately identified and separated, providing a clear target area for fine editing, enhancing the controllability and accuracy of editing. Finally, according to the user's instructions, combined with the three-dimensional neural point cloud segmentation mask, fine editing can be quickly and efficiently achieved, and a high-quality edited image can be generated without retraining the neural radiance field, significantly improving the editing efficiency and interaction experience, thereby achieving the technical effect of improving the image editing efficiency and editing controllability based on neural radiance fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The accompanying drawings herein are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0072] Figure 1 Schematic diagram of an application data processing system architecture provided by an embodiment of the present application;

[0073] Figure 2 Flow chart of the method for image editing based on neural radiance fields provided by an embodiment of the present application Figure 1 ;

[0074] Figure 3 Frame diagram of the method for image editing based on neural radiance fields provided by an embodiment of the present application

[0075] Figure 4 Flow chart of the method for image editing based on neural radiance fields provided by an embodiment of the present application Figure 2 ;

[0076] Figure 5 Flow chart of the method for image editing based on neural radiance fields provided by an embodiment of the present application Figure 3 ;

[0077] Figure 6 Flow chart of the method for image editing based on neural radiance fields provided by an embodiment of the present application Figure 4 ;

[0078] Figure 7 Schematic flow chart of the image editing method based on neural radiance fields provided by an embodiment of this application Figure 5 ;

[0079] Figure 8 Schematic illustration of the effect of the image editing method based on neural radiance fields provided by an embodiment of this application Figure 1 ;

[0080] Figure 9 Schematic illustration of the effect of the image editing method based on neural radiance fields provided by an embodiment of this application Figure 2 ;

[0081] Figure 10 Schematic illustration of the effect of the image editing method based on neural radiance fields provided by an embodiment of this application Figure 3 ;

[0082] Figure 11 Schematic structural diagram of the image editing device based on neural radiance fields provided by an embodiment of this application;

[0083] Figure 12 Schematic structural diagram of the image editing device based on neural radiance fields provided by an embodiment of this application

[0084] Through the above-mentioned drawings, specific embodiments of this application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of this application in any way, but to illustrate the concept of this application to those skilled in the art by referring to specific embodiments. Detailed implementation manners

[0085] Here, exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with this application. On the contrary, they are merely examples of devices and methods consistent with some aspects of this application as detailed in the appended claims.

[0086] Since in the prior art, there is a lack of efficient fine-grained editing capabilities, it is difficult to achieve fast and controllable interactive editing while maintaining high-quality rendering, and there are technical problems such as low efficiency and poor controllability in image editing based on neural radiance fields.

[0087] In view of the above problems, an image editing method, device, equipment and storage medium based on neural radiance fields provided by the present application, on the one hand, samples the image to be processed to extract key pixel information in the image, providing high-quality input data for subsequent optimization of the neural radiance field to ensure the accuracy of editing and retention of details. On the other hand, by adding a radiation rotation invariance constraint, the optimization model can better process images under different viewpoints and lighting conditions, improving the robustness and consistency of feature extraction, providing stable feature information for subsequent editing. At the same time, according to the feature information corresponding to the sampling points, a target neural renderer is generated, and the generated target neural renderer can accurately express the geometric and appearance information of the three-dimensional scene, providing a high-quality rendering basis for subsequent three-dimensional point cloud segmentation and editing. In addition, by generating a three-dimensional neural point cloud segmentation mask, different objects or components in the scene can be accurately identified and separated, providing a clear target area for fine editing, improving the controllability and accuracy of editing. Finally, according to the user instruction, combined with the three-dimensional neural point cloud segmentation mask, fine editing can be quickly and efficiently achieved, and a high-quality edited image can be generated without retraining the neural radiance field, significantly improving the editing efficiency and interactive experience, thereby achieving the technical effects of improving the image editing efficiency and editing controllability based on neural radiance fields.

[0088] The following will specifically describe the technical solutions of the present application and how the technical solutions of the present application solve the above technical problems with specific embodiments. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the drawings.

[0089] Figure 1 FIG. is a schematic diagram of an application data processing system architecture provided by an embodiment of the present application, and the application data processing system is a computer device. As Figure 1 shown, the above architecture includes at least one of a data acquisition device 101, a processing device 102, and a display device 103.

[0090] It can be understood that the structure schematically shown in the embodiments of the present application does not constitute a specific limitation on the architecture of the application data processing system. In other feasible embodiments of the present application, the above architecture may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or have different component arrangements, which can be specifically determined according to the actual application scenario and will not be limited here. Figure 1 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0091] In the specific implementation process, the data acquisition device 101 may include an input / output interface or a communication interface, and the data acquisition device 101 can be connected to the processing device through the input / output interface or the communication interface.

[0092] The processing device 102 can obtain multiple images to be processed, and perform sampling processing on the images to be processed to obtain pixel information of multiple sampling points; input the pixel information of the sampling points into a preset neural radiance field optimization model to output feature information corresponding to the sampling points through the preset neural radiance field optimization model, wherein a radiance rotation invariance constraint is added to the preset neural radiance field optimization model; generate a target neural renderer according to the feature information corresponding to the sampling points; generate a three-dimensional neural point cloud segmentation mask according to the target neural renderer; in response to a user's editing request, generate an edited image according to the three-dimensional neural point cloud segmentation mask.

[0093] The display device 103 can also be a touch display screen or the screen of a terminal device, and is used to receive user instructions while displaying the above content to implement interaction with the user.

[0094] It should be understood that the above processing device can be implemented by a processor reading and executing instructions in a memory, or can be implemented by a chip circuit.

[0095] In addition, the network architecture and service scenarios described in the embodiments of the present application are for more clearly explaining the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those of ordinary skill in the art know that with the evolution of the network architecture and the emergence of new service scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0096] Figure 2 Flow schematic of the image editing method based on neural radiance field provided by the embodiments of the present application Figure 1 As Figure 2 shown, the image editing method based on neural radiance field provided in this embodiment includes:

[0097] S201. Obtain multiple images to be processed, and perform sampling processing on the images to be processed to obtain pixel information of multiple sampling points;

[0098] According to the requirements of image editing, capture images of the scene to be processed from multiple perspectives, obtain multiple images to be processed, perform sampling processing on each image, and obtain pixel information of corresponding multiple sampling points.

[0099] S202. Input the pixel information of the sampling points into a preset neural radiance field optimization model to output feature information corresponding to the sampling points through the preset neural radiance field optimization model, wherein a radiance rotation invariance constraint is added to the preset neural radiance field optimization model;

[0100] In this embodiment, the preset neural radiance field optimization model refers to a neural network model, which can be obtained through deep learning. The training method of this model is not particularly limited in this embodiment.

[0101] Input the pixel information of each obtained sampling point into the preset neural radiance field optimization model to obtain the feature information corresponding to each sampling point output by the model.

[0102] S203. Generate a target neural renderer according to the feature information corresponding to the sampling points;

[0103] Generate a target neural renderer by using the feature information corresponding to the sampling points.

[0104] S204. Generate a three-dimensional neural point cloud segmentation mask according to the target neural renderer;

[0105] Render the scene by using the target neural renderer to generate a three-dimensional neural point cloud segmentation mask for each sampling point.

[0106] S205. In response to the user's editing request, generate an edited image according to the three-dimensional neural point cloud segmentation mask.

[0107] Receive the user's editing request, determine the target area selected by the user according to the three-dimensional neural point cloud segmentation mask, perform an editing operation on the target area, and use the target neural renderer to generate an edited image in real time.

[0108] Specifically, Figure 3 is a schematic framework diagram of the image editing method based on neural radiance field provided by the embodiment of the present application. As Figure 3 shown, this framework utilizes the complementary advantages of implicit NeRF representation and point-based representation.

[0109] First, a rotation-invariant point-based implicit representation is introduced in the neural point cloud scene expression module to enhance the rendering quality after fine-editing NeRF. Represent the scene with neural points, and optimize the embedding of each neural point by using the gradient flow generated by the loss between the rendered image and the ground truth training image;

[0110] Secondly, in the neural point cloud projection module, a radiance rotation invariant (RRI) constraint is proposed to ensure the preservation of view-related features during translation or rotation. To achieve this constraint, a rotation-invariant neural inverse distance weighted interpolation module is designed to replace the Cartesian coordinates as the network input and effectively aggregate the neural points;

[0111] In addition, the traditional NeRF scene representation is decomposed into a scene-independent neural renderer and several scene-specific implicit point fields to improve the efficiency of cross-scene synthesis. At the same time, a cross-scene renderer and several neural point fields are also learned, and the local scene content is encoded through learnable embeddings;

[0112] After constructing the neural point field for each scene, the user can finely and efficiently edit the appearance and disappearance of the nerves in the implicit field by manipulating the corresponding implicit point framework without retraining;

[0113] Based on click-based user prompts, fine-grained implicit field editing is performed on the 2D images of multi-view rendering, and an off-the-shelf zero-shot 2D segmentation model (SAM) is used to lift the multi-view inconsistent 2D masks to a consistent 3D neural point field for real-time editing to generate the corresponding views.

[0114] It should be noted that Figure 3 This is only for reference in showing the effects and does not represent the improvement points, nor does it affect the protection scope of the embodiments of this application.

[0115] The image editing method based on neural radiance fields provided by the embodiments of this application, on the one hand, samples the image to be processed to extract the key pixel information in the image, providing high-quality input data for the subsequent optimization of the neural radiance field to ensure the accuracy of editing and the retention of details. On the other hand, by adding a radiance rotation invariant constraint, the optimized model can better process images under different viewpoints and lighting conditions, improving the robustness and consistency of feature extraction, providing stable feature information for subsequent editing. At the same time, according to the feature information corresponding to the sampling points, a target neural renderer is generated, and the generated target neural renderer can accurately represent the geometric and appearance information of the three-dimensional scene, providing a high-quality rendering basis for subsequent three-dimensional point cloud segmentation and editing. In addition, by generating a three-dimensional neural point cloud segmentation mask, different objects or components in the scene can be accurately identified and separated, providing a clear target area for fine editing, improving the controllability and accuracy of editing. Finally, according to the user's instructions, combined with the three-dimensional neural point cloud segmentation mask, fine editing can be quickly and efficiently achieved, and a high-quality edited image can be generated without retraining the neural radiance field, significantly improving the editing efficiency and interactive experience, thus achieving the technical effect of improving the efficiency and controllability of image editing based on neural radiance fields.

[0116] Figure 4 It is a flow schematic of the image editing method based on neural radiance fields provided by the embodiments of this application Figure 2 , as Figure 4 shown, based on the above embodiments, the generation process of the target neural renderer is described in detail, including:

[0117] S401. Determine point-based scene representations for multiple specific scenarios based on the feature information corresponding to the sampling points;

[0118] Decompose the scene into multiple point-based representation units using the feature information corresponding to the sampling points to form point-based scene representations for multiple specific scenarios.

[0119] S402. Obtain the original neural renderer;

[0120] In this embodiment, the original neural renderer refers to a neural network model, which can be obtained through deep learning. The training method of this model is not particularly limited in this embodiment.

[0121] S403. Perform training processing on the original neural renderer according to the point-based scene representations for multiple specific scenarios to obtain the target neural renderer.

[0122] Use the point-based scene representations for multiple specific scenarios as inputs and input them into the original neural renderer for training processing to obtain the trained target neural renderer.

[0123] Specifically, decompose the neural radiance field into a scene-independent neural renderer and several point-based scene representations for specific scenarios.

[0124] Represent the scene with neural points and optimize the embedding of each neural point using the gradient flow generated by the loss between the rendered image and the ground truth training image.

[0125] The rotation-invariant neural inverse distance weighted interpolation module can share weights across scenes and learn a scene-independent neural point renderer. During the training phase, randomly select a scene and an image as a batch, and minimize the squared error between the rendered color and the real color C(r) through the following formula, and repeat the optimization process until the cross-scene neural renderer and the specific-scene neural points converge:

[0126]

[0127] where, is the rendered color and C(r) is the real color.

[0128] The image editing method based on neural radiance fields provided by the embodiments of the present application reduces the computational complexity through a scene representation method based on sampling points, while retaining the detailed information of the scene. At the same time, by fine-tuning the original neural renderer, the target neural renderer can generate more realistic and refined rendering results. In addition, the target neural renderer supports independent operations on local regions of the scene, providing strong technical support for subsequent fine editing. Finally, by combining regularization methods such as radiance rotation invariance constraints, the target neural renderer can adapt to different viewpoints and lighting conditions, improving the stability and consistency of rendering, and achieving the technical effects of improving the efficiency and controllability of image editing based on neural radiance fields.

[0129] Figure 5 is a schematic flowchart of the image editing method based on neural radiance fields provided by the embodiments of the present application Figure 3 , as Figure 5 shown, on the basis of the above embodiments, this embodiment details the generation process of the 3D neural point cloud segmentation mask and the generation process of the edited image, including:

[0130] S501. Generate a segmentation mask and a log probability map according to the target neural renderer;

[0131] In this embodiment, the segmentation mask is used to identify the regions of different objects or components in the scene, and the log probability map represents the probability that each pixel belongs to a specific category.

[0132] Use the target neural renderer to render the scene to generate a segmentation mask and a log probability map for each pixel.

[0133] Specifically, given multi-view rendered images:

[0134]

[0135] The user provides click-based hints pro i for each image I i to generate a segmentation mask and a log probability map:

[0136] ,

[0137] where M is the segmentation mask and L is the log probability map.

[0138] S502. Construct a real-time mapping function;

[0139] Construct a real-time mapping function to map user interactions in the two-dimensional image space to the three-dimensional neural point cloud space.

[0140] Specifically, the inconsistent N segmentation results on the two-dimensional image are promoted to a globally consistent three-dimensional neural point cloud segmentation mask, and the following real-time mapping function is constructed:

[0141]

[0142] where is the corresponding image segmentation result, and T i is the corresponding camera pose parameter.

[0143] S503. Generate a three-dimensional neural point cloud segmentation mask according to the real-time mapping function, the segmentation mask, and the log probability map;

[0144] According to the real-time mapping function, determine the three-dimensional point cloud region corresponding to the user interaction operation, and then use the segmentation mask and the log probability map to further refine the target region to generate an initial three-dimensional neural point cloud segmentation mask.

[0145] Specifically, with the above f lift , a three-dimensional neural point cloud segmentation mask PM can be obtained for the neural point cloud P.

[0146] For each image, first estimate the depth of the neural point field:

[0147]

[0148] Then, project the neural point cloud onto this view:

[0149]

[0150] where is the rotated camera pose of the image I i , and K is the camera internal parameter. For each neural point cloud p with a position at will be projected onto the [u, v] coordinates in the pixel space with a depth of z p .

[0151] S504. Determine the neural mask region growth and pruning strategy according to the spatial Euclidean distance and the feature cosine distance between neural point clouds;

[0152] Determine the geometric proximity relationship between point clouds through the spatial Euclidean distance, and then evaluate the appearance similarity between point clouds through the feature cosine distance. Combine the two to determine the neural mask region growth and pruning strategy.

[0153] Specifically, for each neural point cloud p, first define a Euclidean distance weight to measure the distance between this neural point and the estimated object surface, that is:

[0154]

[0155] Among them, is the depth of the corresponding pixel, which is a hyperparameter.

[0156] To assign higher weights to points close to the surface and penalize points far from the surface, a normal distribution is used instead of simply using the Euclidean distance.

[0157] It should be noted that when selecting neural points, these distance weights must be considered, rather than directly selecting the point closest to the camera like in the Z-buffer. Because in the neural point field, the interpolation of multiple neural point features encodes the density and radiance of the given position space. Points around the surface may all contribute to the calculation of density and radiance, rather than relying only on the point closest to the camera or the point closest to the surface. Another reason is that the point cloud scaffold is usually messy in the neural point domain and cannot represent the exact position of the object surface like explicit point cloud reconstruction methods. After obtaining the correspondence between pixels and the point cloud, the results of SAM segmentation can be back-projected onto the neural point cloud.

[0158] In the method provided in the embodiments of the present application, the pseudo-projected log probability map is used because it is closely related to the weight size of the segmentation result. After the user makes click-based prompts on N images, the weights from multiple views are fused to obtain the final 3D neural point mask.

[0159] Since complex scenes usually have complex occlusion relationships, the segmentation results from multiple views often have little overlap. For this reason, this embodiment proposes a simple and effective set learning method to fuse the segmentation weights from different views, that is:

[0160]

[0161] Among them, is the indicator function, and b and τ are hyperparameters.

[0162] In addition, a neural mask region growth and pruning strategy is proposed based on the spatial Euclidean distance and feature cosine distance between point clouds to further improve the mask result:

[0163] A KD tree is pre-constructed. During mask growth, the neighboring points around all mask points are searched, and the cosine distance of the features is calculated, that is:

[0164]

[0165] Among them, is the r-th stage of iterative filtering, and IND is the indexing function.

[0166] If the neural point p j has the feature l jIf it is similar enough to K neighboring nerve points in the surrounding neighborhood, it will be masked. To prune the mask, search within all masked points. If a point is too far from other points, it is regarded as an outlier, that is:

[0167]

[0168] If a masked nerve point p j at position x j is far from the nearest masked point p i it is unmasked.

[0169] S505. Optimize the 3D neural point cloud segmentation mask according to the neural mask region growth and pruning strategy to obtain the optimized 3D neural point cloud segmentation mask;

[0170] According to the region growth strategy, expand the segmentation mask to cover more relevant point clouds, and according to the pruning strategy, remove the noise or irrelevant point clouds in the segmentation mask to generate the optimized 3D neural point cloud segmentation mask.

[0171] S506. In response to the user's editing request, generate an edited image according to the optimized 3D neural point cloud segmentation mask.

[0172] In this embodiment, the editing request includes the user's click operation on the image to be processed.

[0173] According to the user's editing request, combined with the optimized 3D neural point cloud segmentation mask, an edited image is generated.

[0174] The image editing method based on neural radiance fields provided in the embodiments of the present application, on the one hand, can accurately identify different objects or components in the scene by generating a segmentation mask and a log probability map. On the other hand, the real-time mapping function ensures the spatial consistency between the 2D segmentation result and the 3D point cloud, avoiding the accumulation of segmentation errors. At the same time, the real-time mapping function can quickly propagate the 2D editing instructions to the 3D space, improving the efficiency and interactivity of editing. In addition, the region growth and pruning strategy combining geometric and appearance features further optimizes the segmentation result, improving the accuracy and robustness of segmentation. Finally, the optimized 3D neural point cloud segmentation mask provides a clear and accurate target area for subsequent fine editing, significantly improving the controllability and practicality of editing. And in response to the user's editing request, an edited image is generated according to the optimized 3D neural point cloud segmentation mask, enhancing the interactivity with the user and improving the user experience, achieving the technical effect of improving the efficiency and controllability of image editing based on neural radiance fields.

[0175] Figure 6 It is a flow schematic of the image editing method based on neural radiance fields provided in the embodiments of the present applicationFigure 4 , as Figure 6 shown, based on the above embodiments, this embodiment further describes the generation process of the edited image, including:

[0176] S601. In response to a click operation, determine the user's neural point cloud on the image to be processed according to the three-dimensional neural point cloud segmentation mask, where the neural point cloud is used to select the target element;

[0177] When the user performs a click operation on the image to be processed, according to the three-dimensional neural point cloud segmentation mask, quickly locate and extract the neural point cloud related to the click position, and these neural point clouds represent the target elements selected by the user.

[0178] S602. Perform a fine editing operation on the target element according to the click operation to obtain a finely edited image;

[0179] In this embodiment, the fine editing includes but is not limited to partial transformation, deletion, copying, etc.

[0180] After the target element is selected, the system adjusts the geometric attributes of the target element, such as position, size, shape, etc., according to the user's click operation, and then uses the target neural renderer to generate the edited image in real time.

[0181] Specifically, for fine editing, through the selected neural points:

[0182]

[0183] Partially delete, partially copy, or partially change their directions, positions, and features:

[0184]

[0185] Among them, the rotation matrix is represented by R, and the translation vector is represented by t.

[0186] S603. In response to a click operation, determine the user's synthesis target for different specific scenarios according to the three-dimensional neural point cloud segmentation mask;

[0187] In this embodiment, the synthesis target can come from different scenarios for cross-scenario synthesis.

[0188] When the user performs a click operation in different scenarios, the system extracts the synthesis target selected by the user according to the three-dimensional neural point cloud segmentation mask.

[0189] S604. Perform a synthesis operation on the synthesis target according to the click operation to obtain a cross-scenario synthesis image.

[0190] After extracting the synthesis target, the system adjusts the position, size, and pose of the synthesis target according to the user's click operation to match it with the new scene, and then uses the target neural renderer to generate the synthesized image.

[0191] Specifically, for cross-scene synthesis, the corresponding points are enlarged or reduced to construct the scene to be edited. By directly operating and synthesizing the corresponding point scaffolds from different scenes, cross-scene synthesis can be achieved.

[0192] The image editing method based on neural radiance fields provided by the embodiments of this application, on the one hand, through the three-dimensional neural point cloud segmentation mask, can accurately identify and extract the target elements selected by the user, ensuring the accuracy of editing and synthesis, and supporting fine adjustment of the geometry and appearance of the target elements, meeting the high requirements of users for details. On the other hand, it can synthesize the target elements in different scenes into the same scene, expanding the flexibility and application scope of editing. At the same time, the edited and synthesized images maintain high-quality rendering effects, ensuring visual consistency. In addition, the system can quickly respond to user operations, improving the interaction efficiency and user experience, and achieving the technical effect of improving the efficiency and controllability of image editing based on neural radiance fields.

[0193] Figure 7 It is a flow schematic of the image editing method based on neural radiance fields provided by the embodiments of this application Figure 5 , as Figure 7 shown, on the basis of the above embodiments, this embodiment further explains the acquisition process of the preset neural radiance field optimization model, including:

[0194] S701. Generate a neural embedding function according to the radiance rotation invariant constraint;

[0195] In this embodiment, the radiance rotation invariant constraint (RPI) means that under different viewpoints and rotation conditions, the radiance characteristics (such as color, illumination, etc.) of the scene remain consistent.

[0196] Based on the radiance rotation invariant constraint, design and generate the corresponding neural embedding function.

[0197] Specifically, to enhance the rendering quality after editing NeRF, the radiance rotation invariant constraint is introduced. Mathematically, RRI can be expressed as:

[0198]

[0199] Among them, R belongs to the Lie group, representing any rigid transformation, is the implicit function, used to learn the rotation invariant features that conform to RRI, f i represents the feature S is the feature, which is an implicit function, used to learn the rotation invariant features that conform to RRI.

[0200] With the RRI constraint, the solution space of NeRF is reduced, ensuring that the radiation field can produce realistic rendering effects after fine editing;

[0201] Further use

[0202]

[0203] where P represents the neural points in the scene, and each neural point i is located at the three-dimensional spatial position p i Learn the local feature l i .

[0204] During the training phase, the neural points will be sampled along the rays, and the l of each neural point i i will be optimized to represent the local content of the scene.

[0205] is an attribute designed for RRI, initialized as three mutually orthogonal unit normal vectors, and remains unchanged during the training process. When editing the radiation field, the neural points and their corresponding attributes a will be rotated simultaneously i . .

[0206] For a query point with the view direction d, use to represent the view direction.

[0207] Given an arbitrary rotation matrix R, there is

[0208] Since , so ;

[0209] Then, further construct a non-linear equation:

[0210]

[0211] Since 's orthogonality, the above matrix has a rank of 3, so d has a unique solution. For a specific neural point, the view direction can have a bijective relationship with these four variables. When the rotation angle of the neural point is the same as the view direction, these four variables remain unchanged.

[0212] S702. Establish a rotation-invariant neural inverse distance weighted interpolation module based on the neural embedding function;

[0213] Specifically, generate corresponding query points based on the above neural points, and render each pixel by querying and sampling N points along a ray .

[0214] Given a specific query point , search for the N nearest neighboring neural points, and according to the idea of inverse distance weighted interpolation, use a neural network to aggregate the neural points, namely the rotation-invariant neural inverse distance weighted interpolation (RNIDWI) module:

[0215] .

[0216] S703. Embed the rotation-invariant neural inverse distance weighted interpolation module into the original neural radiance field optimization model to obtain a preset neural radiance field optimization model.

[0217] Specifically, in order to enable each neural point to generate features related to the query point, a neural embedding related to the relative position between the neural point and the query point is predicted for each neural point:

[0218]

[0219]

[0220] Among them, is the position encoding (PE) function. During the editing process, while rotating the local neural point cloud, the direction of each neural point itself is also rotated. Compared with directly using the absolute direction in the Cartesian coordinate system, using this relative coordinate can ensure that the input direction features remain unchanged, thus meeting the RRI constraint conditions;

[0221] On the basis of , use two other multi-layer perceptrons (MLPs) to further encode the density and radiance features of the neural points:

[0222]

[0223] Then, perform inverse distance weighted interpolation of l i d and l i r for the neural points around the query point, and input them into the density decoder and the radiance decoder respectively. Finally, obtain the radiance and density of the query point:

[0224]

[0225]

[0226] Finally, use traditional volume rendering technology to implement the final pixel color.

[0227] The image editing method based on neural radiance fields provided by the embodiments of this application, on the one hand, through radiance rotation invariant constraints and neural embedding functions, the model can process scenes under different viewpoints and rotation conditions, improving the robustness of rendering. On the other hand, the neural embedding function and interpolation module can capture the complex geometry and appearance information of the scene, providing support for high-quality rendering. At the same time, the rotation-invariant neural inverse distance weighted interpolation module can efficiently calculate the interpolation results, improving the running efficiency of the model. And after embedding the interpolation module, the preset neural radiance field optimization model can generate higher-quality rendering results, retaining the details and consistency of the scene. In addition, the joint training process is efficient and stable, ensuring the fast convergence and optimization of the model, achieving the technical effects of improving the efficiency and controllability of image editing based on neural radiance fields.

[0228] Figure 8 Schematic diagram of the effect of the image editing method based on neural radiance fields provided by the embodiments of this application Figure 1 , such as Figure 8 shown, the result of fine and efficient editing of the NeRF synthetic dataset. Edit the neural radiance field by clicking on the prompts in the 2D image, instantly lift it to 3D neural points, and operate the corresponding neural point cloud without retraining.

[0229] In the first case (zooming in), select a hot dog in the scene and zoom it in; in the second case (deleting), a part of the potted plant is deleted, select some branches in the scene and delete them; the third case (transforming) demonstrates the part transformation ability, transform a drum and a chair in the scene; in the fourth case (duplicating), its duplication function is shown, after selecting the boat in the scene, it is duplicated twice on both sides; in the last case (selecting), select and rotate the main part of the microphone in the microphone scene.

[0230] Figure 9 Schematic diagram of the effect of the image editing method based on neural radiance fields provided by the embodiments of this application Figure 2 , such as Figure 9 shown, showing the results of fine-grained editing on the ScanNet dataset. Different from previous work, no instance segmentation annotations of the scene are required, and users can interactively edit parts of the object through prompts. In the first case (Scene-038), one of the chairs is selected and deleted. In addition, the desk in the center of the scene is selected, then duplicated, and repositioned at the position of the deleted chair. In the second case (Scene-192), one of the sofas is selected and transformed.

[0231] Figure 10Schematic diagram of the effect of the image editing method based on neural radiance fields provided by the embodiments of the present application Figure 3 , as Figure 10 shown, demonstrating fine and efficient editing and synthesis results.

[0232] It should be noted that the generation of new scenes does not require retraining. Instead, only the point scaffolds of each learned scene need to be transformed, scaled, locally edited, and synthesized. Multiple objects were synthesized from various scenes in the NeRF synthetic dataset and the ScanNet dataset.

[0233] The first row shows five different scenes with various forms. Then these imageries are transformed and combined at appropriate positions to create a unique implicit scene.

[0234] The second row shows the editing and synthesis results of multiple scenes. The local details of each scene are finely edited and the scenes are synthesized.

[0235] Finally, in the third row, the possibility of synthesizing objects across different datasets is explored. Room 113 faces great challenges in reconstruction and editing due to sparse captured viewpoints, blurred images, and inaccurate camera poses. Despite these difficulties, Room 113 is synthesized with a chair in the NeRF synthetic dataset to demonstrate the fine-grained editing ability of the proposed framework and the flexibility and generality of the scene-independent neural point renderer.

[0236] It should be noted that Figures 8 - 10 it is only for reference in showing the effect, not an improvement point, and does not affect the protection scope of the embodiments of the present application.

[0237] Figure 11 Schematic diagram of the structure of the image editing device based on neural radiance fields provided by the embodiments of the present application. The device of this embodiment can be in the form of software and / or hardware. As Figure 11 shown, the image editing device 1100 based on neural radiance fields provided by the embodiments of the present application includes: an image processing module 1101, a feature processing module 1102, a rendering processing module 1103, a mask processing module 1104, and an editing module 1105:

[0238] The image processing module 1101 is configured to obtain a plurality of images to be processed and perform sampling processing on the images to be processed to obtain pixel information of a plurality of sampling points;

[0239] The feature processing module 1102 is configured to input the pixel information of the sampling points into a preset neural radiance field optimization model to output feature information corresponding to the sampling points through the preset neural radiance field optimization model, wherein a radiance rotation invariance constraint is added to the preset neural radiance field optimization model;

[0240] A rendering processing module 1103, configured to generate a target neural renderer according to the feature information corresponding to the sampling points;

[0241] A mask processing module 1104, configured to generate a three-dimensional neural point cloud segmentation mask according to the target neural renderer;

[0242] An editing module 1105, configured to generate an edited image according to the three-dimensional neural point cloud segmentation mask in response to a user's editing request.

[0243] In a possible implementation manner, the rendering processing module 1103 is further configured to:

[0244] Determine multiple point-based scene representations of specific scenes according to the feature information corresponding to the sampling points;

[0245] Obtain an original neural renderer;

[0246] Train the original neural renderer according to the multiple point-based scene representations of specific scenes to obtain a target neural renderer.

[0247] In a possible implementation manner, the mask processing module 1104 is further configured to:

[0248] Generate a segmentation mask and a log probability map according to the target neural renderer;

[0249] Construct a real-time mapping function;

[0250] Generate a three-dimensional neural point cloud segmentation mask according to the real-time mapping function, the segmentation mask, and the log probability map.

[0251] In a possible implementation manner, the mask processing module 1104 is further configured to:

[0252] Determine a neural mask region growth and pruning strategy according to the spatial Euclidean distance and the feature cosine distance between neural point clouds;

[0253] Optimize the three-dimensional neural point cloud segmentation mask according to the neural mask region growth and pruning strategy to obtain an optimized three-dimensional neural point cloud segmentation mask;

[0254] Correspondingly, the editing module 1105 is further configured to:

[0255] Generate an edited image according to the optimized three-dimensional neural point cloud segmentation mask in response to a user's editing request.

[0256] In a possible implementation manner, the editing request includes a click operation of the user on the image to be processed. Correspondingly, the editing module 1105 is further configured to:

[0257] In response to a click operation, determine the user's neural point cloud on the image to be processed according to the three-dimensional neural point cloud segmentation mask, where the neural point cloud is used to select the target element;

[0258] According to the click operation, perform a fine editing operation on the target element to obtain a fine-edited image.

[0259] In a possible implementation manner, the editing module 1105 is further configured to:

[0260] In response to a click operation, determine the synthesis target for different specific scenarios according to the three-dimensional neural point cloud segmentation mask;

[0261] According to the click operation, perform a synthesis operation on the synthesis target to obtain a cross-scenario synthesis image.

[0262] In a possible implementation manner, the feature processing module 1102 is further configured to:

[0263] Generate a neural embedding function according to the radiation rotation invariance constraint;

[0264] Based on the neural embedding function, establish a rotation-invariant neural inverse distance weighted interpolation module;

[0265] Embed the rotation-invariant neural inverse distance weighted interpolation module into the original neural radiance field optimization model to obtain a preset neural radiance field optimization model.

[0266] The image editing device based on the neural radiance field provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be elaborated here in this embodiment.

[0267] Figure 12 This is a schematic structural diagram of the image editing device based on the neural radiance field provided in the embodiment of the present application. As Figure 12 shown, the electronic device 1200 provided in this embodiment includes: at least one processor 1201 and a memory 1202. Optionally, the device 1200 further includes a communication component 1203. Among them, the processor 1201, the memory 1202, and the communication component 1203 are connected through a bus.

[0268] In a specific implementation process, at least one processor 1201 executes the computer execution instructions stored in the memory 1202, so that at least one processor 1201 executes the above method.

[0269] The specific implementation process of the processor 1201 can refer to the above method embodiment, and its implementation principle and technical effect are similar, which will not be elaborated here in this embodiment.

[0270] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU for short), or other general-purpose processors, digital signal processors (DSP for short), application specific integrated circuits (ASIC for short), etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0271] The memory may include a high-speed random access memory (RAM), and may also include non-volatile memory (NVM), such as at least one disk memory.

[0272] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, the buses in the drawings of this application are not limited to only one bus or one type of bus.

[0273] The embodiments of this application also provide a computer program product, including a computer program, which implements the above method when executed by a processor.

[0274] The embodiments of this application also provide a computer-readable storage medium, in which computer-executable instructions are stored, and when the processor executes the computer-executable instructions, the above method is implemented.

[0275] The above-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk or an optical disk. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0276] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an Application Specific Integrated Circuits (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in a device.

[0277] The division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Additionally, the couplings or direct couplings or communication connections shown or discussed between each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be in electrical, mechanical, or other forms.

[0278] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0279] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0280] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0281] Those of ordinary skill in the art can understand that all or part of the steps to implement the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the foregoing storage medium includes: various media such as ROM, RAM, magnetic disk or optical disc that can store program codes.

[0282] Finally, it should be noted that those skilled in the art will readily think of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include the common general knowledge or conventional technical means in the technical field not disclosed in the present invention. It is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. An image editing method based on neural radiation field, characterized in that: include: Acquire a plurality of images to be processed, and perform sampling processing on the images to be processed to obtain pixel information of a plurality of sampling points; Inputting the pixel information of the sampling point into a preset neural radiation field optimization model, so as to output the feature information corresponding to the sampling point through the preset neural radiation field optimization model, wherein a radiation rotation invariant constraint is added to the preset neural radiation field optimization model; Generate a target neural renderer according to the feature information corresponding to the sampling points; generating a three-dimensional neural point cloud segmentation mask according to the target neural renderer; In response to a user's editing request, an edited image is generated according to the three-dimensional neural point cloud segmentation mask.

2. The method according to claim 1, characterized in that The step of generating a target neural renderer according to the feature information corresponding to the sampling point comprises: Determining point-based scene representations of a plurality of specific scenes according to feature information corresponding to the sampling points; Get the original neural renderer; The original neural renderer is trained according to the point-based scene representations of the multiple specific scenes to obtain a target neural renderer.

3. The method according to claim 1, characterized in that The step of generating a three-dimensional neural point cloud segmentation mask according to the target neural renderer comprises: generating a segmentation mask and a log-probability map according to the target neural renderer; Build real-time mapping functions; A three-dimensional neural point cloud segmentation mask is generated according to the real-time mapping function, the segmentation mask and the logarithmic probability map.

4. The method according to claim 3, characterized in that After generating a three-dimensional neural point cloud segmentation mask according to the target neural renderer, the method further includes: Determine the neural mask region growing and pruning strategy based on the spatial Euclidean distance and feature cosine distance between neural point clouds; According to the neural mask region growth and pruning strategy, the three-dimensional neural point cloud segmentation mask is optimized to obtain an optimized three-dimensional neural point cloud segmentation mask; Accordingly, in response to the user's editing request, generating an edited image according to the three-dimensional neural point cloud segmentation mask includes: In response to a user's editing request, an edited image is generated according to the optimized three-dimensional neural point cloud segmentation mask.

5. The method according to claim 2 or 3, characterized in that: The editing request includes a click operation by a user on the image to be processed; Accordingly, in response to the user's editing request, generating an edited image according to the three-dimensional neural point cloud segmentation mask includes: In response to the click operation, determining a neural point cloud of the user on the image to be processed according to the three-dimensional neural point cloud segmentation mask, wherein the neural point cloud is used to select a target element; According to the click operation, a fine editing operation is performed on the target element to obtain a fine editing image.

6. The method according to claim 5, characterized in that The step of generating an edited image in response to a user's editing request according to the three-dimensional neural point cloud segmentation mask further comprises: In response to the click operation, determining a synthesis target of the user for different specific scenes according to the three-dimensional neural point cloud segmentation mask; According to the click operation, a synthesis operation is performed on the synthesis target to obtain a cross-scene synthetic image.

7. The method according to any one of claims 1 to 3, characterized in that: Before inputting the pixel information of the sampling point into a preset neural radiation field optimization model to output feature information corresponding to the sampling point through the preset neural radiation field optimization model, the method further includes: Generate a neural embedding function based on the radiation rotation invariance constraint; Based on the neural embedding function, a rotation-invariant neural inverse distance weighted interpolation module is established; The rotationally invariant neural inverse distance weighted interpolation module is embedded into the original neural radiation field optimization model to obtain a preset neural radiation field optimization model.

8. An image editing device based on neural radiation field, characterized in that: include: An image processing module, used for acquiring a plurality of images to be processed, and performing sampling processing on the images to be processed to obtain pixel information of a plurality of sampling points; A feature processing module, used for inputting the pixel information of the sampling point into a preset neural radiation field optimization model, so as to output feature information corresponding to the sampling point through the preset neural radiation field optimization model, wherein a radiation rotation invariant constraint is added to the preset neural radiation field optimization model; A rendering processing module, used for generating a target neural renderer according to the feature information corresponding to the sampling point; A mask processing module, used for generating a three-dimensional neural point cloud segmentation mask according to the target neural renderer; The editing module is used to generate an edited image according to the three-dimensional neural point cloud segmentation mask in response to a user's editing request.

9. An image editing device based on neural radiation field, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the image editing method based on neural radiation field according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the image editing method based on neural radiation field according to any one of claims 1 to 7.

Images