Method and device for searching and editing objects in NeRF

By receiving query requests and extracting features, the similarity of the feature vectors of NeRF 3D scene points is solved, and the problem of difficult objects in the NeRF model is realized, and the accurate identification and editing of objects in the NeRF model is achieved.

CN118916510APending Publication Date: 2024-11-08BEIJING QUALITY ENERGY CORE TECH CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410924604.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-10
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing NeRF models are difficult to find and edit specific objects in it, and the calculated results cannot be manually intervened or edited.

Method used

By receiving the query request, the features of the query content are extracted, the similarity to the feature vector of NeRF 3D scenes is calculated, the point feature vectors are edited and updated, and the 3D scene is finally rendered based on the updated feature vectors, supporting text and image queries.

Benefits of technology

Accurate identification and editing of objects in the NeRF model is realized, and relevant features can be queryed through text or image to further determine the identification object and achieve the editing purpose.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916510A_ABST
    Figure CN118916510A_ABST
Patent Text Reader

Abstract

The invention relates to a method and device for searching and editing an object in NeRF, and the method comprises the steps: receiving a query request, carrying out the feature extraction of the query content of the query request, obtaining a query feature, obtaining a point feature vector of each point in a 3D scene based on NeRF, calculating the similarity between the query feature and the point feature vector, and carrying out the searching and editing of the object in the NeRF. And determining a query result corresponding to the query content according to whether the similarity exceeds a preset similarity threshold, editing the query result, updating the corresponding point feature vector, and rendering the 3D scene of the NeRF according to the updated point feature vector. Through the method, features can be fused to obtain multi-modal features, the multi-modal features are accurately reflected in a 3D scene, related features can be queried through texts or images, recognition objects are further determined, and the purpose of editing objects and the like is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of neural network algorithm technology, and in particular to a method and device for searching and editing objects in NeRF. Background Art

[0002] Neural Radiance Field (NeRF) is a neural network algorithm that can reconstruct complex 3D scenes from a partial set of 2D images. Various simulations, games, media, and Internet of Things (IoT) applications require 3D images to make digital interactions more realistic and accurate. NeRF learns the scene geometry, objects, and angles of a specific scene, and then renders realistic 3D views from new perspectives, automatically generating synthetic data to fill in the gaps. It has attracted widespread attention in the field of computer vision as a new type of view synthesis technology with implicit scene representation. As a novel view synthesis and 3D reconstruction method, the NeRF model has a wide range of applications in robotics, city mapping, autonomous navigation, VR / AR, and other fields.

[0003] However, NeRF models are all connected as a whole, and all objects are integrated, so if you need to find and edit an object in the model and edit it, it is difficult to do so. Usually NeRF does not provide such functions, and NeRF does not have human intervention, so the calculated results cannot be edited. Summary of the invention

[0004] In view of this, the present application proposes a method for searching and editing objects in NeRF to solve the problems embodied in the above-mentioned background technology.

[0005] According to one aspect of the present application, a method for searching and editing objects in NeRF is provided, comprising:

[0006] receiving a query request;

[0007] Extracting features from the query content of the query request to obtain query features;

[0008] Get the point feature vector of each point in the NeRF-based 3D scene;

[0009] Calculating the similarity between the query feature and the point feature vector, and determining a query result corresponding to the query content according to whether the similarity exceeds a preset similarity threshold;

[0010] Editing the query result and updating the corresponding point feature vector;

[0011] The NeRF 3D scene is rendered according to the updated point feature vector.

[0012] As an optional implementation scheme of the present application, feature extraction is performed on the query content of the query request to obtain query features, including:

[0013] When the query content is a text query, a text encoder is used to extract language features to obtain language query features;

[0014] When the query content is an image query, a visual encoder is used to extract visual features to obtain visual query features.

[0015] As an optional implementation scheme of the present application, determining the query result corresponding to the query content according to whether the similarity exceeds a preset similarity threshold includes:

[0016] Determining whether the similarity exceeds a preset similarity threshold;

[0017] When the similarity exceeds the preset similarity threshold, a query result in the NeRF-based 3D scene is determined according to the point feature vector correspondence.

[0018] As an optional implementation scheme of the present application, the step of obtaining a point feature vector of each point in the NeRF-based 3D scene includes:

[0019] Obtain the 3D scene image to be constructed;

[0020] Respectively extracting visual features and language features of the 3D scene image to be constructed;

[0021] fusing the visual features and the language features to obtain multimodal features;

[0022] Performing feature matching training on the pre-trained NeRF model according to the 3D scene image to be constructed using the multimodal features to obtain a training result;

[0023] The pre-trained NeRF model generates a corresponding 3D scene according to the training result.

[0024] As an optional implementation scheme of the present application, respectively extracting visual features and language features of the 3D scene image to be constructed includes:

[0025] Using a visual encoder to extract visual features of the 3D scene image to be constructed;

[0026] Obtaining a descriptive label for the 3D scene image to be constructed;

[0027] Calculating pixel embedding of the 3D scene image to be constructed using an image encoder to obtain a pixel embedding calculation result;

[0028] The descriptive label and the pixel embedding calculation result are interacted to obtain the language features of the 3D scene image to be constructed.

[0029] As an optional implementation scheme of the present application, the fusion processing of the visual features and the language features to obtain multimodal features includes:

[0030] Combining the visual features with the language features and performing similarity measurement calculation to obtain a weight matrix;

[0031] Normalizing the weight matrix, and performing weighted calculation on the visual feature and the language feature to obtain a weighted result;

[0032] The weighted results are summed up to fuse the visual features and the language features to obtain multimodal features.

[0033] As an optional implementation scheme of the present application, the fusing of the visual features and the language features to obtain multimodal features further includes:

[0034] Construct a common feature space;

[0035] Applying linear transformation to the visual feature and the language feature in the common feature space respectively, and obtaining a linear transformation result of the visual feature and a linear change result of the language feature;

[0036] The linear transformation result of the visual feature and the linear change result of the language feature are fused to obtain a multimodal feature.

[0037] As an optional implementation scheme of the present application, the multimodal features are subjected to feature matching training for a pre-trained NeRF model according to the 3D scene image to be constructed to obtain a training result, including:

[0038] Get a predefined loss function;

[0039] Obtaining an objective function for the feature matching according to the loss function;

[0040] The pre-trained NeRF model performs feature matching training according to the objective function to obtain a training result.

[0041] The present application also provides a device for searching and editing objects in NeRF, including:

[0042] A query receiving module, used for receiving a query request;

[0043] A feature extraction module, used to extract features from the query content of the query request to obtain query features;

[0044] The point feature vector acquisition module is used to obtain the point feature vector of each point in the NeRF-based 3D scene;

[0045] A query result acquisition module is used to calculate the similarity between the query feature and the point feature vector, and determine the query result corresponding to the query content according to whether the similarity exceeds a preset similarity threshold;

[0046] An editing module, used for editing the query result and updating the corresponding point feature vector;

[0047] A rendering module is used to render the NeRF 3D scene according to the updated point feature vector.

[0048] The present application also provides an electronic device, including:

[0049] processor;

[0050] a memory for storing processor-executable instructions;

[0051] Wherein, the processor is configured to implement the above-mentioned method for searching and editing objects in NeRF when executing the executable instructions.

[0052] Beneficial effects of this application:

[0053] The present invention extracts features from the query content of the query request to obtain query features, obtains point feature vectors of each point in the 3D scene based on NeRF, calculates the similarity between the query features and the point feature vectors, and determines the query result corresponding to the query content according to whether the similarity exceeds a preset similarity threshold. It is possible to fuse visual features and language features to obtain multimodal features, and accurately reflect them in the 3D scene, and can further determine the identification object through text or image query related features to achieve the purpose of editing objects, etc.

[0054] Other features and aspects of the present application will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present application and, together with the description, serve to explain the principles of the present application.

[0056] Figure 1 A flowchart showing a method for searching and editing objects in NeRF according to an embodiment of the present application;

[0057] Figure 2 A block diagram of a device for searching and editing objects in NeRF according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0058] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0059] Among them, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application or simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0060] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0061] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0062] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0063] Multimodality refers to the use of two or more senses or information sources to interact with each other. In the field of artificial intelligence, multimodal technology involves the fusion of different types of data and information (such as images, audio, text, video, etc.) to achieve more accurate and efficient applications.

[0064] Example 1

[0065] Figure 1 FIG. 2 is a flowchart of a method for searching and editing objects in NeRF according to an embodiment of the present application. Figure 1 As shown, the flow chart includes:

[0066] S100, receiving a query request;

[0067] In this embodiment, when it is necessary to search and edit an object in the NeRF model, a query request is sent to the NeRF model through the content to be searched. After receiving the query request, the NeRF model further analyzes the query request to obtain a query result. It should be noted that the search object can be an object, a local area, etc.

[0068] S200, extracting features from the query content of the query request to obtain query features;

[0069] In this embodiment, the query request includes text query and image query. Different methods are used to extract query content features for different queries, and the query content is converted into a vector representation so that the obtained query features are more suitable for the objects to be searched and edited.

[0070] S300, obtaining a point feature vector of each point in the 3D scene based on NeRF;

[0071] Each point in the NeRF model is represented by a vector, including a three-dimensional spatial position vector and a three-dimensional vector representing the direction of sight. These vectors together form the basis for the NeRF model to represent and render scenes. Therefore, when searching for objects using query features, it is necessary to obtain the point feature vector of each point in the 3D scene that has been rendered by NeRF.

[0072] S400, calculating the similarity between the query feature and the point feature vector, and determining the query result corresponding to the query content according to whether the similarity exceeds a preset similarity threshold;

[0073] In this embodiment, the cosine similarity method is used to calculate the similarity between the query feature and the point feature vector, and a similarity threshold is set to determine which areas in the 3D scene are more matched with the query feature. It should be noted that the specific setting of the threshold is set by the user according to actual conditions and is not limited here.

[0074] S500, editing the query result and updating the corresponding point feature vector;

[0075] Areas are gradually excluded based on the threshold, and the query result is finally determined. Various editing operations are performed on the identified object corresponding to the query result, such as changing the color, moving the object, etc. After the identified object is edited, its attributes are modified, and the corresponding point feature vector will also change. Therefore, the point feature vector of the identified object needs to be updated to reflect the edited state.

[0076] S600: Render the NeRF 3D scene according to the updated point feature vector.

[0077] Apply the updated point feature vector and re-render the entire 3D scene to observe the effect of the edits.

[0078] As an optional implementation scheme of the present application, feature extraction is performed on the query content of the query request to obtain query features, including: when the query content is a text query, using a text encoder to extract language features to obtain language query features; when the query content is an image query, using a visual encoder to extract visual features to obtain visual query features.

[0079] When the query content is a text query, a pre-trained text encoder is used to extract the language features of the query content to obtain language query features. In this embodiment, a text encoder based on CLIP (Contrastive Language-Image Pre-training) combined with LSeg (Language-driven Semantic Segmentation) algorithm is used.

[0080] When the query content is an image query, a pre-trained visual encoder is used to extract visual features of the query content to obtain visual query features. In this embodiment, a visual encoder based on the DINO (self-distillation with nolabels) algorithm is used.

[0081] As an optional implementation scheme of the present application, the query result corresponding to the query content is determined according to whether the similarity exceeds a preset similarity threshold, including: judging whether the similarity exceeds a preset similarity threshold; when the similarity exceeds the preset similarity threshold, determining the query result in the NeRF-based 3D scene according to the point feature vector correspondence.

[0082] For example, after setting the threshold, when the query content is a local area, the density of the area that is less than the threshold, that is, does not meet the threshold, is set to 0, and the area is removed from the scene first, and the area is excluded, thereby narrowing the area where the query results corresponding to the query content are located. When the query content is an object, after determining the area where the object is located, the specific object is gradually determined by calculating the similarity of the objects in the area and comparing them with the similarity threshold.

[0083] As an optional implementation scheme of the present application, a point feature vector of each point in a NeRF-based 3D scene is obtained, including: obtaining an image of the 3D scene to be constructed; extracting visual features and language features of the image of the 3D scene to be constructed respectively; fusing the visual features and language features to obtain multimodal features; performing feature matching training on a pre-trained NeRF model based on the multimodal features and the image of the 3D scene to be constructed to obtain a training result; and the pre-trained NeRF model generates a corresponding 3D scene based on the training result.

[0084] When editing an object in NeRF, the object to be edited is determined based on the generated 3D scene. Prior to this, it is necessary to generate the 3D scene corresponding to the object. The data source for generating a 3D scene using the characteristics of the NeRF model is the source image data. In this embodiment, the source image data is the 3D scene image to be constructed. The visual model based on the DINO (self-distillation with no labels, label-free self-distillation) algorithm and the language model based on the LSeg (Language-driven Semantic Segmentation, language-driven semantic segmentation) algorithm are used as teacher models for multimodal feature distillation. In multimodal feature distillation, DINO is used to identify visual features and LSeg is used to identify language features. The purpose is to successfully transfer the 2D features extracted from the image to be processed to the 3D scene using the pre-trained NeRF model, ensure that the features learned by the student model, i.e., the pre-trained NeRF model, are consistent with the original 2D features, and the pre-trained NeRF model is continuously trained based on visual features and language features, thereby enhancing its understanding and editing capabilities in the 3D scene, and generating a 3D scene corresponding to the 3D scene image to be constructed.

[0085] As an optional implementation scheme of the present application, visual features and language features of the 3D scene image to be constructed are extracted separately, including: using a visual encoder to extract the visual features of the 3D scene image to be constructed; obtaining descriptive labels for the 3D scene image to be constructed; using an image encoder to calculate pixel embedding of the 3D scene image to be constructed to obtain pixel embedding calculation results; interacting the descriptive labels and the pixel embedding calculation results to obtain language features of the 3D scene image to be constructed.

[0086] The visual encoder corresponding to the visual model based on the DINO algorithm is used to extract visual features from the 3D scene image to be constructed, and the feature representation is adjusted through a series of learnable linear transformations to maximize the feature consistency of the same image under different data enhancements. These features can be directly used for tasks such as classification and segmentation, and good results can be achieved in a basic linear layer. DINO is a self-supervised learning model based on the Transformer architecture that can extract visual features from a large number of unlabeled images without the need for labeled data. This method uses the idea of ​​self-distillation to enable the two networks to supervise each other, thereby improving the robustness of the feature representation.

[0087] The text encoder corresponding to the language model based on the LSeg algorithm is used to calculate the embedding of the descriptive input label. For example, for the descriptive input labels of objects such as "grass" and "buildings" in the 3D scene image to be constructed, the text encoder converts the natural language label into a feature representation, that is, a text embedding. At the same time, the language model based on the LSeg algorithm uses a Transformer-based image encoder to calculate the dense embedding of each pixel of the 3D scene image to be constructed to obtain the pixel embedding. The training task of the image encoder is to make the pixel embedding consistent with the text embedding of the corresponding semantic category.

[0088] Furthermore, in the middle of the language model, a fusion layer (Deep Fusion) is used to further interact the per-pixel dense embedding calculated by the Transformer-based image encoder and the text embedding, so that the final image-text joint feature space can be trained better. In this way, the features of the image and the features of the text are trained to be stronger and more relevant, thereby improving the overall performance of the NeRF model as a student model.

[0089] You can also use DINO and LSeg together, first extracting high-quality visual features through DINO, and then combining these visual features with language descriptions through LSeg for semantic image segmentation or other multimodal tasks. Effectively using DINO and LSeg to extract visual features and language features and combining them can perform complex visual and language tasks.

[0090] As an optional implementation scheme of the present application, visual features and language features are fused to obtain multimodal features, including: combining visual features and language features, and performing similarity measurement calculations to obtain a weight matrix; normalizing the weight matrix, and performing weighted calculations with visual features and language features respectively to obtain weighted results; summing the weighted results, fusing visual features and language features to obtain multimodal features.

[0091] To further integrate visual features and language features, a weighted summation method can be used. Perform dot product calculations on the visual features and language features to measure similarity and obtain a weight matrix. Use the softmax function to normalize the weight matrix to obtain a normalized weight matrix. Perform a weighted summation of the normalized weight matrix, visual features, and language features. The summation result is the final fused multimodal feature.

[0092] As an optional implementation scheme of the present application, the visual features and the language features are fused to obtain multimodal features, which also includes: constructing a common feature space; applying linear transformations to the visual features and the language features in the common feature space respectively, to obtain the linear transformation results of the visual features and the linear change results of the language features respectively; and fusing the linear transformation results of the visual features and the linear change results of the language features to obtain multimodal features.

[0093] Linear transformation can also be used to integrate visual features and language features. Specifically, a common feature space is constructed, in which linear transformations are applied to visual features and language features respectively, and the linearly transformed visual features and language features are fused to generate a unified multimodal representation. The common feature space can be used to learn the shared representation of images and texts, and linear transformation can help adjust the importance and influence of different modal features.

[0094] Furthermore, the weighted summation and linear transformation methods can be combined. Specifically, the weighted summation method is used to quickly generate preliminary visual-language fusion features, and the linear transformation method is used to further optimize the preliminary visual-language fusion features to make them more coordinated and consistent in the common feature space.

[0095] As an optional implementation scheme of the present application, feature matching training is performed on a pre-trained NeRF model based on the multimodal features and the 3D scene image to be constructed to obtain a training result, including: obtaining a pre-defined loss function; obtaining a target function for feature matching based on the loss function; and the pre-trained NeRF model performs feature matching training based on the target function to obtain a training result.

[0096] In order to train the NeRF model and its system, ensure that the multimodal features are correctly reflected in the 3D scene, and establish connections between different modalities, the stochastic gradient descent (SGD) optimization algorithm is first used for optimization.

[0097] Then, a loss function is defined. In this step, a multi-task loss function is defined, which includes not only task-specific loss terms but also additional terms for establishing inter-modal connections. In this embodiment, the loss weights used for the pre-trained NeRF model involve loss terms including feature distillation loss, multi-modal similarity loss, and joint contrastive learning loss.

[0098] Feature Distillation Loss: This loss term encourages the 3D vision and language features to match those of the pre-trained NeRF model, thereby transferring the knowledge of the pre-trained NeRF model to the 3D scene representation.

[0099] Multimodal similarity loss: This loss is used to encourage visual features and language features to match each other, so that the features of the two modalities have consistent distribution and semantics.

[0100] Joint Contrastive Learning Loss: A contrastive learning approach enables different parts of the same object to be clustered together in the feature space while keeping parts of different objects separate. This helps establish clear boundaries and improves the accuracy of scene decomposition.

[0101] Combine all the above loss items to train the NeRF model, and get the training results of each loss item. Based on the training results, we can also get the training status between each task, and then balance the relationship between each task by adjusting the loss weight, so that we can finally get a 3D scene representation that can accurately reflect the multimodal features through the NeRF model, and generate a 3D scene representation corresponding to the 3D scene image to be constructed.

[0102] During the training process, it is necessary to continuously adjust and optimize in order to find the best model parameter configuration. In addition, regularization techniques and early stopping are used to prevent overfitting, thereby improving the generalization ability of the model.

[0103] Through the above method, visual features and language features can be fused to obtain multimodal features, which can be accurately reflected in the 3D scene. In addition, relevant features can be queried through text or images to further determine the recognized object and achieve the purpose of editing objects, etc.

[0104] Example 2

[0105] Based on the same principle as the above method, a device for searching and editing objects in NeRF is also proposed, see Figure 2 , a device 100 for searching and editing objects in NeRF according to an embodiment of the present disclosure includes:

[0106] A query receiving module 110, configured to receive a query request;

[0107] The feature extraction module 120 is used to extract features from the query content of the query request to obtain query features;

[0108] A point feature vector acquisition module 130 is used to acquire a point feature vector of each point in the 3D scene based on NeRF;

[0109] A query result acquisition module 140 is used to calculate the similarity between the query feature and the point feature vector, and determine the query result corresponding to the query content according to whether the similarity exceeds a preset similarity threshold;

[0110] An editing module 150, used to edit the query result and update the corresponding point feature vector;

[0111] The rendering module 160 is used to render the NeRF 3D scene according to the updated point feature vector.

[0112] Obviously, those skilled in the art should understand that all or part of the processes in the above-mentioned embodiment method can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned control methods. The modules or steps of the present invention can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Optionally, they can be implemented by program codes executable by computing devices, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules respectively, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the present invention is not limited to any specific combination of hardware and software.

[0113] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment method can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned control methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated as: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0114] Example 3

[0115] Furthermore, an electronic device is proposed, comprising:

[0116] processor;

[0117] a memory for storing processor-executable instructions;

[0118] The processor is configured to implement the method for searching and editing objects in NeRF of Embodiment 1 when executing executable instructions.

[0119] The electronic device of the embodiment of the present disclosure includes a processor and a memory for storing processor executable instructions, wherein the processor is configured to implement any of the above methods for searching and editing objects in NeRF when executing the executable instructions.

[0120] It should be noted that the number of processors may be one or more. At the same time, the electronic device of the embodiment of the present disclosure may also include an input device and an output device. The processor, memory, input device, and output device may be connected via a bus or in other ways, which are not specifically limited here.

[0121] The memory is a computer-readable storage medium for the method of searching and editing objects in NeRF, and can be used to store software programs, computer executable programs, and various modules, such as the program or module corresponding to the method of searching and editing objects in NeRF in the embodiment of the present disclosure. The processor executes various functional applications and data processing of the electronic device by running the software programs or modules stored in the memory.

[0122] The input device can be used to receive input numbers or signals. The signal can be a key signal related to user settings and function control of the device / terminal / server. The output device can include a display device such as a display screen.

[0123] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for searching and editing objects in NeRF, characterized in that include: receiving query requests; Extracting features from the query content of the query request to obtain query features; Get the point feature vector of each point in the NeRF-based 3D scene; Calculating the similarity between the query feature and the point feature vector, and determining a query result corresponding to the query content according to whether the similarity exceeds a preset similarity threshold; Editing the query result and updating the corresponding point feature vector; The NeRF 3D scene is rendered according to the updated point feature vector.

2. The method for searching and editing objects in NeRF according to claim 1, characterized in that: Extracting features from the query content of the query request to obtain query features includes: When the query content is a text query, a text encoder is used to extract language features to obtain language query features; When the query content is an image query, a visual encoder is used to extract visual features to obtain visual query features.

3. The method for searching and editing objects in NeRF according to claim 2, characterized in that: The determining, according to whether the similarity exceeds a preset similarity threshold, a query result corresponding to the query content includes: Determining whether the similarity exceeds a preset similarity threshold; When the similarity exceeds the preset similarity threshold, a query result in the NeRF-based 3D scene is determined according to the point feature vector correspondence.

4. The method for searching and editing objects in NeRF according to claim 1, characterized in that: The step of obtaining a point feature vector of each point in the NeRF-based 3D scene includes: Obtain the 3D scene image to be constructed; Respectively extracting visual features and language features of the 3D scene image to be constructed; fusing the visual features and the language features to obtain multimodal features; Performing feature matching training on the pre-trained NeRF model according to the 3D scene image to be constructed using the multimodal features to obtain a training result; The pre-trained NeRF model generates a corresponding 3D scene according to the training result.

5. The method for searching and editing objects in NeRF according to claim 4, characterized in that: The extracting of visual features and language features of the 3D scene image to be constructed respectively includes: Using a visual encoder to extract visual features of the 3D scene image to be constructed; Obtaining a descriptive label for the 3D scene image to be constructed; Calculating pixel embedding of the 3D scene image to be constructed using an image encoder to obtain a pixel embedding calculation result; The descriptive label and the pixel embedding calculation result are interacted to obtain the language features of the 3D scene image to be constructed.

6. The method for searching and editing objects in NeRF according to claim 4, characterized in that: The fusing of the visual features and the language features to obtain multimodal features includes: Combining the visual features with the language features and performing similarity measurement calculation to obtain a weight matrix; Normalizing the weight matrix, and performing weighted calculation on the visual feature and the language feature to obtain a weighted result; The weighted results are summed up to fuse the visual features and the language features to obtain multimodal features.

7. The method for searching and editing objects in NeRF according to claim 4, characterized in that: The fusing the visual features and the language features to obtain multimodal features further includes: Construct a common feature space; Applying linear transformation to the visual feature and the language feature in the common feature space respectively, and obtaining a linear transformation result of the visual feature and a linear change result of the language feature; The linear transformation result of the visual feature and the linear change result of the language feature are fused to obtain a multimodal feature.

8. The method for searching and editing objects in NeRF according to claim 5, characterized in that: The method of performing feature matching training on the pre-trained NeRF model according to the 3D scene image to be constructed by using the multimodal features to obtain a training result includes: Get a predefined loss function; Obtaining an objective function for the feature matching according to the loss function; The pre-trained NeRF model performs feature matching training according to the objective function to obtain a training result.

9. A device for searching and editing objects in NeRF, characterized in that include: A query receiving module, used for receiving a query request; A feature extraction module, used to extract features from the query content of the query request to obtain query features; The point feature vector acquisition module is used to obtain the point feature vector of each point in the NeRF-based 3D scene; A query result acquisition module is used to calculate the similarity between the query feature and the point feature vector, and determine the query result corresponding to the query content according to whether the similarity exceeds a preset similarity threshold; An editing module, used for editing the query result and updating the corresponding point feature vector; A rendering module is used to render the NeRF 3D scene according to the updated point feature vector.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method for searching and editing objects in NeRF as described in any one of claims 1 to 8 when executing executable instructions.

Citation Information

Cited By

  • Data search method and system based on artificial intelligence

    CN119227013A

  • An artificial intelligence-based data search method and system

    CN119227013B