A method, system, and computer program product for registering neural radiation fields.

Abstract

The present invention relates to the technical field of 3D scene registration, and discloses a neural radiation field registration method, system and computer program product. The registration method comprises: performing hierarchical downsampling on the point cloud, and determining the dense points corresponding to the coarse points by the nearest neighbor search method, and constructing an image point cloud block; injecting corresponding color and density information into the embedding vector by the deterministic projection function; according to the image point cloud block and the embedding vector, the deep semantic descriptor of the NeRF model is calculated by the edge convolution function; and the deep semantic descriptors of the two NeRF models to be registered are matched by using a coarse-to-fine matching strategy. The present invention proposes an algorithm framework for learning robust semantic descriptors using the rich cross-modal features in NeRF, thereby extracting effective point correspondences between NeRF pairs.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D scene registration, and in particular to a registration method, system and computer program product for a neural radiation field. Background Art

[0002] 3D scene registration is a fundamental problem in 3D computer vision, which involves estimating the relative transformation between two 3D scenes of arbitrary poses. This technology facilitates various real-world applications such as autonomous driving, VR (Virtual Reality) / AR (Augmented Reality), SLAM (Simultaneous Localization and Mapping). The registration task of three-dimensional data has been widely and deeply studied and applied on common 3D data representations such as depth maps, point clouds or meshes.

[0003] Recent advances in NeRF (neural radiance field) have brought new perspectives to research in this area. NeRF is a fully connected neural network that uses a rendering loss function to train the network to reproduce the perspective of the scene. Since 2D scene input images are discrete, NeRF renders the complete scene by interpolating them. NeRF is an efficient way to generate images for synthetic data. Research on large-scale NeRF has shown the potential of stringing together NeRF blocks to achieve city-level scene representation.

[0004] However, in the prior art, it is generally assumed that all NeRF blocks are trained in a unified camera coordinate system. In practice, it is challenging to ensure the availability of relative camera poses during image acquisition, as this requires high-precision multi-sensor fusion to track real-time trajectories and poses. However, the current existing NeRF registration methods only consider the extraction of semantic descriptors from global geometry or rendered photometric information. For example, the Chinese patent with application number "CN202410219005.7" and name "A Deep Registration Method and System for Neural Radiance Fields" introduces an SDF network (Signed Distance Function) in the process of training NeRF and iteratively optimizes it through the Levenberg-Marquardt algorithm to align the NeRF fields together. These methods ignore the rich cross-modal information presented in the embedded feature space of NeRF itself.

[0005] Therefore, for those skilled in the art, how to enhance the local view-independent text semantic features and improve the robustness of NeRF-based semantic descriptors has become an urgent problem to be solved by those skilled in the art. Summary of the invention

[0006] The object of the present invention is to provide a method, system and computer program product for registration of neural radiation fields, so as to solve or at least partially solve the technical problems mentioned in the above background technology.

[0007] To achieve this object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a method for registering a neural radiation field, comprising:

[0009] Using the parameters of NeRF’s encoder network, we extract the volume density and view-independent embedding vector of the query point location.

[0010] Using the parameters of NeRF's decoder network, calculate the color of the query point;

[0011] The center of the non-empty voxel grid is sampled as a point set to construct the point cloud of the NeRF model;

[0012] The point cloud is layered downsampled, and the dense points corresponding to the coarse points are determined by nearest neighbor search to construct an image point cloud block; wherein the coarse points are points selected from the coarsest layer of the point cloud, and the dense points are points selected from the second dense layer of the point cloud;

[0013] Inject the corresponding color and density information into the embedding vector through a deterministic projection function;

[0014] According to the image point cloud blocks and embedded vectors, the edge convolution function is used to calculate the deep semantic descriptor of the NeRF model;

[0015] A coarse-to-fine matching strategy is adopted to match the deep semantic descriptors of the two NeRF models to be registered.

[0016] Optionally, the extracting the volume density of the query point position and the view-independent embedding vector by using the parameters of the NeRF encoder network specifically includes:

[0017] Given a query point position within the bounding box , using the parameters of NeRF's encoder network , extract the volume density and the view-independent embedding vector , expressed as a function:

[0018] ;

[0019] The parameters of the NeRF decoder network are used to calculate the color of the query point position, specifically including:

[0020] Parameters of the decoder network using NeRF , calculate the color of the location query point ,Right now:

[0021] ;

[0022] Where n is the total number of images used to train NeRF, is the viewing angle of the i-th image, Represent the camera position and viewing direction respectively, is the characteristic dimension.

[0023] Optionally, the center of the non-empty voxel grid is sampled as a point set to construct a point cloud of the NeRF model, specifically:

[0024] For the input NeRF model, the occupancy mask Mo in the occupancy grid is used to speed up the sampling speed, and only the center of the non-empty voxel grid is used as the point set for sampling; specifically,

[0025] For the input NeRF model N X , build point cloud , so that ;

[0026] in, For NeRF model N X The set of points used for sampling, is a preset hyperparameter.

[0027] Optionally, the point cloud is layered downsampled, and dense points corresponding to rough points are determined by nearest neighbor search to construct image point cloud blocks, specifically:

[0028] KPConv is used to perform hierarchical downsampling of the point cloud and extract multi-level geometric features at the same time; the multi-level geometric features include points selected from the coarsest layer and the second dense layer and their corresponding features, specifically including:

[0029] For the constructed point cloud X, the rough point is represented as , the feature corresponding to the rough point is expressed as , the dense points are represented as , the features corresponding to the dense points are expressed as ;in, and is the characteristic dimension;

[0030] Given a rough point , determine the dense points corresponding to the rough points by nearest neighbor search, and construct the image point cloud block , expressed as:

[0031] ;in, is the preset radius threshold; Represents nearest neighbor search.

[0032] Optionally, the injecting corresponding color and density information into the embedding vector through a deterministic projection function is specifically:

[0033] Using sinusoidal position encoding as a deterministic projection function , is the embedding vector Inject the corresponding color and density information to get , It is expressed as:

[0034] ;

[0035] in, Represents a concatenation operation.

[0036] Optionally, the deep semantic descriptor of the NeRF model is obtained by calculating the edge convolution function according to the image point cloud block and the embedding vector, specifically:

[0037] Define the edge convolution function as: , the deep semantic descriptor of the NeRF model is defined as: ;

[0038] in, , is the bias weight used to capture the local high-frequency texture details in the point cloud block; x j and x k for Two connected nodes in the Indicates weight The convolutional layer, .

[0039] Optionally, the coarse-to-fine matching strategy is adopted to match the deep semantic descriptors of the two NeRF models to be registered, specifically:

[0040] GeoTransformer is used to align the acquired deep semantic descriptors.

[0041] In a second aspect, the present invention further provides a neural radiation field registration system, comprising:

[0042] A NeRF network information processing unit, for extracting the volume density and view-independent embedding vector of the query point location using the parameters of the NeRF encoder network;

[0043] The NeRF network information processing unit is also used to calculate the color of the query point position using the parameters of the NeRF decoder network;

[0044] A point cloud construction unit, electrically connected to the NeRF network information processing unit, for sampling the center of a non-empty voxel grid as a point set to construct a point cloud of a NeRF model;

[0045] A point cloud block construction unit, electrically connected to the point cloud construction unit, for performing hierarchical down-sampling on the point cloud, and determining dense points corresponding to the coarse points by a nearest neighbor search method, to construct an image point cloud block; wherein the coarse points are points selected from the coarsest layer of the point cloud, and the dense points are points selected from the second dense layer of the point cloud;

[0046] An information injection unit, electrically connected to the NeRF network information processing unit, for injecting corresponding color and density information into the embedding vector through a deterministic projection function;

[0047] A semantic descriptor calculation unit, electrically connected to the information injection unit and the point cloud block construction unit, for obtaining a deep semantic descriptor of a NeRF model by using an edge convolution function according to the image point cloud block and the embedding vector;

[0048] The registration unit is electrically connected to the semantic descriptor calculation unit and is used to adopt a coarse-to-fine matching strategy to match the deep semantic descriptors of the two NeRF models to be registered.

[0049] Optionally, the NeRF network information processing unit is specifically used for:

[0050] Given a query point position within the bounding box , using the parameters of NeRF's encoder network , extract the volume density and the view-independent embedding vector , expressed as a function:

[0051] ;

[0052] Parameters of the decoder network using NeRF , calculate the color of the location query point ,Right now:

[0053] ;

[0054] Where n is the total number of images used to train NeRF, is the viewing angle of the i-th image, Represent the camera position and viewing direction respectively, is the characteristic dimension.

[0055] Optionally, the point cloud construction unit is specifically used for:

[0056] For the input NeRF model, the occupancy mask Mo in the occupancy grid is used to speed up the sampling speed, and only the center of the non-empty voxel grid is used as the point set for sampling; specifically,

[0057] For the input NeRF model N X , build point cloud , so that ;

[0058] in, For NeRF model N X The set of points used for sampling, is a preset hyperparameter.

[0059] Optionally, the point cloud block construction unit is specifically used to:

[0060] KPConv is used to perform hierarchical downsampling of the point cloud and extract multi-level geometric features at the same time; the multi-level geometric features include points selected from the coarsest layer and the second dense layer and their corresponding features, specifically including:

[0061] For the constructed point cloud X, the rough point is represented as , the feature corresponding to the rough point is expressed as , the dense points are represented as , the features corresponding to the dense points are expressed as ;in, and is the characteristic dimension;

[0062] Given a rough point , determine the dense points corresponding to the rough points by nearest neighbor search, and construct the image point cloud block , expressed as:

[0063] ;in, is the preset radius threshold; Represents nearest neighbor search.

[0064] Optionally, the information injection unit is specifically used to:

[0065] Using sinusoidal position encoding as a deterministic projection function , is the embedding vector Inject the corresponding color and density information to get , It is expressed as:

[0066] ;

[0067] in, Represents a concatenation operation.

[0068] Optionally, the semantic descriptor calculation unit is specifically used to:

[0069] Define the edge convolution function as: , the deep semantic descriptor of the NeRF model is defined as: ;

[0070] in, , is the bias weight used to capture the local high-frequency texture details in the point cloud block; x j and x k for Two connected nodes in the Indicates weight The convolutional layer, .

[0071] Optionally, the registration unit is specifically used for:

[0072] GeoTransformer is used to align the acquired deep semantic descriptors.

[0073] In a third aspect, the present invention further provides a computer program product, comprising a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the above-mentioned neural radiation field registration method is implemented.

[0074] Compared with the prior art, the present invention has the following beneficial effects:

[0075] The present invention proposes a neural radiation field registration method and a cross-modal NeRF registration framework, which uses implicit color embedding in the radiation field to learn robust local feature descriptors; proposes a transformation-invariant deep semantic descriptor that can capture local high-frequency implicit color features; outputs point cloud correspondence constrained by semantic features, which can be secondary optimized using algorithms such as random sample consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0077] Figure 1 A flow chart of a neural radiation field registration method provided in an embodiment of the present invention.

[0078] Figure 2 A neural radiation field registration system architecture diagram provided in an embodiment of the present invention.

[0079] Figure 3 A pair of NeRF models to be registered is provided in an embodiment of the present invention.

[0080] Figure 4 A point cloud and embedded vector visualization image of a pair of NeRF models to be registered provided in an embodiment of the present invention.

[0081] Figure 5 A visualization image of a deep semantic descriptor of a NeRF model to be registered provided in an embodiment of the present invention.

[0082] Figure 6 A pair of point correspondence visualization images of a NeRF model to be registered provided in an embodiment of the present invention.

[0083] Figure 7 A visualized image of the final registration of a pair of NeRF models to be registered provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0084] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0085] Embodiment 1:

[0086] Please refer to Figure 1 , Figure 1 A flow chart of a neural radiation field registration method provided by an embodiment of the present invention, the method specifically includes:

[0087] Step 110: Using the parameters of the NeRF encoder network, extract the volume density and view-independent embedding vector of the query point location.

[0088] Please refer to Figure 3 and Figure 4 , Figure 3 A pair of NeRF models to be registered provided by an embodiment of the present invention, Figure 4 A point cloud and embedded vector visualization image of a pair of NeRF models to be registered provided in an embodiment of the present invention;

[0089] In this step, for ease of understanding, NeRF can be regarded as an encoder-decoder network; specifically, given a query point position within the bounding box , using the parameters of NeRF's encoder network , extract the volume density and the view-independent embedding vector , expressed as a function:

[0090] ;in, is the characteristic dimension.

[0091] Step 120: Calculate the color of the query point using the parameters of the NeRF decoder network.

[0092] In this step, the final color of the image rendered using NeRF is view-dependent and can be calculated by integrating along the sampling ray; the implementation method of step 120 is specifically as follows:

[0093] Parameters of the decoder network using NeRF , calculate the color of the location query point ,Right now:

[0094] ;

[0095] Where n is the total number of images used to train NeRF, is the viewing angle of the i-th image, Represent the camera position and viewing direction respectively.

[0096] Step 130: Sample the center of the non-empty voxel grid as a point set to construct a point cloud of the NeRF model.

[0097] In this step, for the input NeRF model, the occupancy mask Mo in the occupancy grid is used to speed up the sampling speed, and only the center of the non-empty voxel grid is used as the point set for sampling; X For example, the method of constructing point cloud includes:

[0098] For NeRF model N X , build point cloud , so that ;

[0099] in, For NeRF model N X The set of points used for sampling, is a preset hyperparameter.

[0100] Similarly, the above method is used to input another NeRF model N YConstruct point cloud Y. Since the construction method and principle are the same, they will not be repeated here.

[0101] Step 140: perform hierarchical downsampling on the point cloud, and determine the dense points corresponding to the coarse points by nearest neighbor search to construct an image point cloud block.

[0102] In this step, the coarse points are points selected from the coarsest layer of the point cloud, and the dense points are points selected from the second dense layer of the point cloud. The calculation process is described below using point cloud X as an example, which specifically includes:

[0103] KPConv is used to perform hierarchical downsampling of the point cloud and extract multi-level geometric features at the same time; wherein the multi-level geometric features include points selected from the coarsest layer and the second dense layer and their corresponding features:

[0104] For the constructed point cloud X, the rough point is represented as , the feature corresponding to the rough point is expressed as , the dense points are represented as , the features corresponding to the dense points are expressed as ;in, and is the characteristic dimension;

[0105] Given a rough point , determine the dense points corresponding to the rough points by nearest neighbor search, and construct the image point cloud block , expressed as:

[0106] ;in, is the preset radius threshold; Represents nearest neighbor search.

[0107] It should be noted that KPConv used in this step is a convolution specially designed for point clouds. It uses a kernel area defined by a radius to achieve high performance in areas of different densities. KPConv belongs to the disclosed prior art, so its principle will not be described in detail in this embodiment.

[0108] Step 150: Inject corresponding color and density information into the embedding vector through a deterministic projection function.

[0109] The embedded feature is to use a point in space (such as point x) to query the corresponding NeRF, so the embedded vector can be understood as the feature of NeRF output, that is, the output value of NeRF; then use the embedded vector combined with the viewing angle information, and then input NeRF to get the color information of the point in space.

[0110] In this example, although the pair of NeRF models to be registered share the same scene, their view-dependent embeddings are not in the same feature space because they are trained separately. However, NeRFs at the same position usually have similar colors. Therefore, we use the deterministic projection function Inject the corresponding color and density information. This has the following two advantages:

[0111] (1) The output features depend on the corresponding NeRF photometric information, which is shared by a pair of NeRF models;

[0112] (2) It enhances the mutual information across frames, which may stabilize the learning process.

[0113] Specifically, this step uses sinusoidal position encoding as the deterministic projection function , is the embedding vector Inject the corresponding color and density information to get , It is expressed as:

[0114] ;

[0115] in, Represents a concatenation operation.

[0116] Step 160: According to the image point cloud block and the embedding vector, the edge convolution function is used to calculate and obtain the deep semantic descriptor of the NeRF model.

[0117] Please refer to Figure 5 , Figure 5 A visualization image of a deep semantic descriptor of a NeRF model to be registered provided by an embodiment of the present invention;

[0118] In order to capture local high-frequency textures, this step introduces a novel texture-aware graph convolutional network. First, the image point cloud block is constructed through step 140 to obtain the set of connected nodes in the convolution function. ; Then, given two connected nodes x j and x k , we define the following edge convolution function:

[0119] ;

[0120] The deep semantic descriptor of the NeRF model is defined as: ;

[0121] in, , is the bias weight used to capture local high-frequency texture details in the point cloud block, which should emphasize points that are close but have obvious feature differences; x j and x k for Two connected nodes in the Indicates weight The convolutional layer, .

[0122] Step 170: Use a coarse-to-fine matching strategy to match the obtained deep semantic descriptors of the two NeRF models to be registered.

[0123] Deep semantic descriptors use sparse downsampled points to ensure feature discriminability. However, directly building the final point correspondences under such sparsity may lead to registration failure. To address this issue, this step adopts a coarse-to-fine registration method that optimizes sparse correspondences with dense geometric features.

[0124] Specifically, the coarse-to-fine strategy proposed by GeoTransformer is used to align the acquired deep semantic descriptors.

[0125] In this step, the relevant principles of the registration technology proposed by GeoTransformer can be referred to the published article, namely: the academic paper titled "Geotransformer: Fast and robust point cloud registration with geometric transformer" with the first author being "Qin, Zheng".

[0126] Please refer to Figure 6 and Figure 7 , Figure 6 A pair of point correspondence visualization images of a NeRF model to be registered provided by an embodiment of the present invention, Figure 7 A visualized image of the final registration of a pair of NeRF models to be registered provided by an embodiment of the present invention. Figure 7 The upper right corner and Figure 7 The lower left images are rendered using the aligned NeRF.

[0127] This embodiment proposes a neural radiation field registration method, which uses the rich cross-modal features in NeRF to learn robust semantic descriptors, thereby extracting an algorithm framework for point correspondence between valid NeRF pairs; specifically, it includes:

[0128] (1) Leveraging implicit color embedding in the radiance field to learn robust local feature descriptors;

[0129] (2) A transformation-invariant deep semantic descriptor is proposed to capture local high-frequency implicit color features;

[0130] (3) The point cloud correspondence constrained by semantic features is output, which can be optimized twice using algorithms such as random sample consensus.

[0131] Embodiment 2:

[0132] Please refer to Figure 2 , Figure 2 A neural radiation field registration system architecture diagram provided in an embodiment of the present invention. The registration system includes:

[0133] A NeRF network information processing unit 10, for extracting a volume density and a view-independent embedding vector of a query point position using parameters of an encoder network of NeRF;

[0134] The NeRF network information processing unit 10 is also used to calculate the color of the query point position using the parameters of the NeRF decoder network;

[0135] A point cloud construction unit 20, electrically connected to the NeRF network information processing unit 10, for sampling the center of a non-empty voxel grid as a point set to construct a point cloud of a NeRF model;

[0136] The point cloud block construction unit 30 is electrically connected to the point cloud construction unit 20, and is used to perform hierarchical downsampling on the point cloud, and determine the dense points corresponding to the coarse points by a nearest neighbor search method to construct an image point cloud block; wherein the coarse points are points selected from the coarsest layer of the point cloud, and the dense points are points selected from the second dense layer of the point cloud;

[0137] An information injection unit 40, electrically connected to the NeRF network information processing unit 10, for injecting corresponding color and density information into the embedding vector through a deterministic projection function;

[0138] A semantic descriptor calculation unit 50, electrically connected to the information injection unit 40 and the point cloud block construction unit 30, for obtaining a deep semantic descriptor of a NeRF model by using an edge convolution function according to the image point cloud block and the embedding vector;

[0139] The registration unit 60 is electrically connected to the semantic descriptor calculation unit 50 and is used to match the deep semantic descriptors of the two NeRF models to be registered by adopting a coarse-to-fine matching strategy.

[0140] Specifically, the NeRF network information processing unit 10 is specifically used for:

[0141] Given a query point position within the bounding box , using the parameters of NeRF's encoder network , extract the volume density and the view-independent embedding vector , expressed as a function:

[0142] ;

[0143] Parameters of the decoder network using NeRF , calculate the color of the location query point ,Right now:

[0144] ;

[0145] Where n is the total number of images used to train NeRF, is the viewing angle of the i-th image, Represent the camera position and viewing direction respectively, is the characteristic dimension.

[0146] Specifically, the point cloud construction unit 20 is specifically used for:

[0147] For the input NeRF model, the occupancy mask Mo in the occupancy grid is used to speed up the sampling speed, and only the center of the non-empty voxel grid is used as the point set for sampling; specifically,

[0148] For the input NeRF model N X , build point cloud , so that ;

[0149] in, For NeRF model N X The set of points used for sampling, is a preset hyperparameter.

[0150] Specifically, the point cloud block construction unit 30 is specifically used for:

[0151] KPConv is used to perform hierarchical downsampling of the point cloud and extract multi-level geometric features at the same time; the multi-level geometric features include points selected from the coarsest layer and the second dense layer and their corresponding features, specifically including:

[0152] For the constructed point cloud X, the rough point is represented as , the feature corresponding to the rough point is expressed as , the dense points are represented as , the features corresponding to the dense points are expressed as ;in, and is the characteristic dimension;

[0153] Given a rough point , determine the dense points corresponding to the rough points by nearest neighbor search, and construct the image point cloud block , expressed as:

[0154] ;in, is the preset radius threshold; Represents nearest neighbor search.

[0155] Specifically, the information injection unit 40 is specifically used for:

[0156] Using sinusoidal position encoding as a deterministic projection function , is the embedding vector Inject the corresponding color and density information to get , It is expressed as:

[0157] ;

[0158] in, Represents a concatenation operation.

[0159] Specifically, the semantic descriptor calculation unit 50 is specifically used for:

[0160] Define the edge convolution function as: , the deep semantic descriptor of the NeRF model is defined as: ;

[0161] in, , is the bias weight used to capture the local high-frequency texture details in the point cloud block; x j and x k for Two connected nodes in the Indicates weight The convolutional layer, .

[0162] Specifically, the registration unit 60 is specifically used for:

[0163] GeoTransformer is used to align the acquired deep semantic descriptors.

[0164] This embodiment proposes a neural radiation field registration system, which utilizes the rich cross-modal features in NeRF to learn robust semantic descriptors, thereby extracting an algorithm framework for point correspondences between valid NeRF pairs.

[0165] Based on the same concept, an embodiment of the present invention further provides a computer-readable storage medium, in which at least one instruction is stored. The instruction is loaded and executed by a processor to implement a neural radiation field registration method provided by an embodiment of the present invention.

[0166] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0167] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0168] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0169] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0170] Based on the same concept, an embodiment of the present invention further provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, a neural radiation field registration method provided by an embodiment of the present invention is implemented.

[0171] The computer program product may be loaded on a computer device, the components of which may include but are not limited to: one or more processors or processing units, a system memory, and a bus connecting different system components (including the system memory and the processing unit).

[0172] Computer devices typically include a variety of computer system readable media. These media can be any available media that can be accessed by the computer device, including volatile and non-volatile media, removable and non-removable media.

[0173] The system memory may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The computer device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system may be used to read and write non-removable, non-volatile magnetic media. The computer program product has a set (e.g., at least one) of program modules that are configured to perform the functions of various embodiments of the present invention.

[0174] A program / utility having a set (at least one) of program modules may be stored, for example, in a memory, such program modules including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules generally perform the functions and / or methods of the embodiments described herein.

[0175] The computer device can also communicate with one or more external devices (such as keyboards, pointing devices, displays, etc.), and can also communicate with one or more devices that enable users to interact with the computer device, and / or communicate with any device (such as a network card, a modem, etc.) that enables the computer device to communicate with one or more other computing devices. This communication can be carried out through an input / output (I / O) interface. In addition, the computer device can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter. As shown in the figure, the network adapter communicates with other modules of the computer device through a bus. It should be understood that other hardware and / or software modules can be used in conjunction with the computer device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0176] The processing unit executes various functional applications and data processing by running the programs stored in the system memory, for example, implementing a neural radiation field registration method provided in an embodiment of the present invention.

[0177] Those skilled in the art can understand that all or part of the steps of the above embodiments can be completed by hardware, or by a program to instruct the relevant hardware to complete the steps. The program can be stored in a computer-readable storage medium, and the storage medium mentioned above can be a read-only memory, a disk or an optical disk, etc. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0178] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for registering a neural radiation field, characterized in that: include: Using the parameters of NeRF’s encoder network, we extract the volume density and view-independent embedding vector of the query point location. Using the parameters of NeRF's decoder network, calculate the color of the query point; The center of the non-empty voxel grid is sampled as a point set to construct the point cloud of the NeRF model; The point cloud is layered downsampled, and the dense points corresponding to the coarse points are determined by nearest neighbor search to construct an image point cloud block; wherein the coarse points are points selected from the coarsest layer of the point cloud, and the dense points are points selected from the second dense layer of the point cloud; Inject the corresponding color and density information into the embedding vector through a deterministic projection function; According to the image point cloud blocks and embedded vectors, the edge convolution function is used to calculate the deep semantic descriptor of the NeRF model; A coarse-to-fine matching strategy is adopted to match the deep semantic descriptors of the two NeRF models to be registered. The point cloud is layered downsampled, and the dense points corresponding to the rough points are determined by nearest neighbor search to construct the image point cloud block, specifically: KPConv is used to perform hierarchical downsampling of the point cloud and extract multi-level geometric features at the same time; the multi-level geometric features include points selected from the coarsest layer and the second dense layer and their corresponding features, specifically including: For the constructed point cloud X, the rough point is represented as The feature corresponding to the rough point is expressed as The dense points are represented as The features corresponding to the dense points are expressed as Among them, d k and d s is the characteristic dimension; Given a rough point Determine the dense points corresponding to the rough points by nearest neighbor search and construct the image point cloud block p i , expressed as: Among them, τ is the preset radius threshold; K n (·) indicates nearest neighbor search.

2. A neural radiation field registration method according to claim 1, characterized in that: The parameters of the NeRF encoder network are used to extract the volume density of the query point position and the view-independent embedding vector, specifically including: Given a query point position within the bounding box The parameters θ of the encoder network using NeRF e , extract the volume density σ x and the view-independent embedding vector It can be expressed as a function: s x ,e x =F n (x;θ e ); The parameters of the NeRF decoder network are used to calculate the color of the query point position, specifically including: The parameters θ of the decoder network using NeRF d , calculate the color c of the position query point x ,Right now: Where n is the total number of images used to train NeRF, r i =(o i ,d i ) is the viewing angle of the i-th image, o i ,d i Represent the camera position and viewing direction respectively, d e is the characteristic dimension.

3. A neural radiation field registration method according to claim 2, characterized in that: The center of the non-empty voxel grid is sampled as a point set to construct a point cloud of the NeRF model, specifically: For the input NeRF model, the occupancy mask Mo in the occupancy grid is used to speed up the sampling speed, and only the center of the non-empty voxel grid is used as the point set for sampling; specifically, For the input NeRF model N X , build point cloud Make in, For NeRF model N X The point set used for sampling, σ t is a preset hyperparameter.

4. A neural radiation field registration method according to claim 3, characterized in that: The corresponding color and density information is injected into the embedding vector through the deterministic projection function, specifically: Use sinusoidal position encoding as the deterministic projection function S 4→de (·), is the embedding vector e i ∈e x Inject the corresponding color and density information to get It is expressed as: Among them, [·;·] represents the concatenation operation.

5. A neural radiation field registration method according to claim 4, characterized in that: The deep semantic descriptor of the NeRF model is obtained by calculating the edge convolution function according to the image point cloud block and the embedded vector, specifically: Define the edge convolution function as: The deep semantic descriptor of the NeRF model is defined as: in, is the bias weight used to capture the local high-frequency texture details in the point cloud block; x j and x k are two connected nodes in ε; Represents with weight θ g The convolutional layer, 6. A neural radiation field registration method according to claim 5, characterized in that: The coarse-to-fine matching strategy is adopted to match the deep semantic descriptors of the two NeRF models to be registered, specifically: GeoTransformer is used to align the acquired deep semantic descriptors.

7. A neural radiation field registration system, characterized in that: include: A NeRF network information processing unit, for extracting the volume density and view-independent embedding vector of the query point location using the parameters of the NeRF encoder network; The NeRF network information processing unit is also used to calculate the color of the query point position using the parameters of the NeRF decoder network; A point cloud construction unit, electrically connected to the NeRF network information processing unit, for sampling the center of a non-empty voxel grid as a point set to construct a point cloud of a NeRF model; A point cloud block construction unit, electrically connected to the point cloud construction unit, for performing hierarchical down-sampling on the point cloud, and determining dense points corresponding to the coarse points by a nearest neighbor search method, to construct an image point cloud block; wherein the coarse points are points selected from the coarsest layer of the point cloud, and the dense points are points selected from the second dense layer of the point cloud; An information injection unit, electrically connected to the NeRF network information processing unit, for injecting corresponding color and density information into the embedding vector through a deterministic projection function; A semantic descriptor calculation unit, electrically connected to the information injection unit and the point cloud block construction unit, for obtaining a deep semantic descriptor of a NeRF model by using an edge convolution function according to the image point cloud block and the embedding vector; A registration unit, electrically connected to the semantic descriptor calculation unit, for matching the obtained deep semantic descriptors of the two NeRF models to be registered by adopting a coarse-to-fine matching strategy; The point cloud block construction unit is specifically used for: KPConv is used to perform hierarchical downsampling of the point cloud and extract multi-level geometric features at the same time; the multi-level geometric features include points selected from the coarsest layer and the second dense layer and their corresponding features, specifically including: For the constructed point cloud X, the rough point is represented as The feature corresponding to the rough point is expressed as The dense points are represented as The features corresponding to the dense points are expressed as Among them, d k and d s is the characteristic dimension; Given a rough point Determine the dense points corresponding to the rough points by nearest neighbor search and construct the image point cloud block p i , expressed as: Among them, τ is the preset radius threshold; K n (·) indicates nearest neighbor search.

8. A neural radiation field registration system according to claim 7, characterized in that: The NeRF network information processing unit is specifically used for: Given a query point position within the bounding box The parameters θ of the encoder network using NeRF e , extract the volume density σ x and the view-independent embedding vector It can be expressed as a function: s x ,e x =F n (x;θ e ); The parameters θ of the decoder network using NeRF d , calculate the color c of the position query point x ,Right now: Where n is the total number of images used to train NeRF, r i =(o i ,d i ) is the viewing angle of the i-th image, o i ,d i Represent the camera position and viewing direction respectively, d e is the characteristic dimension.

9. A neural radiation field registration system according to claim 8, characterized in that: The point cloud construction unit is specifically used for: For the input NeRF model, the occupancy mask Mo in the occupancy grid is used to speed up the sampling speed, and only the center of the non-empty voxel grid is used as the point set for sampling; specifically, For the input NeRF model N X , build point cloud Make in, For NeRF model N X The point set used for sampling, σ t is a preset hyperparameter.

10. A neural radiation field registration system according to claim 9, characterized in that: The information injection unit is specifically used for: Use sinusoidal position encoding as the deterministic projection function S 4→de (·), is the embedding vector e i ∈e x Inject the corresponding color and density information to get It is expressed as: Among them, [·;·] represents the concatenation operation.

11. A neural radiation field registration system according to claim 10, characterized in that: The semantic descriptor calculation unit is specifically used for: Define the edge convolution function as: The deep semantic descriptor of the NeRF model is defined as: in, is the bias weight used to capture the local high-frequency texture details in the point cloud block; x j and x k are two connected nodes in ε; h θg Represents with weight θ g The convolutional layer, 12. A neural radiation field registration system according to claim 11, characterized in that: The registration unit is specifically used for: GeoTransformer is used to align the acquired deep semantic descriptors.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, a neural radiation field registration method as described in any one of claims 1 to 6 is implemented.

Images