An infrared image positioning and orientation method and device with a neural radiation field as a reference

Abstract

The application discloses an infrared image positioning and pose estimation method and device with a neural radiance field as a reference. The method comprises the following steps: inputting an infrared image and prior pose information into a trained neural radiance field model to obtain a rendered RGB image and depth information of the rendered RGB image; performing two-dimensional image feature point matching on the infrared image and the rendered RGB image to obtain two-dimensional matching points, and converting the two-dimensional matching points into three-dimensional space points in combination with the depth information; calculating a rigid transformation matrix based on the three-dimensional space points to obtain estimated pose information; repeating the processing steps of the neural radiance field model, the two-dimensional image feature point matching step and the step of obtaining the estimated pose information by taking the estimated pose information as the prior pose information until the estimated pose information meets a preset requirement, and taking the latest estimated pose information as the final estimated pose information. The application can avoid errors caused by light changes and can accurately estimate the pose under weak light or even no light.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an infrared image positioning and orientation method, device, storage medium and electronic device that uses a neural radiation field as a reference. Background Technology

[0002] Image pose describes the position and orientation of the camera in three-dimensional space, and is usually expressed using six degrees of freedom.

[0003] Currently, pose estimation techniques based on neural radiation fields typically include two methods: one is to obtain a rendered image using the neural radiation field and then iteratively calculate the photometric loss between the query image and the rendered image at the current pose to recover the pose of the query image; the other is to establish a geometric correspondence between the image rendered by the neural radiation field and the query image to perform pose recovery. However, in actual pose estimation, due to the large variations in day and night illumination, pose estimation of RGB images is usually accompanied by significant errors, resulting in low accuracy. Summary of the Invention

[0004] This application provides an infrared image positioning and pose estimation method, device, storage medium, and electronic device that uses a neural radiation field as a reference. It can avoid errors introduced by changes in illumination and can perform accurate pose estimation under low light or even no light conditions.

[0005] This application provides an infrared image positioning and orientation method with reference to a neural radiation field, including:

[0006] Acquire infrared images and prior pose information of the infrared images;

[0007] The infrared image and the prior pose information are input into the trained neural radiation field model to obtain the rendered RGB image and the depth information of the rendered RGB image.

[0008] Two-dimensional image feature point matching is performed on the infrared image and the rendered RGB image to obtain two-dimensional matching points, and the two-dimensional matching points are converted into three-dimensional spatial points by combining the depth information;

[0009] The rigid body transformation matrix is ​​calculated based on the three-dimensional spatial points to obtain the estimated pose information;

[0010] The estimated pose information is used as the prior pose information to repeat the above-described neural radiation field model processing steps, two-dimensional image feature point matching steps, and steps to obtain the estimated pose information, until the obtained estimated pose information meets the preset requirements. The latest estimated pose information is then used as the final estimated pose information.

[0011] Furthermore, according to the above-mentioned infrared image positioning and attitude determination method with reference to the neural radiation field, the neural radiation field model includes two layers of multilayer perceptron, and the prior pose information includes position information, orientation angle and pitch angle.

[0012] The infrared image and the prior pose information are input into the trained neural radiation field model to obtain the rendered RGB image and the depth information of the rendered RGB image, including:

[0013] The infrared image and the location information are input into the first layer of the multilayer perceptron to obtain the density of each point and the feature vector of each point;

[0014] The feature vector, the orientation angle, and the pitch angle are input into the second layer of the multilayer perceptron to obtain the color value of each point;

[0015] Based on the color value of each point and the volume rendering function, the rendered RGB image is obtained;

[0016] The depth information of the rendered RGB image is calculated based on the prior pose information.

[0017] Furthermore, according to the above-mentioned infrared image positioning and pose determination method using neural radiation fields as a reference, the volume rendering function is:

[0018]

[0019] in, , Indicates along the light ray The RGB color values ​​obtained from the rendering It is the first Volume density at each sampling point It is the first The sampling point and the first +1 distance between sampling points, i.e., sampling interval. Refers to the first Each sampling point is the RGB color value output by the second layer multilayer perceptron.

[0020] Furthermore, according to the above-mentioned infrared image positioning and pose determination method using neural radiation fields as a reference, the process of performing two-dimensional image feature point matching on the infrared image and the rendered RGB image to obtain two-dimensional matching points, and then converting the two-dimensional matching points into three-dimensional spatial points by combining the depth information, includes:

[0021] The infrared image and the rendered RGB image are input into the SuperPoint neural network to obtain the infrared feature coordinate set and infrared feature vector of the infrared image, and the RGB feature coordinate set and RGB feature vector of the rendered RGB image.

[0022] The infrared feature coordinate set, infrared feature vector, RGB feature coordinate set, and RGB feature vector are input into the SuperGlue neural network to obtain the key point matching matrix and confidence vector.

[0023] Furthermore, according to the above-mentioned infrared image positioning and pose determination method using neural radiation fields as a reference, the estimated pose information is calculated using the following formula:

[0024]

[0025] in, Represents the pixel coordinates in the infrared image. Represents the coordinates of a three-dimensional point in space. The desired pose information includes R= 、 Translation vector t = , ( () indicates the camera's internal parameters.

[0026] Furthermore, according to the above-mentioned infrared image positioning and attitude determination method with reference to the neural radiation field, the method further includes:

[0027] The latest estimated pose information is input into the neural radiation field model to obtain the corresponding rendered RGB image and depth information;

[0028] Feature point matching is performed on the rendered RGB image and depth information. The angle between the camera and each pair of matched feature points is calculated. When the error of the angle is not greater than the error angle threshold, the pose estimation is confirmed to be accurate. The operation is stopped and the estimated pose information of the current infrared image is used as the final estimated pose information.

[0029] Furthermore, according to the above-mentioned infrared image positioning and pose determination method using neural radiation fields as a reference, the training process of the neural radiation field model includes:

[0030] Acquire infrared images, prior pose information of the infrared images, and RGB information of light rays in the observation data;

[0031] The infrared image and the prior pose information are input into the trained neural radiation field model to obtain the rendered RGB image and the depth information of the rendered RGB image.

[0032] The mean square error between the rendered RGB image and the RGB information of light in the observation data is calculated as a loss function, and the parameters of the neural radiation field model are updated based on the loss function.

[0033] This application also provides an infrared image positioning and attitude determination device with reference to a neural radiation field, including:

[0034] The acquisition module is used to acquire infrared images and the prior pose information of the infrared images;

[0035] The neural radiation field module is used to input the infrared image and the prior pose information into the trained neural radiation field model to obtain the rendered RGB image and the depth information of the rendered RGB image.

[0036] The matching and conversion module is used to perform two-dimensional image feature point matching on the infrared image and the rendered RGB image to obtain two-dimensional matching points, and to convert the two-dimensional matching points into three-dimensional spatial points by combining the depth information.

[0037] The pose estimation module is used to calculate the rigid body transformation matrix based on the three-dimensional space points to obtain the estimated pose information;

[0038] The iterative module is used to repeat the above-mentioned neural radiation field model processing steps, two-dimensional image feature point matching steps, and steps to obtain estimated pose information, using the estimated pose information as prior pose information, until the obtained estimated pose information meets the preset requirements, and the latest estimated pose information is used as the final estimated pose information.

[0039] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute any of the above-described infrared image positioning and attitude determination methods with reference to a neural radiation field.

[0040] This application also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used in the steps of the infrared image positioning and attitude determination method with reference to the neural radiation field described in any of the above claims.

[0041] This application provides an infrared image localization and pose estimation method, apparatus, storage medium, and electronic device based on a neural radiation field. This application utilizes a neural radiation field model to obtain a rendered RGB image and depth information from the prior pose information of the infrared image. Two-dimensional image matching is performed between the rendered RGB image and the infrared image. The two-dimensional matching points are then transformed into three-dimensional matching points using the depth map. A PnP solver with RANSAN is used to calculate the rigid body transformation matrix to obtain the estimated pose information. Therefore, this invention has the following advantages: high pose estimation accuracy; compared to using RGB images with coarse prior pose information for pose estimation; avoids errors introduced by illumination changes; and can perform pose estimation under low light or even no light conditions. Attached Figure Description

[0042] The technical solution and other beneficial effects of this application will become apparent from the following detailed description of specific embodiments in conjunction with the accompanying drawings.

[0043] Figure 1 A flowchart of an infrared image positioning and attitude determination method with reference to a neural radiation field, provided for embodiments of this application.

[0044] Figure 2 This is a schematic diagram illustrating a feature point matching example provided in an embodiment of this application.

[0045] Figure 3 A checkerboard overlay of an infrared image with accurate pose estimation and a rendered RGB image obtained based on its pose information, provided in an embodiment of this application.

[0046] Figure 4 This is a schematic diagram of the structure of an infrared image positioning and attitude determination device with reference to a neural radiation field, provided in an embodiment of this application.

[0047] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] This application provides an infrared image positioning and attitude determination method, apparatus, storage medium, and electronic device that uses a neural radiation field as a reference. The infrared image positioning and attitude determination apparatus provided in this application, which uses a neural radiation field as a reference, can be integrated into an electronic device, such as a terminal or server. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0050] Please see Figure 1 , Figure 1 The flowchart illustrates an infrared image positioning and orientation method with reference to a neural radiation field, provided in an embodiment of this application. This method, applied in electronic devices, includes the following steps:

[0051] S1, acquire infrared image and prior pose information of infrared image.

[0052] Use a camera capable of acquiring precise pose information during image capture to obtain infrared images of the area to be estimated. The number of images should be sufficient for constructing the neural radiation field. Simultaneously acquire coarse pose information while capturing the infrared images. A narrow field-of-view mode of the infrared camera can be used for this purpose.

[0053] S2, input the infrared image and prior pose information into the trained neural radiation field model to obtain the rendered RGB image. and depth information for rendering RGB images .

[0054] Before step S2, a neural radiation field model is trained using RGB images with accurate pose information, serving as a reference map for visual pose estimation. During network training, the mean squared error between the predicted RGB color information of the light rays and the RGB information of the light rays in the observed data is used as a loss function to guide the parameter updates in the neural radiation field model.

[0055] The training process of the neural radiation field model specifically includes:

[0056] A1, acquire infrared images, prior pose information of infrared images, and RGB information of light in the observation data;

[0057] A2, input the infrared image and prior pose information into the trained neural radiation field model to obtain the rendered RGB image and the depth information of the rendered RGB image;

[0058] A3 calculates the mean square error between the rendered RGB image and the RGB information of light in the observation data as the loss function, and updates the parameters of the neural radiation field model based on the loss function.

[0059] In one embodiment, the neural radiation field model includes a two-layer multilayer perceptron, and the prior pose information includes position information, orientation angle, and pitch angle. Step S2 includes the following steps:

[0060] S21, the infrared image and location information are input into the first layer of the multilayer perceptron to obtain the density of each point and the feature vector of each point.

[0061] The first layer multilayer perceptron receives the position of the midpoint of the scene. Predict the density at each point Here, density refers to the degree of opacity at that point in space. The expression is as follows:

[0062]

[0063] in, This refers to the feature vector of the corresponding point output by the first layer of the multilayer perceptron.

[0064] S22, the feature vector, orientation angle and pitch angle are input into the second layer of the multilayer perceptron to obtain the color value of each point.

[0065] The second-layer multilayer perceptron network receives the feature vector generated by the first-layer multilayer perceptron network and the direction of the light ray at that point as input parameters, and then predicts the color value of that point. The expression is as follows:

[0066]

[0067] in, These refer to the azimuth angle and pitch angle corresponding to the image pose, respectively.

[0068] S23, based on the color value of each point and the volume rendering function, obtain the rendered RGB image.

[0069] Light in space is represented as , where o represents the starting point of the light ray (i.e., the camera center of the image). Indicates the direction of light. The sampling step size is defined as follows. By sampling a series of spatial points along the light rays traversing the target scene and combining this with the volume rendering function, the color value of the light rays and their termination position in the scene can be predicted, thereby outputting a rendered image and a depth map. The volume rendering function formula is as follows:

[0070]

[0071] in, , Indicates along the light ray The RGB color values ​​obtained from the rendering It is the first Volume density at each sampling point It is the first The sampling point and the first +1 distance between sampling points, i.e., sampling interval. Refers to the first Each sampling point is the RGB color value output by the second layer multilayer perceptron.

[0072] S24, calculate the depth information of the rendered RGB image based on prior pose information.

[0073] Specifically, the 3D points are first transformed from the world coordinate system to the camera coordinate system using prior pose information. Then, the 3D points in the camera coordinate system are projected to 2D pixel coordinates using camera intrinsic parameters. Finally, the Z value (or Euclidean distance) in the camera coordinate system is directly taken as the depth.

[0074] S3 performs two-dimensional image feature point matching on infrared images and rendered RGB images to obtain two-dimensional matching points, and combines depth information to convert the two-dimensional matching points into three-dimensional spatial points.

[0075] In one embodiment, Figure 2 This is a schematic diagram illustrating feature point matching examples provided in the embodiments of this application, such as... Figure 2 As shown, there are a total of 3 sets of matching examples. The left column is the rendered RGB image, and the right column is the infrared image. S3 includes the following steps:

[0076] S31, input the infrared image and the rendered RGB image into the SuperPoint neural network to obtain the infrared feature coordinate set and infrared feature vector of the infrared image, and the RGB feature coordinate set and RGB feature vector of the rendered RGB image.

[0077] S32, the infrared feature coordinate set, infrared feature vector, RGB feature coordinate set, and RGB feature vector are input into the SuperGlue neural network to obtain the key point matching matrix and confidence vector.

[0078] Specifically, infrared images and rendering RGB images The data is input into the SuperPoint neural network (a self-supervised feature point detection and description network). For each image, the SuperPoint neural network outputs an infrared image. set of infrared feature point coordinates Rendering RGB images RGB feature coordinate set The set of RGB feature coordinates for rendering RGB images and the RGB feature vectors for rendering RGB images .Will , , , Inputting a SuperGlue neural network (using a graph neural network to learn feature matching) outputs a keypoint matching matrix and a confidence vector. The matching matrix represents... and Which key points in the image have corresponding relationships? The confidence vector represents the matching confidence of the corresponding points. Among these, infrared images... and rendering RGB images The number of matched points is equal and corresponds one-to-one.

[0079] The SuperPoint neural network employs a fully convolutional neural network (FCN), and its structure includes: a shared encoder: VGG-style convolutional layers for extracting multi-scale features; a feature point detection head: outputting a 65-channel tensor (64 8×8 grid regions + 1 "feature point-free" channel), which calculates the feature point probability through Softmax; and a descriptor generation head: outputting a 256-dimensional descriptor, which is then bilinearly interpolated and L2 normalized to obtain the final descriptor.

[0080] The SuperGlue neural network mainly consists of two key modules: Attention Map Neural Network (GNN): The input is feature points (position + descriptor) from two images. Information is aggregated through multiple GNNs, and self-attention and cross-attention mechanisms are used to simulate the "back and forth observation" process when humans match features, thereby enhancing the specificity of features. The Keypoint Encoder fuses visual descriptors and positional information to improve matching accuracy. Optimal Matching Layer: The matching score matrix is ​​calculated, and the Sinkhorn algorithm is used to solve the differentiable optimal transmission problem to obtain the soft assignment matrix. A "dustbin" mechanism is introduced to handle occluded or unmatched feature points.

[0081] S4 calculates the rigid body transformation matrix based on three-dimensional spatial points to obtain estimated pose information.

[0082] The rigid body transformation matrix is ​​calculated using the PnP solver with RANSAN to obtain the estimated pose information.

[0083] Using a rendering depth map infrared images The two-dimensional matching points are converted into three-dimensional spatial points in the reference coordinates of the neural radiation field. This is achieved using the camera's intrinsic parameters (…). Based on the correspondence between 3D feature points and 2D feature points in infrared images, the precise pose of the camera is estimated using the PnP solver of the RANSAC algorithm. The formula is as follows:

[0084]

[0085] in, Represents the pixel coordinates in the infrared image. Represents the coordinates of a three-dimensional point in space. The desired pose information includes the rotation matrix. R= 、 Translation vector t = , ( () indicates the camera's internal parameters.

[0086] S5, using the estimated pose information as prior pose information, repeat the above processing steps of the neural radiation field model, the two-dimensional image feature point matching step, and the step of obtaining the estimated pose information (i.e., steps S2-S4) until the obtained estimated pose information meets the preset requirements, and use the latest estimated pose information as the final estimated pose information.

[0087] The preset requirements are set as follows:

[0088] The latest estimated pose information is input into the neural radiation field model to obtain the corresponding rendered RGB image and depth information. Feature point matching is performed on the rendered RGB image and depth information, and the angle between the camera and each pair of matched feature points is calculated. When the error of the angle is not greater than the error angle threshold, the pose estimation is confirmed to be accurate, the operation stops, and the estimated pose information of the current infrared image is used as the final estimated pose information. In one embodiment, the error angle threshold can be 1°. It is recommended to set the maximum number of repetitions to 3. Figure 3 A checkerboard overlay of an infrared image with accurate pose estimation and a rendered RGB image obtained based on its pose information, as provided in the embodiments of this application, is shown below. Figure 3 As shown, the infrared image with accurate pose estimation in the embodiment is superimposed with the rendered RGB image obtained based on its pose information in a checkerboard pattern. The smooth transition of the checkerboard square edges indicates high pose estimation accuracy.

[0089] Based on the method described in the above embodiments, this embodiment will further describe it from the perspective of an infrared image positioning and attitude determination device with reference to a neural radiation field. The infrared image positioning and attitude determination device with reference to a neural radiation field can be implemented as an independent entity or integrated into an electronic device. The electronic device can be a terminal, server, or other devices. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a microprocessor box, or other devices.

[0090] Please see Figure 4 , Figure 4 This application provides a detailed description of an infrared image positioning and attitude determination device with reference to a neural radiation field, which is applied in electronic devices. The device may include:

[0091] The acquisition module is used to acquire infrared images and the prior pose information of the infrared images;

[0092] The neural radiation field module is used to input the infrared image and the prior pose information into the trained neural radiation field model to obtain the rendered RGB image and the depth information of the rendered RGB image.

[0093] The matching and conversion module is used to perform two-dimensional image feature point matching on the infrared image and the rendered RGB image to obtain two-dimensional matching points, and to convert the two-dimensional matching points into three-dimensional spatial points by combining the depth information.

[0094] The pose estimation module is used to calculate the rigid body transformation matrix based on the three-dimensional space points to obtain the estimated pose information;

[0095] The iterative module is used to repeat the above-mentioned neural radiation field model processing steps, two-dimensional image feature point matching steps, and steps to obtain estimated pose information, using the estimated pose information as prior pose information, until the obtained estimated pose information meets the preset requirements, and the latest estimated pose information is used as the final estimated pose information.

[0096] In specific implementation, the above modules and / or units can be implemented as independent entities, or they can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of the above modules and / or units, please refer to the previous method embodiments. For the specific beneficial effects that can be achieved, please also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0097] In addition, this application also provides an electronic device, which may be a computer, tablet computer, or other similar device. This electronic device can implement the steps of any embodiment of the infrared image positioning and attitude determination method with reference to a neural radiation field provided in this application. Therefore, it can achieve the beneficial effects that any infrared image positioning and attitude determination method with reference to a neural radiation field provided in this invention can achieve, as detailed in the preceding embodiments, and will not be repeated here.

[0098] Figure 5 A specific structural block diagram of an electronic device provided in an embodiment of the present invention is shown. This electronic device can be used to implement the infrared image positioning and attitude determination method with reference to the neural radiation field provided in the above embodiments. The electronic device 500 can be a terminal, server, or other device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0099] RF circuit 510 is used to receive and transmit electromagnetic waves, converting electromagnetic waves into electrical signals and vice versa, thereby enabling communication with communication networks or other devices. RF circuit 510 may include various existing circuit elements used to perform these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, subscriber identity modules (SIM cards), memory, etc. RF circuit 510 can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices via wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks (WLANs), or metropolitan area networks (MANs). The aforementioned wireless networks may use various communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging, and short messages, and any other suitable communication protocols, including those that have not yet been developed.

[0100] The memory 520 can be used to store software programs and modules, such as the program instructions / modules corresponding to those in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520, such as taking pictures with the front-facing camera, processing the captured images, and switching the display colors of the content displayed on the screen. The memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 520 may further include memory remotely located relative to the processor 580, and these remote memories can be connected to the electronic device 500 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0101] The input unit 530 can be used to receive input numeric or character information, and to generate a keyboard and mouse related to user settings and function control.

[0102] Display unit 540 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof. Display unit 540 may include display panel 541, which may optionally be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar forms.

[0103] Audio circuitry 560, speaker 561, and microphone 562 provide an audio interface between the user and electronic device 500. Audio circuitry 560 converts received audio data into electrical signals and transmits them to speaker 561, where speaker 561 converts them into sound signals for output. Conversely, microphone 562 converts collected sound signals into electrical signals, which are then received by audio circuitry 560, converted back into audio data, and processed by processor 580. The audio data is then transmitted via RF circuitry 510 to, for example, another terminal, or output to memory 520 for further processing. Audio circuitry 560 may also include an earphone jack to facilitate communication between external headphones and electronic device 500.

[0104] Electronic device 500, through transmission module 570 (e.g., Wi-Fi module), can help users receive requests, send information, etc., providing users with wireless broadband internet access. Although transmission module 570 is shown in the figure, it is understood that it is not an essential component of electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0105] The processor 580 is the control center of the electronic device 500. It connects to various parts of the phone via various interfaces and lines, and performs various functions and processes data of the electronic device 500 by running or executing software programs and / or modules stored in the memory 520, and by calling data stored in the memory 520, thereby providing overall monitoring of the electronic device. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 580.

[0106] Electronic device 500 also includes a power supply 590 (such as a battery) that supplies power to various components. In some embodiments, the power supply may be logically connected to processor 580 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 590 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0107] Although not shown, the electronic device 500 also includes cameras (such as front-facing cameras and rear-facing cameras), Bluetooth modules, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. One or more programs contain instructions for performing the following operations:

[0108] Acquire infrared images and prior pose information of the infrared images;

[0109] The infrared image and the prior pose information are input into the trained neural radiation field model to obtain the rendered RGB image and the depth information of the rendered RGB image.

[0110] Two-dimensional image feature point matching is performed on the infrared image and the rendered RGB image to obtain two-dimensional matching points, and the two-dimensional matching points are converted into three-dimensional spatial points by combining the depth information;

[0111] The rigid body transformation matrix is ​​calculated based on the three-dimensional spatial points to obtain the estimated pose information;

[0112] The estimated pose information is used as the prior pose information to repeat the above-described neural radiation field model processing steps, two-dimensional image feature point matching steps, and steps to obtain the estimated pose information, until the obtained estimated pose information meets the preset requirements. The latest estimated pose information is then used as the final estimated pose information.

[0113] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0114] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of the present invention provide a storage medium storing multiple instructions that can be loaded by a processor to execute the steps of any embodiment of the infrared image positioning and orientation method with reference to a neural radiation field provided by the present invention.

[0115] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0116] Since the instructions stored in the storage medium can execute the steps in any embodiment of the infrared image positioning and attitude determination method with reference to the neural radiation field provided in the embodiments of the present invention, the beneficial effects that any infrared image positioning and attitude determination method with reference to the neural radiation field provided in the embodiments of the present invention can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0117] The foregoing has provided a detailed description of an infrared image positioning and orientation method, apparatus, storage medium, and electronic device with reference to a neural radiation field, as provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for positioning and attitude determination using infrared images with reference to neural radiation fields, characterized in that, The method includes: Acquire infrared images and prior pose information of the infrared images; The infrared image and the prior pose information are input into the trained neural radiation field model to obtain the rendered RGB image and the depth information of the rendered RGB image. Two-dimensional image feature point matching is performed on the infrared image and the rendered RGB image to obtain two-dimensional matching points, and the two-dimensional matching points are converted into three-dimensional spatial points by combining the depth information; The rigid body transformation matrix is ​​calculated based on the three-dimensional spatial points to obtain the estimated pose information; The estimated pose information is used as the prior pose information to repeat the above-described neural radiation field model processing steps, two-dimensional image feature point matching steps, and steps to obtain the estimated pose information, until the obtained estimated pose information meets the preset requirements. The latest estimated pose information is then used as the final estimated pose information.

2. The infrared image positioning and attitude determination method using a neural radiation field as a reference according to claim 1, characterized in that, The neural radiation field model includes two layers of multilayer perceptron, and the prior pose information includes position information, orientation angle and pitch angle; The infrared image and the prior pose information are input into the trained neural radiation field model to obtain the rendered RGB image and the depth information of the rendered RGB image, including: The infrared image and the location information are input into the first layer of the multilayer perceptron to obtain the density of each point and the feature vector of each point; The feature vector, the orientation angle, and the pitch angle are input into the second layer of the multilayer perceptron to obtain the color value of each point; Based on the color value of each point and the volume rendering function, the rendered RGB image is obtained; The depth information of the rendered RGB image is calculated based on the prior pose information.

3. The infrared image positioning and attitude determination method using a neural radiation field as a reference according to claim 2, characterized in that, The volume rendering function is: in, , Indicates along the light ray The RGB color values ​​obtained from the rendering It is the first Volume density at each sampling point It is the first The sampling point and the first +1 distance between sampling points, i.e., sampling interval. Refers to the first Each sampling point is the RGB color value output by the second layer multilayer perceptron.

4. The infrared image positioning and attitude determination method using a neural radiation field as a reference according to claim 1, characterized in that, Two-dimensional image feature point matching is performed on the infrared image and the rendered RGB image to obtain two-dimensional matching points. These two-dimensional matching points are then converted into three-dimensional spatial points using the depth information, including: The infrared image and the rendered RGB image are input into the SuperPoint neural network to obtain the infrared feature coordinate set and infrared feature vector of the infrared image, and the RGB feature coordinate set and RGB feature vector of the rendered RGB image. The infrared feature coordinate set, infrared feature vector, RGB feature coordinate set, and RGB feature vector are input into the SuperGlue neural network to obtain the key point matching matrix and confidence vector.

5. The infrared image positioning and attitude determination method using a neural radiation field as a reference according to claim 1, characterized in that, The estimated pose information is calculated using the following formula: in, Represents the pixel coordinates in the infrared image. Represents the coordinates of a three-dimensional point in space. The desired pose information includes R= 、 Translation vector t = , ( () indicates the camera's internal parameters.

6. The infrared image positioning and attitude determination method using a neural radiation field as a reference according to claim 5, characterized in that, The method further includes: The latest estimated pose information is input into the neural radiation field model to obtain the corresponding rendered RGB image and depth information; Feature point matching is performed on the rendered RGB image and depth information. The angle between the camera and each pair of matched feature points is calculated. When the error of the angle is not greater than the error angle threshold, the pose estimation is confirmed to be accurate. The operation is stopped and the estimated pose information of the current infrared image is used as the final estimated pose information.

7. The infrared image positioning and attitude determination method using a neural radiation field as a reference according to claim 1, characterized in that, The training process of the neural radiation field model includes: Acquire infrared images, prior pose information of the infrared images, and RGB information of light rays in the observation data; The infrared image and the prior pose information are input into the trained neural radiation field model to obtain the rendered RGB image and the depth information of the rendered RGB image. The mean square error between the rendered RGB image and the RGB information of light in the observation data is calculated as a loss function, and the parameters of the neural radiation field model are updated based on the loss function.

8. An infrared image positioning and attitude determination device that uses a neural radiation field as a reference, characterized in that, include: The acquisition module is used to acquire infrared images and the prior pose information of the infrared images; The neural radiation field module is used to input the infrared image and the prior pose information into the trained neural radiation field model to obtain the rendered RGB image and the depth information of the rendered RGB image. The matching and conversion module is used to perform two-dimensional image feature point matching on the infrared image and the rendered RGB image to obtain two-dimensional matching points, and to convert the two-dimensional matching points into three-dimensional spatial points by combining the depth information. The pose estimation module is used to calculate the rigid body transformation matrix based on the three-dimensional space points to obtain the estimated pose information; The iterative module is used to repeat the above-mentioned neural radiation field model processing steps, two-dimensional image feature point matching steps, and steps to obtain estimated pose information, using the estimated pose information as prior pose information, until the obtained estimated pose information meets the preset requirements, and the latest estimated pose information is used as the final estimated pose information.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the infrared image positioning and attitude determination method with reference to a neural radiation field as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, The device includes a processor and a memory, the processor being electrically connected to the memory, the memory being used to store instructions and data, and the processor being used to execute the steps of the infrared image positioning and attitude determination method with reference to a neural radiation field as described in any one of claims 1 to 7.

Images