Visual positioning method, device and equipment, computer readable storage medium and product
By introducing environmental condition characteristics into the visual positioning method and training the conditioned neural radiation field, the problem of low positioning accuracy caused by differences in environmental conditions in the prior art is solved, and a higher visual positioning accuracy is achieved.
Patent Information
- Application Number
- CN202411274601.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-12-31
AI Technical Summary
In the case of large differences in environmental conditions, existing visual positioning methods are prone to mismatch, resulting in low positioning accuracy.
Through scene reconstruction based on scene images, the environmental condition characteristics of the keyframe images are extracted and trained in the neural radiation field network to generate a conditioned neural radiation field for visual positioning. The method includes using a convolutional neural network and an environmental adaptation layer network to extract features, combining a cross attention mechanism module and a fully connected layer, optimizing the camera position to improve positioning accuracy.
By introducing environmental condition characteristics, conditioned neural radiation fields related to environmental conditions are trained, which significantly improves the accuracy of visual positioning, reduces mismatch, and improves positioning accuracy.
Smart Images
Figure CN119228888A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of communication technology, and in particular to a visual positioning method, device, equipment, computer-readable storage medium and product. Background Art
[0002] In the prior art, when visual positioning is performed on a certain frame, the feature points of the frame are first extracted and the feature vectors are retrieved, and several key frames that are most similar to it are retrieved through the prior map; then the 3D point cloud corresponding to the 2D feature points of the frame is calculated through feature matching, and finally the camera pose of the frame is calculated through perspective-n-point (PnP).
[0003] However, when there are large differences in environmental conditions between the query image to be located and the associated frame in the prior map, more mismatches will be obtained, thereby reducing the accuracy of positioning. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide a visual positioning method, device, equipment, computer-readable storage medium and product to solve the problem of low positioning accuracy of existing visual positioning methods.
[0005] In order to solve the above problems, the present application provides a visual positioning method, including:
[0006] Reconstructing the scene based on the collected scene image to obtain a key frame image and a camera pose corresponding to the key frame image;
[0007] Extracting environmental condition features from the key frame image;
[0008] According to the camera pose corresponding to the key frame image and the environmental condition characteristics, a neural radiation field network is trained to obtain a conditional neural radiation field;
[0009] Based on the conditional neural radiation field, a query image is visually localized.
[0010] Wherein, extracting environmental condition features from the key frame image includes:
[0011] Input the key frame image into a convolutional neural network (CNN) backbone network to obtain a feature vector output by the CNN backbone network;
[0012] The feature vector is input into the environment adaptation layer network to obtain the environmental condition feature output by the environment adaptation layer network.
[0013] Wherein, the environment adaptation layer network includes N layers of cross attention mechanism modules; N is an integer greater than 1;
[0014] Inputting the feature vector into the environmental adaptation layer network to obtain the environmental condition features output by the environmental adaptation layer network includes:
[0015] The feature vector is input into the N-layer cross attention mechanism module, cross-calculated with the environmental condition encoding layer by layer, and the environmental condition feature is output.
[0016] Wherein, the neural radiation field network comprises: a first fully connected layer and a second fully connected layer;
[0017] The step of training the neural radiation field network according to the camera pose corresponding to the key frame image and the environmental condition characteristics to obtain a conditional neural radiation field includes:
[0018] Preprocessing the camera pose corresponding to the key frame image to obtain sampling point position information and sampling point direction information;
[0019] Inputting the sampling point position information into the first fully connected layer, the first fully connected layer outputs the color density of each point and other spatial direction information of the point;
[0020] Inputting the sampling point direction information, the environmental condition characteristics and other spatial direction information of the point into the second fully connected layer, and the second fully connected layer outputs the color of each point;
[0021] Aggregating the color density of the points output by the first fully connected layer and the color of the points output by the second fully connected layer to generate a target image;
[0022] The target image and the key frame image are similarly calculated to obtain a loss function, and the neural radiation field network is trained using the loss function to obtain a conditional neural radiation field.
[0023] Wherein, based on the conditional neural radiation field, visual positioning of the query image includes:
[0024] By optimizing the camera pose by gradient, multiple camera poses corresponding to the query image are obtained;
[0025] Inputting a plurality of camera poses into the conditional neural radiance field to generate a plurality of rendered images;
[0026] Calculate the photometric loss of the generated multiple rendering images and the query image;
[0027] The camera pose corresponding to the rendered image with the smallest photometric loss is determined as the visual positioning result of the query image.
[0028] The method of optimizing the camera pose by gradient to obtain multiple camera poses corresponding to the query image includes:
[0029] Randomly or based on the previous visual positioning result, get the initial camera pose corresponding to the query image;
[0030] According to the initialized camera pose of the query image and the gradient optimized camera pose function, a plurality of camera poses corresponding to the query image are determined.
[0031] The present application also provides a visual positioning device, including:
[0032] A reconstruction module, used to reconstruct the scene based on the collected scene image, to obtain a key frame image and a camera pose corresponding to the key frame image;
[0033] An extraction module, used for extracting environmental condition features from the key frame image;
[0034] A training module, used for training a neural radiation field network according to the camera pose corresponding to the key frame image and the environmental condition characteristics to obtain a conditional neural radiation field;
[0035] A positioning module is used to visually locate the query image based on the conditional neural radiation field.
[0036] An embodiment of the present application also provides a visual positioning device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor implements the visual positioning method as described above when executing the program.
[0037] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the visual positioning method as described above are implemented.
[0038] An embodiment of the present application also provides a computer program product, including computer instructions, which implement the steps of the visual positioning method as described above when executed by a processor.
[0039] The above technical solution of the present application has at least the following beneficial effects:
[0040] In the visual positioning method, apparatus, device, computer-readable storage medium and product of the embodiments of the present application, environmental condition features are introduced under the framework of the neural radiation field network, so that conditional neural radiation fields related to environmental conditions can be trained, and visual positioning is performed based on the trained conditional neural radiation fields, thereby improving the accuracy of visual positioning. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 A flowchart showing the steps of the visual positioning method provided in an embodiment of the present application;
[0042] Figure 2 A schematic diagram showing the principle of three-dimensional reconstruction based on colmap in the visual positioning method provided in an embodiment of the present application;
[0043] Figure 3 A schematic diagram showing the principle of implicit reconstruction and visual positioning based on a conditional neural radiation field in a visual positioning method provided in an embodiment of the present application;
[0044] Figure 4 A schematic diagram showing the training of a neural radiation field network in the visual positioning method provided in an embodiment of the present application;
[0045] Figure 5 A schematic diagram showing the structure of a visual positioning device provided in an embodiment of the present application;
[0046] Figure 6 A schematic diagram showing the structure of a visual positioning device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0047] In order to make the technical problems, technical solutions and advantages to be solved by the present application clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.
[0048] like Figure 1 As shown, the embodiment of the present application provides a visual positioning method, including:
[0049] Step 101, performing scene reconstruction based on the collected scene image to obtain a key frame image and a camera pose corresponding to the key frame image;
[0050] Step 102, extracting environmental condition features from the key frame image;
[0051] Step 103, training a neural radiation field network according to the camera pose corresponding to the key frame image and the environmental condition characteristics to obtain a conditional neural radiation field;
[0052] Step 104: Visually locate the query image based on the conditional neural radiation field.
[0053] The embodiment of the present application introduces environmental condition features under the framework of the neural radiation field network, so that a conditional neural radiation field related to the environmental conditions can be trained, and visual positioning can be performed based on the trained conditional neural radiation field, thereby improving the accuracy of visual positioning.
[0054] As an optional embodiment, step 101 includes: reconstructing the scene based on the collected scene image and colmap (open source visual 3D reconstruction software), and outputting key frame images and corresponding camera poses. For example, the camera pose includes: the position T and direction R of the camera.
[0055] For example, Figure 2 As shown, the 3D reconstruction based on colmap includes:
[0056] 1. Data collection: Use cameras and other devices to collect images of the scene, that is, scene images.
[0057] 2. Feature extraction: Extract feature points from the image, such as the scale-invariant feature transform algorithm SIFT or the accelerated robust feature algorithm SURF.
[0058] 3. Feature matching: Match the feature points between different images to calculate their relative position and posture.
[0059] 4. Sparse reconstruction: Use matched feature points to calculate the 3D shape and position of objects in the scene.
[0060] Depth estimation: For each pixel, its depth estimate in the scene is calculated by triangulation. Specifically, assuming the camera position is known, for a given pixel, its 3D position in the scene is calculated by matching its corresponding points in different images. Then, the depth estimate of the pixel is obtained by calculating the distance between the camera position and the pixel position.
[0061] 5. Dense reconstruction: For each pixel, its depth value in the scene is calculated through stereo matching. Specifically, for each pair of adjacent images, the depth of each pixel in the scene is estimated by calculating the disparity between them, that is, the horizontal offset of the corresponding pixels in the left and right images. These depth values can form a dense depth map.
[0062] Through the above steps, the scene can be reconstructed to obtain an explicit map, that is, the key frame image and its corresponding camera pose T, 2D feature points and 3D points.
[0063] In at least one embodiment of the present application, steps 102 to 104 may be referred to as implicit reconstruction and visual positioning based on the conditional neural radiation field; implicit reconstruction and visual positioning based on the conditional neural radiation field may be divided into two major steps:
[0064] A: Corresponding to step 102 and step 103, the conditional neural radiation field is trained based on the data output from step 101. The scene is implicitly reconstructed into the neural network parameters, that is, an environmental condition branch is added to the neural radiation field network framework, and the conditional neural radiation field is trained based on the data output from step 101 and the environmental condition characteristics;
[0065] B: Corresponding to step 104, visual positioning is performed based on the trained conditional neural radiation field.
[0066] In at least one embodiment of the present application, step 102 includes:
[0067] Input the key frame image into a convolutional neural network (CNN) backbone network to obtain a feature vector output by the CNN backbone network;
[0068] The feature vector is input into the environment adaptation layer network to obtain the environmental condition feature output by the environment adaptation layer network.
[0069] The embodiment of the present application adds a CNN backbone network and an environmental adaptation layer network to the neural radiation field network to achieve visual positioning based on environmental factors and improve the accuracy of visual positioning.
[0070] Optionally, the environment adaptation layer network includes N layers of cross attention mechanism modules; N is an integer greater than 1;
[0071] Inputting the feature vector into the environmental adaptation layer network to obtain the environmental condition features output by the environmental adaptation layer network includes:
[0072] The feature vector is input into the N-layer cross attention mechanism module, cross-calculated with the environmental condition encoding layer by layer, and the environmental condition feature is output; N is an integer greater than 1.
[0073] like Figure 3 and Figure 4 As shown, in order to extract environmental condition features from the image to constrain the point color, the embodiment of the present application introduces an environmental adaptation layer network. The environmental adaptation layer network introduces an N-layer cross attention mechanism module, and the environmental condition potential space is represented as S = {S v}, there are V kinds of environmental conditions, each of which corresponds to coding, is the parameter in the tth module.
[0074]
[0075] That is, the extraction process of environmental condition features is as follows: the key frame image is firstly extracted through the CNN backbone network to extract the feature vector f i 1 , and then passed to the N-layer cross attention mechanism module, layer by layer with the environmental condition encoding Cross calculation, the final output is f i N+1 As a characteristic of environmental conditions.
[0076] As an optional embodiment, the neural radiation field network includes: a first fully connected layer and a second fully connected layer; accordingly, step 103 includes:
[0077] Preprocessing the camera pose corresponding to the key frame image to obtain sampling point position information and sampling point direction information;
[0078] Inputting the sampling point position information into a first fully connected layer, the first fully connected layer outputs the color density of each point and other spatial direction information of the point;
[0079] Inputting the sampling point direction information, the environmental condition characteristics and other spatial direction information of the point into a second fully connected layer, and the second fully connected layer outputs the color of each point;
[0080] Aggregating the color density of the points output by the first fully connected layer and the color of the points output by the second fully connected layer to generate a target image;
[0081] The target image and the key frame image are similarly calculated to obtain a loss function, and the neural radiation field network is trained using the loss function to obtain a conditional neural radiation field.
[0082] The loss function is used to train the neural radiation field network to obtain the conditional neural radiation field, including:
[0083] By minimizing the loss function and back propagation, the parameters of the neural radiation field network are updated to obtain a conditional neural radiation field.
[0084] When performing visual positioning, the embodiment of the present application adds reverse transmission to the neural radiation field network, and obtains the photometric loss of the generated image and the query image of the conditional neural radiation field network under different camera postures in a gradient-optimized manner. The camera posture corresponding to the minimum photometric loss is used as the positioning result corresponding to the query image, thereby further improving the accuracy of visual positioning.
[0085] For example, the colmap reconstruction result (i.e., the key frame image and the camera pose corresponding to the key frame image) is preprocessed, and the spatial direction information data (x, y, z, θ, φ, r, g, b, σ) of the pixel points are sampled based on the position and direction of the image. x, y, z represent the viewing angle position, θ, φ represent the observation direction, and x, y, z, θ, φ represent the observation direction. A ray is determined, which together represents the camera pose; r, g, b represent the point color observed under the ray, and σ represents the point color density.
[0086] like Figure 4 As shown, the neural radiation field network is a backbone structure composed of a fully connected layer MLP (the basic structure in the neural network). The embodiment of the present application adds an environmental condition branch to the framework of the neural radiation field network, and obtains the following Figure 4 The structural diagram of the neural radiation field network is shown in Figure 2. Figure 4 As shown, the process of training the neural radiation field network is as follows:
[0087] According to the data output by colmap, the camera pose (point position and direction) and key frame image of the sampling point are obtained. The sampling point position xyz and the sampling point direction θ, φ after sampling are input into the fully connected layer MLP (i.e., the first fully connected layer). The first fully connected layer outputs the point color r, g, b and point color density σ of each sampling point. The environmental condition branch is added to the neural radiation field network framework to train the real image (when the neural radiation field network is trained). Figure 4 The real image in corresponds to the key frame image. When visual positioning is performed Figure 4 The real image in the image corresponds to the query image) and the CNN backbone network is used to extract features, and then the environmental condition features are passed to the environmental adaptation layer network to obtain the environmental condition features, and the environmental condition features are output to the fully connected layer MLP (i.e., the second fully connected layer) to constrain the point color rgb. Finally, the point color density σ output by the first fully connected layer and the point color rgb output by the second fully connected layer are aggregated to generate an image (i.e., the target image), and the generated target image is compared with the real image to calculate the loss function (where r is the sampling point position and direction, C(r) represents the true value of the color, (r) represents the network output color estimation value), and the network parameters are optimized through back propagation to obtain the trained conditional neural radiation field.
[0088] In at least one embodiment of the present application, step 104 includes:
[0089] By optimizing the camera pose by gradient, multiple camera poses corresponding to the query image are obtained;
[0090] Inputting a plurality of camera poses into the conditional neural radiance field to generate a plurality of rendered images;
[0091] Calculate the photometric loss of the generated multiple rendering images and the query image;
[0092] The camera pose corresponding to the rendered image with the smallest photometric loss is determined as the visual positioning result of the query image.
[0093] The method of optimizing the camera pose by gradient to obtain multiple camera poses corresponding to the query image includes:
[0094] Randomly or based on the previous visual positioning result, get the initial camera pose corresponding to the query image;
[0095] According to the initialized camera pose of the query image and the gradient optimized camera pose function, a plurality of camera poses corresponding to the query image are determined.
[0096] For example, first determine the initial camera pose T (image position and orientation) of the query image randomly or based on the previous positioning result, and pass the query image to the real image input layer. Optimize the camera pose T based on the gradient, and the optimization function is where T is the camera pose matrix, I is the query image, Θ is the network parameter of the trained conditional neural radiance field, SE(3) is the Lie group in Lie algebra, and the loss function is
[0097] That is, through iteration, the camera pose T is continuously optimized by gradient. Figure 3 The query image is kept unchanged, and different camera poses T are input into the trained conditional neural radiance field to generate the target image. The photometric loss between each target image and the query image is calculated to obtain the camera pose that minimizes the photometric loss between the target image generated by the conditional neural radiance field and the query image. The camera pose is the positioning result corresponding to the query image.
[0098] In summary, the embodiment of the present application first reconstructs the scene based on the scene image, and outputs the key frame image and its corresponding camera pose (i.e., position T and direction R). Then, based on the data output by the scene reconstruction, implicit reconstruction based on the conditional neural radiation field is implemented, wherein the conditional neural radiation field is composed of a CNN backbone network, an environmental adaptation layer network, and a neural radiation field network. Finally, based on the trained conditional neural radiation field, the rendering image is generated through iterative calculation to minimize the loss of the query image and return the positioning result (i.e., position and direction), thereby improving the accuracy of visual positioning.
[0099] like Figure 5 As shown, the embodiment of the present application also provides a visual positioning device, including:
[0100] A reconstruction module 501 is used to perform scene reconstruction based on the collected scene image to obtain a key frame image and a camera pose corresponding to the key frame image;
[0101] An extraction module 502, used to extract environmental condition features from the key frame image;
[0102] A training module 503 is used to train a neural radiation field network according to the camera pose corresponding to the key frame image and the environmental condition characteristics to obtain a conditional neural radiation field;
[0103] The positioning module 504 is used to perform visual positioning on the query image based on the conditional neural radiation field.
[0104] As an optional embodiment, the extraction module includes:
[0105] The first acquisition submodule is used to input the key frame image into a convolutional neural network (CNN) backbone network to obtain a feature vector output by the CNN backbone network;
[0106] The second acquisition submodule is used to input the feature vector into the environmental adaptation layer network to obtain the environmental condition features output by the environmental adaptation layer network.
[0107] As an optional embodiment, the environment adaptation layer network includes N layers of cross attention mechanism modules; N is an integer greater than 1;
[0108] The second acquisition submodule includes:
[0109] An acquisition unit is used to input the feature vector into the N-layer cross-attention mechanism module, cross-calculate with the environmental condition encoding layer by layer, and output the environmental condition feature.
[0110] As an optional embodiment, the neural radiation field network includes: a first fully connected layer and a second fully connected layer; the training module includes:
[0111] A first processing submodule is used to pre-process the camera posture corresponding to the key frame image to obtain sampling point position information and sampling point direction information;
[0112] A second processing submodule, configured to input the sampling point position information into a first fully connected layer, and the first fully connected layer outputs the color density of each point and other spatial direction information of the point;
[0113] A third processing submodule, configured to input the sampling point direction information, the environmental condition characteristics, and other spatial direction information of the point into a second fully connected layer, and the second fully connected layer outputs the color of each point;
[0114] a fourth processing submodule, configured to aggregate the color density of the points output by the first fully connected layer and the color of the points output by the second fully connected layer to generate a target image;
[0115] The training submodule is used to calculate the similarity between the target image and the key frame image to obtain a loss function, and use the loss function to train the neural radiation field network to obtain a conditional neural radiation field.
[0116] As an optional embodiment, the training submodule includes:
[0117] The training unit is used to update the parameters of the neural radiation field network by minimizing the loss function and back propagation to obtain a conditional neural radiation field.
[0118] As an optional embodiment, the positioning module includes:
[0119] The first submodule is used to obtain multiple camera poses corresponding to the query image in a manner of gradient optimization of the camera pose;
[0120] A second submodule is used to input multiple camera poses into the conditional neural radiation field to generate multiple rendered images;
[0121] A third submodule is used to calculate the photometric loss of the generated multiple rendering images and the query image;
[0122] The fourth submodule is used to determine the camera pose corresponding to the rendered image with the smallest photometric loss as the visual positioning result of the query image.
[0123] As an optional embodiment, the first submodule includes:
[0124] A first determination unit is used to obtain an initialization camera pose corresponding to a query image randomly or based on a previous visual positioning result;
[0125] The second determining unit is used to determine multiple camera poses corresponding to the query image according to the initialized camera pose of the query image and the gradient optimized camera pose function.
[0126] The embodiment of the present application first reconstructs the scene based on the scene image, and outputs the key frame image and its corresponding camera pose (i.e., position T and direction R). Then, based on the data output by the scene reconstruction, implicit reconstruction based on the conditional neural radiation field is implemented, wherein the conditional neural radiation field is composed of a CNN backbone network, an environmental adaptation layer network, and a neural radiation field network. Finally, based on the trained conditional neural radiation field, the rendering image is generated through iterative calculation to minimize the loss of the query image and return the positioning result (i.e., position and direction), thereby improving the accuracy of visual positioning.
[0127] It should be noted that the visual positioning device provided in the embodiment of the present application is a device capable of executing the above-mentioned visual positioning method. All embodiments of the above-mentioned visual positioning method are applicable to the device and can achieve the same or similar beneficial effects, which will not be repeated here.
[0128] like Figure 6 As shown, an embodiment of the present application also provides a visual positioning device, including a memory 610, a processor 600, and a computer program stored in the memory 610 and executable on the processor 600. When the processor 600 executes the program, each process in the visual positioning method embodiment as described above is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0129] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, each process in the visual positioning method embodiment described above is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0130] The embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the visual positioning method embodiment described above are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.
[0131] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.
[0132] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A device that specifies functions in one or more processes and / or one or more blocks.
[0133] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable storage medium produce a paper product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0134] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that the computer or other programmable device executes a series of operating steps to produce a computer-implemented process, thereby providing instructions executed on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0135] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A visual positioning method, characterized in that: include: Reconstructing the scene based on the collected scene image to obtain a key frame image and a camera pose corresponding to the key frame image; Extracting environmental condition features from the key frame image; According to the camera pose corresponding to the key frame image and the environmental condition characteristics, a neural radiation field network is trained to obtain a conditional neural radiation field; Based on the conditional neural radiation field, a query image is visually localized.
2. The method according to claim 1, characterized in that The step of extracting environmental condition features from the key frame image comprises: Input the key frame image into a convolutional neural network (CNN) backbone network to obtain a feature vector output by the CNN backbone network; The feature vector is input into the environment adaptation layer network to obtain the environmental condition feature output by the environment adaptation layer network.
3. The method according to claim 2, characterized in that The environment adaptation layer network includes N layers of cross attention mechanism modules; N is an integer greater than 1; Inputting the feature vector into the environmental adaptation layer network to obtain the environmental condition features output by the environmental adaptation layer network includes: The feature vector is input into the N-layer cross attention mechanism module, cross-calculated with the environmental condition encoding layer by layer, and the environmental condition feature is output.
4. The method according to any one of claims 1 to 3, characterized in that: The neural radiation field network includes: a first fully connected layer and a second fully connected layer; The step of training the neural radiation field network according to the camera pose corresponding to the key frame image and the environmental condition characteristics to obtain a conditional neural radiation field includes: Preprocessing the camera pose corresponding to the key frame image to obtain sampling point position information and sampling point direction information; Inputting the sampling point position information into the first fully connected layer, the first fully connected layer outputs the color density of each point and other spatial direction information of the point; Inputting the sampling point direction information, the environmental condition characteristics and other spatial direction information of the point into the second fully connected layer, and the second fully connected layer outputs the color of each point; Aggregating the color density of the points output by the first fully connected layer and the color of the points output by the second fully connected layer to generate a target image; The target image and the key frame image are similarly calculated to obtain a loss function, and the neural radiation field network is trained using the loss function to obtain a conditional neural radiation field.
5. The method according to claim 1, characterized in that Based on the conditional neural radiation field, visual positioning of the query image is performed, including: By optimizing the camera pose by gradient, multiple camera poses corresponding to the query image are obtained; Inputting a plurality of camera poses into the conditional neural radiance field to generate a plurality of rendered images; Calculate the photometric loss of the generated multiple rendering images and the query image; The camera pose corresponding to the rendered image with the smallest photometric loss is determined as the visual positioning result of the query image.
6. The method according to claim 5, characterized in that The method of optimizing the camera pose by gradient to obtain multiple camera poses corresponding to the query image includes: Randomly or based on the previous visual positioning result, get the initial camera pose corresponding to the query image; According to the initialized camera pose of the query image and the gradient optimized camera pose function, a plurality of camera poses corresponding to the query image are determined.
7. A visual positioning device, characterized in that: include: A reconstruction module, used to reconstruct the scene based on the collected scene image, to obtain a key frame image and a camera pose corresponding to the key frame image; An extraction module, used for extracting environmental condition features from the key frame image; A training module, used for training a neural radiation field network according to the camera pose corresponding to the key frame image and the environmental condition characteristics to obtain a conditional neural radiation field; A positioning module is used to visually locate the query image based on the conditional neural radiation field.
8. A visual positioning device, comprising a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that: When the processor executes the program, the visual positioning method as described in any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the visual positioning method as described in any one of claims 1 to 6 are implemented.
10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the visual positioning method according to any one of claims 1 to 6.