Training Method, Localization Method and Device of Scene Coordinate Regression Network

Abstract

An embodiment of the present application provides a training method, a positioning method, and a device thereof for a scene coordinate regression network. The method includes: pre-training K sub-networks of an ACE network according to a set of scene images, and generating a first training set according to the set of scene images and a feature extraction network; training a target ACE network using the first training set, where the target ACE network includes a gating network and K sub-networks, the gating network is used to determine the weights of training samples assigned to each sub-network according to the image features in the training samples, and the output result of the target ACE network is a fused scene coordinate obtained by weighting the weights corresponding to the K sub-networks and the output scene coordinates. In this method, the gating network can learn global spatial partition features based on the input local features, select an appropriate sub-network based on the input local features and the global spatial partition features, improve the accuracy of sub-network selection, and further improve the accuracy of scene coordinate prediction and the positioning accuracy of the camera pose.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a training method, localization method and apparatus for a scene coordinate regression network. Background Technology

[0002] Visual positioning technology refers to the technology of determining the pose of a camera (or other target) in three-dimensional space by combining image information collected by visual sensors such as cameras with the spatial features of the scene (such as texture, geometric structure, and prior maps). It is a core supporting technology in fields such as robot navigation, augmented reality (AR), and autonomous driving.

[0003] Scene Coordinate Regression (SCR) is a commonly used visual localization method. SCR implicitly encodes scene information through neural networks. These networks implicitly learn the correspondence between 2D and 3D elements (i.e., 2D pixels in an image and 3D points in a map). Based on this implicit correspondence, the 3D coordinates of a 2D pixel within the 3D scene can be predicted. Using multiple pairs of 2D-3D matching points, the Perspective-n-Point (PnP) algorithm is employed to calculate the camera pose. Accelerated Coordinate Encoding (ACE) further revolutionizes the training paradigm of SCR, completing scene encoding in just 5 minutes and completely bypassing explicit 3D reconstruction. However, ACE faces a core limitation: the capacity of a single network is insufficient for large scenes. To address this issue, the ACE method proposes a "scene clustering and segmentation + multi-sub-network training" strategy for large scenes: the scene is divided into multiple sub-regions according to the camera pose and a sub-network is trained for each sub-region. During inference, multiple sub-networks are run in parallel and the output results are selected to improve accuracy.

[0004] However, when clustering large scenes based on camera pose, unreasonable partition boundaries may occur. Such improper clustering can lead to blurred feature representations, thereby reducing positioning accuracy. Summary of the Invention

[0005] This application provides a training method, a localization method, and an apparatus for a scene coordinate regression network, which can improve the localization accuracy of camera pose.

[0006] In a first aspect, embodiments of this application provide a training method for a scene coordinate regression network, the method comprising:

[0007] The scene image set is divided into K sub-image sets according to the camera pose of the images, where K is an integer greater than or equal to 2;

[0008] Using the K sub-image sets, K sub-networks of the ACE network are pre-trained respectively, and the sub-networks are used to predict the scene coordinates of pixels based on the input image features;

[0009] A first training set is generated based on the scene image set and the feature extraction network, and the first training set includes multiple first training samples;

[0010] The target ACE network is trained using the first training set. The target ACE network includes the feature extraction network and a hybrid expert module. The hybrid expert module includes a gating network and K expert units, which are the K sub-networks. The gating network is used to determine the probability of a training sample being assigned to each sub-network based on the image features in the training samples, and to determine the weight corresponding to each sub-network based on the probability. The output of the target ACE network is a fused scene coordinate obtained by weighting the scene coordinates output by the K sub-networks and the weights corresponding to the K sub-networks.

[0011] In some exemplary embodiments, before training the target ACE network using the first training set, the method further includes:

[0012] A second training set is generated based on the scene image set and the feature extraction network, and the second training set includes multiple second training samples.

[0013] With the parameters of the K sub-networks fixed, the gating network of the target ACE network is pre-trained using the second training set.

[0014] In some exemplary embodiments, the gating network is used to determine the probability of a training sample being assigned to each sub-network based on image features in the training samples, to correct the probability of each sub-network based on the learnable bias term of each sub-network, and to determine the weight corresponding to each sub-network based on the corrected probability.

[0015] In some exemplary embodiments, the first training sample includes the pixel coordinates of a pixel, the image features of the image block to which the pixel is located, and the camera pose of the image to which the pixel belongs;

[0016] The step of training the target ACE network using the first training set includes:

[0017] The image features in the first training sample are input into the gating network, and the probability of the first training sample being assigned to each sub-network is determined by the gating network.

[0018] The probability of each subnetwork is adjusted based on the bias term of each subnetwork.

[0019] Based on the corrected probabilities of the K sub-networks, determine the weights corresponding to each sub-network;

[0020] The first training sample is input into each sub-network to obtain the output of each sub-network, and the output of each sub-network is the scene coordinate of the pixel.

[0021] The fused scene coordinates of the pixel are obtained by weighting the weights corresponding to the K sub-networks and the scene coordinates output by the K sub-networks respectively.

[0022] For each subnetwork, the long-term utilization rate of the subnetwork is determined according to the weights corresponding to the subnetwork, and the bias term of the subnetwork is updated according to the long-term utilization rate and the target utilization rate.

[0023] For each subnetwork, the reprojection loss of the subnetwork is determined based on the pixel coordinates of the pixel, the fused scene coordinates of the pixel, and the camera pose of the image to which the pixel belongs. Based on the reprojection loss of the subnetwork, the parameters of the subnetwork and the parameters of the gated network are updated.

[0024] In some exemplary embodiments, the second training sample includes the pixel coordinates of the pixel, the image features of the image block where the pixel is located, and the camera pose of the image to which the pixel belongs;

[0025] The step of fixing the parameters of the K sub-networks and pre-training the gated network of the target ACE network using the second training set includes:

[0026] The image features in the second training sample are input into the gating network, and the probability of the second training sample being assigned to each sub-network is determined by the gating network.

[0027] The probability of each subnetwork is adjusted based on the bias term of each subnetwork.

[0028] Based on the corrected probabilities of the K sub-networks, determine the weights corresponding to each sub-network;

[0029] The second training sample is input into each sub-network to obtain the output of each sub-network, and the output of each sub-network is the scene coordinate of the pixel.

[0030] The fused scene coordinates of the pixel are obtained by weighting the weights corresponding to the K sub-networks and the scene coordinates output by the K sub-networks respectively.

[0031] For each subnetwork, the long-term utilization rate of the subnetwork is determined according to the weights corresponding to the subnetwork, and the bias term of the subnetwork is updated according to the long-term utilization rate and the target utilization rate.

[0032] For each subnetwork, the reprojection loss of the subnetwork is determined based on the pixel coordinates of the pixel, the fused scene coordinates of the pixel, and the camera pose of the image to which the pixel belongs. The parameters of the gated network are then updated based on the reprojection loss of the subnetwork.

[0033] In some exemplary embodiments, determining the weight corresponding to each sub-network based on the corrected probabilities of the K sub-networks includes:

[0034] Generate Gumbel noise for each subnetwork;

[0035] Add the Gumbel noise of each subnetwork to the corrected probability of the corresponding subnetwork to obtain the target probability of each subnetwork.

[0036] Based on the target probabilities of the K sub-networks, the weights corresponding to each sub-network are calculated using the Softmax function.

[0037] In some exemplary embodiments, the step of calculating the weights corresponding to each sub-network using the Softmax function based on the target probabilities of the K sub-networks includes:

[0038] The weights for each subnetwork are calculated using the following formula:

[0039]

[0040] Among them, w i,j This represents the weights assigned to the j-th subnetwork by the i-th training sample. G represents the corrected probability that the i-th training sample is assigned to the j-th subnetwork. i,j express The corresponding Gumbel noise, where τ is the temperature parameter.

[0041] In some exemplary embodiments, determining the long-term utilization rate of the sub-network based on the weights corresponding to the sub-network includes:

[0042] The long-term utilization rate of the sub-network is calculated using the following formula:

[0043]

[0044] Where α is the momentum factor and N is the number of samples in the training set. This represents the exponential moving average utilization of the j-th network at the t-th iteration. This represents the weight assigned to the j-th subnetwork by the i-th training sample during the t-th iteration.

[0045] In some exemplary embodiments, the target utilization rate is taken as 1 / K;

[0046] The step of updating the bias term of the sub-network based on the long-term utilization rate and target utilization rate of the sub-network includes:

[0047] Calculate the difference between the long-term utilization rate and the target utilization rate of the sub-network;

[0048] Multiply the usage difference by the negative of the learning rate factor of the bias term of the sub-network to obtain the updated value of the bias term of the sub-network.

[0049] The updated bias term of the sub-network is obtained by adding the bias term of the sub-network to the updated bias term of the sub-network.

[0050] In some exemplary embodiments, generating the first training set or the second training set based on the scene image set and the feature extraction network includes:

[0051] For each scene image in the scene image set, the scene image is divided into M image blocks;

[0052] The feature extraction network extracts the image features of each of the M image patches;

[0053] M training samples are generated based on the image features of the M image blocks, wherein each training sample includes the image features of an image block, the pixel coordinates of the pixels in the image block, and the camera pose of the scene image to which the image block belongs;

[0054] Samples are collected from the complete sample set generated from the scene image set using a randomization method to form the training set for the current batch.

[0055] In some exemplary embodiments, the step of using the K sub-image sets to pre-train K sub-networks of the Accelerated Coordinate Encoding (ACE) network includes:

[0056] For each sub-image set, a corresponding sub-training set is generated based on the sub-image set and the feature extraction network, and the sub-training set includes multiple third training samples;

[0057] The sub-network is pre-trained using the sub-training set corresponding to the sub-image set.

[0058] In some exemplary embodiments, the third training sample includes the pixel coordinates of the pixel, the image features of the image block where the pixel is located, and the camera pose of the image to which the pixel belongs;

[0059] The step of pre-training the sub-network corresponding to the sub-image set using the sub-training set includes:

[0060] The third training sample is input into the sub-network to predict the scene coordinates of the pixel.

[0061] The reprojection loss of the sub-network is determined based on the pixel coordinates, scene coordinates, and camera pose of the pixel.

[0062] The parameters of the subnetwork are updated based on the reprojection loss of the subnetwork.

[0063] Secondly, embodiments of this application provide a localization method using a scene coordinate regression network, the method comprising:

[0064] Obtain the image to be located;

[0065] Using the image to be located and the target ACE network trained by the method described in the first aspect, the scene coordinates of M pixels in the image to be located are determined in the scene. The target ACE network includes a feature extraction network and a hybrid expert module. The hybrid expert module includes a gating network and K expert units, and each expert unit is a sub-network.

[0066] The feature extraction network is used to divide the image to be located into M image blocks and extract the image features of each image block respectively.

[0067] The gated network is used to determine the target sub-network for each input information from K sub-networks based on the image features of each input information in M ​​input information, and input each input information into the corresponding target sub-network. The output of the target sub-network includes the scene coordinates of the pixels in the input information in the scene. Each image block corresponds to one input information, and each input information includes the image features of an image block and the pixel coordinates of the pixels in the image block.

[0068] The camera pose of the image to be located is determined based on the pixel coordinates of the M pixels and the scene coordinates.

[0069] In some exemplary embodiments, determining the camera pose of the image to be located based on the pixel coordinates of the M pixels and the scene coordinates includes:

[0070] Abnormal pixels among the M pixels are removed using a random sampling consistency algorithm;

[0071] Based on the pixel coordinates of the remaining pixels and the scene coordinates, the initial camera pose of the image to be located is estimated using the n-point perspective PnP algorithm.

[0072] The initial camera pose of the image to be located is optimized to obtain the camera pose of the image to be located.

[0073] In some exemplary embodiments, determining the scene coordinates of M pixels in the image to be located within the scene using the image to be located and the target ACE network trained by the method described in the first aspect includes:

[0074] The image to be located is divided into M image blocks;

[0075] The feature extraction network extracts the image features of each of the M image patches;

[0076] M input information is generated based on the image features of the M image blocks;

[0077] For each input information, the image features in the input information are input into the gating network, and the probability of the input information being assigned to each sub-network is determined by the gating network.

[0078] The probability of each subnetwork is adjusted based on the bias term of each subnetwork.

[0079] Based on the corrected probabilities of the K sub-networks, determine the weights corresponding to each sub-network;

[0080] Based on the weights corresponding to the K sub-networks, the sub-network with the largest weight is determined as the target sub-network used for the input information;

[0081] The target sub-network processes the input information to obtain the scene coordinates of the pixels in the input information within the scene.

[0082] Thirdly, embodiments of this application provide a training apparatus for a scene coordinate regression network, the apparatus comprising:

[0083] The first training module is used to divide the scene image set into K sub-image sets according to the camera pose of the images, and use the K sub-image sets to pre-train K sub-networks of the Accelerated Coordinate Encoding (ACE) network. The sub-networks are used to predict the scene coordinates of pixels based on the input image features, where K is an integer greater than or equal to 2.

[0084] The second training module is used to generate a first training set based on the scene image set and the feature extraction network. The first training set includes multiple first training samples. The first training set is used to train the target ACE network. The target ACE network includes the feature extraction network and a hybrid expert module. The hybrid expert module includes the gating network and K expert units. The K expert units are the K sub-networks. The gating network is used to determine the probability of a training sample being assigned to each sub-network based on the image features in the training samples, and to determine the weight corresponding to each sub-network based on the probability. The output of the target ACE network is a fused scene coordinate obtained by weighting the scene coordinates output by the K sub-networks and the weights corresponding to the K sub-networks respectively.

[0085] In some exemplary embodiments, the apparatus further includes:

[0086] The third training module is used to generate a second training set based on the scene image set and the feature extraction network before the second training module trains the target ACE network. The second training set includes multiple second training samples. The parameters of the K sub-networks are fixed, and the gating network of the target ACE network is pre-trained using the second training set.

[0087] In some exemplary embodiments, the gating network is used to determine the probability of a training sample being assigned to each sub-network based on image features in the training samples, to correct the probability of each sub-network based on the learnable bias term of each sub-network, and to determine the weight corresponding to each sub-network based on the corrected probability.

[0088] Fourthly, embodiments of this application provide a localization device for a scene coordinate regression network, the device comprising:

[0089] The acquisition module is used to acquire the image to be located.

[0090] The scene coordinate determination module is used to determine the scene coordinates of M pixels in the image to be located within the scene using the image to be located and the target ACE network trained by the method described in the first aspect. The target ACE network includes a feature extraction network and a hybrid expert module. The hybrid expert module includes a gating network and K expert units, each expert unit being a sub-network. The feature extraction network is used to divide the image to be located into M image blocks and extract image features from each image block. The gating network is used to determine the target sub-network for each input information from the K sub-networks based on the image features of each of the M input information, and input each input information into the corresponding target sub-network. The output of the target sub-network includes the scene coordinates of the pixels in the input information within the scene. Each image block corresponds to one input information, and each input information includes the image features of an image block and the pixel coordinates of the pixels in the image block.

[0091] The pose determination module is used to determine the camera pose of the image to be located based on the pixel coordinates of the M pixels and the scene coordinates.

[0092] Fifthly, embodiments of this application provide an electronic device, the electronic device comprising: a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the method as described in the first or second aspect above.

[0093] Sixthly, embodiments of this application provide a computer-readable storage medium for storing a computer program that causes a computer to perform the methods described in the first or second aspect above.

[0094] In a seventh aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first or second aspect above.

[0095] This application provides a training method, localization method, and apparatus for a scene coordinate regression network. First, K sub-networks of an ACE network are pre-trained based on a scene image set. Then, a first training set is generated based on the scene image set and a feature extraction network, including multiple first training samples. The first training set is used to train a target ACE network, which includes a feature extraction network and a hybrid expert module. The hybrid expert module includes a gating network and K expert units, where the K expert units are the pre-trained K sub-networks. The gating network determines the probability of assigning a training sample to each sub-network based on image features in the training samples, and determines the weight corresponding to each sub-network based on this probability. The output of the target ACE network is a fused scene coordinate obtained by weighting the scene coordinates output by the K sub-networks and their corresponding weights. In this method, the gating network can learn global spatial partitioning features from the input local features. Based on the input local features and the global spatial partitioning features, it selects suitable sub-networks for the input local features, improving the accuracy of sub-network selection and thus improving the accuracy of scene coordinate prediction and camera pose localization. Attached Figure Description

[0096] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0097] Figure 1 This is a flowchart of the training method for the scene coordinate regression network provided in Embodiment 1 of this application;

[0098] Figure 2 This is a schematic diagram illustrating the training of the K subnetworks of the ACE network;

[0099] Figure 3 A schematic diagram of the training architecture for the target ACE network;

[0100] Figure 4 A flowchart illustrating the training method for the scene coordinate regression network provided in Embodiment 2 of this application;

[0101] Figure 5 A flowchart illustrating the training method for the scene coordinate regression network provided in Embodiment 3 of this application;

[0102] Figure 6 A flowchart illustrating the localization method of the scene coordinate regression network provided in Embodiment 4 of this application;

[0103] Figure 7This is a schematic diagram of the training device for the scene coordinate regression network provided in Embodiment 5 of this application;

[0104] Figure 8 This is a schematic diagram of the positioning device for the scene coordinate regression network provided in Embodiment Six of this application;

[0105] Figure 9 This is a schematic diagram of the structure of an electronic device provided in Embodiment 7 of this application. Detailed Implementation

[0106] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0107] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0108] To facilitate understanding of the embodiments of this application, before describing the various embodiments, some concepts involved in all embodiments of this application will be appropriately explained as follows:

[0109] Visual positioning technology refers to determining the pose of a camera (or target object) in three-dimensional space by combining image information collected by visual sensors such as cameras with the spatial features of a three-dimensional scene (such as texture, geometric structure, and prior maps).

[0110] In this embodiment of the application, the camera pose includes the position and orientation of the camera. The camera pose can be a 6DoF pose (3D position and 3 rotation angles).

[0111] The main principle of visual positioning is: first, a high-precision map of the scene is constructed; during positioning, the real-time image is matched with the pre-built map to find the 2D-3D (i.e., 2D pixels in the image and 3D points in the map) correspondence; and based on multiple pairs of 2D-3D matching points, the PnP algorithm is used to calculate the camera pose.

[0112] Scene coordinate regression is a visual localization technique that uses deep neural networks to directly regress the 2D-3D correspondences used for camera pose estimation. It encodes map information into the neural network, directly regressing 3D coordinates in the world coordinate system to their corresponding image pixels, thus eliminating the need for explicit map construction and explicit 2D-3D correspondence matching. Compared to traditional visual localization methods based on feature matching (FM), scene coordinate regression achieves faster relocalization and requires a smaller map size.

[0113] Scene regression coordinate methods can be applied to map relocalization, which refers to the process of determining the position and angle of a robot or camera using an existing map when the robot's or camera's pose is unknown. In Simultaneous Localization and Mapping (SLAM) technology, relocalization refers to the process of recovering the camera's pose in the world coordinate system using an already constructed map after the SLAM system loses its estimation of the camera pose due to tracking loss (such as rapid movement, occlusion, or weakly textured scenes). It is a key capability for the SLAM system to recover from "tracking failure" to "normal localization," directly determining the system's robustness (e.g., whether a robot can find its way back after getting lost).

[0114] Traditional scene coordinate regression methods suffer from excessively long training times, requiring hours or even days for each new scene. They also demand large storage spaces and heavily rely on implicit triangulation, often facing challenges when dealing with repetitive textures and meaningless regions. Furthermore, during relocalization, the reliability of specific regions is unknown, leading to low efficiency as all pixel-to-3D point matching is used for pose estimation. These limitations restrict the application of traditional scene coordinate regression methods. Therefore, Accelerated Coordinate Encoding (ACE) networks were developed to address these challenges.

[0115] The ACE technique divides the traditional scene coordinate regression network into a scene-independent feature backbone and a scene-specific prediction head. This scene-specific prediction head uses a Multilayer Perceptron (MLP), enabling simultaneous optimization of thousands of views in each individual training iteration, thus achieving stable and extremely fast convergence. ACE can map a new scene or scene mapping within 5 minutes without prior knowledge, making it up to 300 times faster than state-of-the-art scene coordinate regression methods, significantly improving training efficiency and making it more practical.

[0116] Furthermore, the ACE network can compile a scene into 4MB of network weights, requiring significantly less storage space than traditional scene coordinate regression networks. This significantly reduces storage consumption while maintaining accuracy, allowing the algorithm to be better applied to resource-constrained devices. Moreover, the ACE network improves relocalization efficiency and accuracy by introducing innovative technologies and architectures, such as generalized convolutional neural networks for feature map extraction and a compact MLP prediction head.

[0117] However, the core limitation of scene regression methods is that the capacity of a single network is insufficient to adapt to large scenes. To address this issue, the ACE network proposes a "scene clustering and segmentation + multi-sub-network training" strategy for large scenes: the scene is divided into multiple sub-regions according to the camera pose, and a sub-network is trained for each sub-region. During inference, multiple sub-networks are run in parallel, and the output results are selected to improve accuracy.

[0118] The ACE network trained based on the "scene clustering and segmentation + multi-sub-network training" strategy will be referred to as the segmented ACE network. This segmented ACE network consists of a scene-independent feature backbone network and multiple sub-networks associated with specific scenes. Each sub-network corresponds to a specific scene or sub-scene.

[0119] In some embodiments, a large scene refers to a three-dimensional scene with a large physical space, high environmental complexity, and rich heterogeneous elements. The spatial scale of a large scene (usually a spatial range of hundreds of meters to kilometers) is usually much larger than that of an ordinary indoor scene (a few meters to tens of meters) or a small outdoor scene. That is, it includes areas with rich textures (such as shop signs and vegetation) as well as areas with weak textures (such as walls and roads).

[0120] For example, large-scale scenarios can include: urban blocks (such as a commercial street or a residential area); industrial parks or scenic areas (such as university campuses or theme parks); and large outdoor facilities (such as the area around airport terminals or port terminals).

[0121] The scene clustering and segmentation described above divides the scene into multiple sub-regions according to the camera pose, and trains a separate ACE sub-network based on the scene image within each sub-region. On the one hand, segmentation based on camera pose may lead to unreasonable partition boundaries. For example, spatially adjacent 3D points may be segmented into different sub-regions due to pose division, causing discontinuity in feature learning. Such inappropriate clustering can lead to blurred feature representations, which in turn reduces localization accuracy.

[0122] On the other hand, the parallel operation of multiple subnetworks during inference significantly increases computational overhead. This not only reduces the efficiency of ACE networks in practical applications but also limits their ability to scale to larger scenarios, as computational costs increase linearly with the number of subnetworks.

[0123] To address the aforementioned shortcomings of the ACE network in large-scale scenes, this application provides a training method for a scene coordinate regression network and a localization method for the scene coordinate regression network. Based on the existing ACE network, a Mixture of Experts (MoE) module is introduced. This MoE module has the ability to implicitly model global information. The local image features of image patches contain detailed information such as texture and structure of the image. After being learned by the MoE module, this information can be naturally mapped to global spatial partition features. By introducing global spatial partition features, the accuracy of the prediction results of the sub-network is improved, thereby improving the localization accuracy.

[0124] Figure 1 This is a flowchart of the training method for the scene coordinate regression network provided in Embodiment 1 of this application, applied in a training device. This training device can be an electronic device such as a personal computer, mobile phone, tablet computer, server, or dedicated equipment. Figure 1 As shown, the method provided in this embodiment includes the following steps.

[0125] S101. Divide the scene image set into K sub-image sets according to the camera pose of the images, where K is an integer greater than or equal to 2.

[0126] During the preparation phase of model training, scene images are collected using a camera to construct a scene image set. This scene image set includes a large number of scene images, which contain rich scene features.

[0127] In the ACE model, scene images are also called mapping images. Mapping images are images used to build maps or provide scene information.

[0128] In large scenes, multiple cameras with different poses are typically installed to capture scene images. The images in the scene image set can be divided into K sub-image sets based on the camera poses, with the images in each sub-image set having the same or similar camera poses. This process can be understood as a clustering process of camera poses.

[0129] Each sub-image set corresponds to a sub-scene (or sub-region). The images in a sub-image set come from the corresponding sub-scene, which is relative to the larger scene. Dividing the scene image set into K sub-image sets according to the camera pose of the images essentially divides the larger scene corresponding to the scene image set into multiple sub-scenes.

[0130] S102. Using K sub-image sets, K sub-networks of the ACE network are pre-trained respectively. The sub-networks are used to predict the scene coordinates of pixels based on the input image features.

[0131] The traditional ACE network consists of a scene-independent feature backbone network and a scene-specific prediction head. The scene-independent feature backbone network is a general convolutional neural network used for feature extraction from images. In this embodiment, the scene-independent feature backbone network is referred to as the feature extraction network.

[0132] Feature extraction networks are used to extract general image features from input images. These networks are pre-trained, and their parameters remain unchanged during the training of the ACE network.

[0133] In the ACE network, the scene-specific prediction head is an MLP (Multi-Level Processing). Compared to traditional convolutional networks, MLPs can better represent the mapping from high-dimensional feature vectors to 3D points in scene space. The scene-specific prediction head is an MLP trained separately for each specific scene. The ACE network can quickly adapt to new scenes by training only the scene-specific prediction head, without retraining the entire network, significantly improving training efficiency and localization accuracy.

[0134] In this embodiment, the K sub-networks are K prediction heads for specific scenes, i.e., K prediction heads for sub-scenes. The sub-networks are used to predict the scene coordinates of pixels based on the input image features.

[0135] A scene image contains multiple pixels, which are 2D points. Pixels in an image can be indexed using pixel coordinates (i.e., UV coordinates). The scene coordinates of a pixel refer to the coordinates of the corresponding 3D point within the scene; the scene coordinates of a pixel are 3D coordinates.

[0136] In this embodiment, the K sub-networks are trained independently, and each sub-network is trained using a sub-image set.

[0137] In one implementation, for each sub-image set, a corresponding sub-training set is generated based on the sub-image set and the feature extraction network. The sub-training set includes multiple third training samples. The sub-network is pre-trained using the sub-training set corresponding to the sub-image set.

[0138] The training process of the ACE network mainly includes two stages: (1) Training buffer generation, which is used to generate image features of image blocks based on the scene images collected in the mapping stage, and generate training samples based on the images of the image blocks to provide a data basis for subsequent training; (2) Main training loop, which uses a large number of diverse training batches to perform multiple loops or iterative training.

[0139] Figure 2 This is a schematic diagram illustrating the training of the K subnetworks of the ACE network, as shown below. Figure 2 As shown, taking the training process of the i-th sub-network as an example, the sub-image set corresponding to the i-th sub-network includes multiple scene images. Each scene image is processed in turn, and the processing process for each image is the same.

[0140] Taking any scene image as an example, the scene image is first divided into M image patches of a fixed size. A feature extraction network is then used to extract features from each image patch, resulting in image features for each patch. These image features are high-dimensional feature vectors, such as 256-dimensional or 512-dimensional vectors. These features include texture, structure, and other characteristic information of the image. The image features of each image patch are also called image descriptors or feature maps.

[0141] For example, dividing a 1024×768 scene image into fixed 16*16 pixel blocks, it will be divided into (1024 / 16)×(768 / 16)=64×48=3072 image blocks, where M is the number of blocks. Each image block corresponds to a local region in the image. Since an image block is obtained by segmenting a scene image, its features are local features relative to the scene image.

[0142] Therefore, in the ACE architecture, a scene image generates a large number of image features, which are the core data for subsequent training of sub-networks corresponding to specific scenes. The feature generation of scene images follows the logic of "dense feature extraction," with the aim of providing sufficiently rich samples for scene coordinate regression.

[0143] The core task of ACE is to predict the corresponding 3D scene coordinates for local regions of an image (e.g., image patches), thereby establishing a dense 2D-3D correspondence (used for subsequent PnP algorithm to solve camera pose). By segmenting the scene image into M image patches and extracting the features of each image patch, the sub-network corresponding to a specific scene can predict the scene coordinates of a specific pixel in each image patch (e.g., the center point of the pixel) individually, thereby obtaining the scene coordinates of M pixels and forming M 2D-3D correspondences.

[0144] Based on the image features of M image patches from any scene image, M training samples are generated and stored in the training buffer. In the ACE model, the training buffer is a fixed-size data structure used to store training data.

[0145] The training buffer contains millions of image features, along with sample data consisting of the pixel locations (i.e., pixel coordinates), camera pose, and camera intrinsic parameters corresponding to those features. Each colored box in the training buffer represents one such sample data point, and sample data points with the same color come from the same scene image.

[0146] The final training buffer contains training samples generated from all images in the sub-image set corresponding to the i-th sub-network. The training buffer storing all training samples constitutes a complete sample set. Each training sample includes the image features of an image patch, the pixel coordinates of the pixels in the image patch, and the camera pose of the scene image to which the image patch belongs. Optionally, the training samples may also include camera intrinsic parameters.

[0147] A training buffer is generated during the first n minutes of training (e.g., the first minute), and then the main training loop begins, iterating over the training buffer.

[0148] In one implementation, the subnetwork can be trained over multiple training epochs. An epoch refers to the training process that involves completely traversing the entire sample set (or dataset) in the training buffer once. For example, if the training buffer stores 20,000 training samples, then an epoch refers to the training process that involves completely traversing all 20,000 training samples once. The number of epochs is a hyperparameter that needs to be adjusted based on the specific task and dataset to avoid underfitting or overfitting.

[0149] An epoch typically contains several iterations. For example, the number of iterations = the total number of training samples / batch size. The model is trained once with one batch of data, which is the smallest unit for parameter updates.

[0150] Samples are collected from a complete sample set generated from a collection of scene images using a randomization method to form the training set for the current batch.

[0151] In each iteration, samples are collected from the complete sample set generated from the sub-image set using a randomization method to form the training set for the current batch. The sub-network is trained using the training set for the current batch, and the final sub-network is formed through multiple iterations.

[0152] In one implementation, the randomization method is a shuffle operation. The shuffle operation involves randomly disrupting the order of samples or image features stored in the training buffer at the beginning of each epoch. The training buffer contains image features from multiple scene images, which may originally be arranged in a certain order, such as according to the image acquisition order or other rules. The shuffle operation breaks this original order, making the arrangement of image features random.

[0153] The shuffling operation ensures that features from different scene images are evenly distributed and mixed throughout the training buffer. This allows for more diverse feature combinations each time the model samples features from the training buffer during subsequent training, moving beyond features limited to the same or a few scene images. This increases the diversity of training data, helping the model learn more generalizable feature representations, preventing over-reliance on feature patterns from specific images, and improving the model's training performance and generalization ability.

[0154] After generating the sub-training set of the i-th sub-network, the multiple third training samples included in the sub-training set are sequentially input into the i-th sub-network. The third training samples include the pixel coordinates of the pixel, the image features of the image block where the pixel is located, and the camera pose of the image to which the pixel belongs.

[0155] For each third training sample, the i-th sub-network predicts the scene coordinates of the pixel based on the third training sample. Based on the pixel coordinates, the scene coordinates of the pixel, and the camera pose (the camera pose included in the third training sample), the reprojection loss of the i-th sub-network is determined. Based on the reprojection loss of the i-th sub-network, the parameters of the sub-network are updated.

[0156] For example, suppose a pixel is represented as x i , in pixels x i A pixel block centered at a point is represented as p. i Then pixel x i Scene coordinates y i It can be expressed by the following formula (1):

[0157] y i=f(p i (1)

[0158] Here, f() represents a subnetwork, and f() is implemented by a neural network.

[0159] Here, we will use an unsupervised scene regression training method for the sub-network as an example for illustration. The unsupervised method is achieved by introducing a reprojection loss. Optionally, the sub-network can also use real scene coordinates as supervision information and adopt supervised scene regression training.

[0160] Reprojection refers to the process of mapping a three-dimensional point (X,Y,Z) in the world coordinate system to two-dimensional pixel coordinates (u,v) on the image plane through camera pose and camera intrinsic parameters.

[0161] In the ACE model, reprojection is often used to verify the accuracy of scene coordinate prediction. Specifically, using the camera intrinsic parameters and the currently estimated camera pose, the scene coordinates (i.e., 3D points) of the predicted pixels are reprojected onto the image plane to obtain the predicted pixel coordinates of the pixels. The Euclidean distance between the predicted pixel coordinates and the actual pixel coordinates (i.e., the pixel coordinates included in the training samples) is calculated to obtain the reprojection error, which is the reprojection loss.

[0162] For example, the reprojection loss of a subnetwork can be expressed by the following formula (2):

[0163]

[0164] Where, x i Represents scene image I j The pixel coordinates of the i-th pixel in the image are y i Represents scene image I j The scene coordinates of the i-th pixel in T j Represents scene image I j The corresponding camera pose, l π denoted by , where w represents the reprojection loss and w represents the learnable parameters of the ACE network.

[0165] Calculating the reprojection loss requires both camera intrinsic and extrinsic parameters. The camera intrinsic parameters are obtained through a camera calibration algorithm. The camera memory can be carried in the training samples or stored separately in the training buffer. When calculating the reprojection loss, the camera intrinsic parameters are retrieved from the training buffer.

[0166] By training the K sub-networks sequentially or in parallel using the above training method, the K sub-networks of the ACE network are obtained. Each sub-network is a prediction head related to a sub-scene, used to predict the scene coordinates of pixels based on the input image features.

[0167] S103. Generate a first training set based on the scene image set and the feature extraction network. The first training set includes multiple first training samples.

[0168] In one implementation, for each scene image in the scene image set, the scene image is divided into M image blocks; image features of each of the M image blocks are extracted by a feature extraction network; M first training samples are generated based on the image features of the M image blocks, wherein each first training sample includes the image features of an image block, the pixel coordinates of the pixels in the image block, and the camera pose of the scene image to which the image block belongs.

[0169] After performing the above processing on all images in the scene image set, a complete sample set corresponding to the scene image set is generated. This complete sample set is also called the training buffer corresponding to the scene image set. Then, training samples are collected from the complete sample set generated from the scene image set using a randomization method to form the training set for the current batch. The first training set in this step can be the training set of any batch.

[0170] The process of generating the first training set is the same as the process of generating the sub-training sets of each sub-network in step S102, so it will not be repeated here. The difference is that the first training set is generated based on the complete set of scene images, while the sub-training sets of the sub-networks are generated based on the sub-image sets of the set of scene images.

[0171] S104. Train the target ACE network using the first training set. The target ACE network includes a feature extraction network and a hybrid expert module. The hybrid expert module includes a gating network and K expert units, and the K expert units are K sub-networks.

[0172] Compared to existing segmented ACE networks, the target ACE network introduces a hybrid expert architecture, which uses a gating network to achieve implicit global description and dynamic selection of subnetworks.

[0173] The concept behind hybrid expert architecture is that for complex tasks, no single model can perfectly handle all situations, while different expert units (or expert networks) may be adept at handling different types of data or task sub-parts. Therefore, hybrid expert architecture combines multiple different expert units, uses a gating network to determine the characteristics of the input data, decides which expert unit(s) should process the input, and finally integrates the outputs of the expert units to obtain the final result.

[0174] Gated networks are responsible for analyzing input data, calculating the "probability" or "weight" of each expert unit processing that input, and thus determining which expert units to assign the input to. A gated network is typically also a neural network, and its output is a probability distribution or weights indicating the contribution of each expert unit to the current input. The weights can be calculated based on the probabilities.

[0175] In this embodiment, each expert unit is a sub-network used to predict the scene coordinates of pixels based on the input training samples. The input to the gating network is the image features of the first training sample. The gating network determines the probability of the first training sample being assigned to each sub-network based on the image features in the first training sample, and determines the weight corresponding to each sub-network based on the probability. Simultaneously, the first training sample is input to K sub-networks respectively, and each of the K sub-networks processes the first training sample to obtain the scene coordinates of the pixels. Based on the weights corresponding to the K sub-networks and the scene coordinates output by the K sub-networks respectively, a weighted fused scene coordinate is obtained, and the target ACE network outputs the fused scene coordinates.

[0176] Figure 3 A schematic diagram of the training architecture for the target ACE network, as shown below. Figure 3 As shown, features are first extracted from the images in the scene image set using a feature extraction network. The extracted image features are stored in the training buffer and training samples are generated. After generating a complete sample set, a randomization method is used to sample the samples in the complete sample set to obtain the first training set. The first training samples are input into a gating network. The gating network determines the weights of the training samples assigned to each sub-network based on the image features. The first training samples are input into K sub-networks respectively. Each sub-network predicts a scene coordinate. Based on the weights of the K sub-networks and the scene coordinates output by the K sub-networks respectively, a weighted average is performed to obtain the fused scene coordinates.

[0177] For example, the output of the target ACE network is represented by the following formula (3):

[0178]

[0179] Among them, w i,j This represents the weights corresponding to the j-th subnetwork, i.e., the weights assigned to the j-th subnetwork by the i-th training sample, where j takes values ​​of 1, 2, ..., K. (Expert) j (f i () represents the scene coordinates output by the j-th sub-network, that is, the scene coordinates output by the j-th sub-network after the i-th training sample is processed by the j-th sub-network. This represents the coordinates of the fused scene corresponding to the i-th training sample, which is the output result of the target ACE network.

[0180] The core advantage of the hybrid expert architecture provided in this application lies in its implicit modeling capability for global information. Image features in the training samples, acting as local feature descriptors, inherently contain detailed information such as image texture and structure. After being learned by the gating network, this information can be naturally mapped to global spatial partitioning features. In other words, the distribution pattern of local features itself contains global information about "which sub-region this feature belongs to." Therefore, the gating network does not need to rely on additional global descriptors (such as scene-level feature vectors or geographic labels). It can achieve accurate classification of sub-regions simply by analyzing the statistical regularities of local features. This effectively solves the problem of missing global information in large scenes while avoiding the computational overhead of feature enhancement, improving the accuracy of the target ACE network's prediction results, and consequently improving the positioning accuracy of the scene coordinate regression-based positioning method.

[0181] Hybrid expert architectures commonly suffer from expert imbalance, where some sub-networks are overused while others are underutilized. Therefore, load balancing of hybrid expert modules is necessary.

[0182] In one load balancing method, load balancing can be forced by introducing additional auxiliary losses (such as entropy loss based on the utilization rate of expert units), where the utilization rate of expert units refers to the utilization rate of the sub-network.

[0183] In this load balancing method, when calculating the loss of the target ACE network, in addition to calculating the reprojection loss of the target ACE network, an auxiliary loss is also calculated based on the utilization rate of the K sub-networks. This auxiliary loss is also called the balancing loss. Then, based on the reprojection loss and the auxiliary loss, a weighted calculation method is used to obtain the loss of the target ACE network.

[0184] In another load balancing method, a load balancing bias term is introduced for each sub-network. This bias term is a learnable dynamic parameter. Accordingly, a gating network is used to determine the probability of a training sample being assigned to each sub-network based on the image features in the training samples. The probability of each sub-network is then adjusted based on the learnable bias term of each sub-network, and the weights corresponding to each sub-network are determined based on the adjusted probabilities.

[0185] During training, this method dynamically adjusts the probability of subnetwork selection by a learnable bias term. This bias term can be adaptively adjusted. This equalization method is applied to the forward computation of the gated network. It can achieve balanced utilization of subnetworks without introducing an additional loss term, while maintaining end-to-end differentiability.

[0186] In this load balancing method, for all training samples, the reprojection loss of the target ACE network is determined based on the pixel coordinates of the pixels, the fused scene coordinates of the pixels, and the camera pose. Based on the reprojection loss of the target ACE network, the parameters of the sub-networks and the gated network are updated.

[0187] This load balancing method directly improves the classification quality of global descriptors by optimizing the balance of expert selection. In large-scene localization, the effective classification of global descriptors depends on the specific modeling ability of each sub-network for a particular spatial region. When the expert load is unbalanced, some sub-networks are forced to learn global feature maps with poor generalization due to overuse, while the region modeling ability of idle sub-networks is not fully activated, resulting in regional coverage bias in global descriptor classification.

[0188] Understandably, the output of each subnetwork of the target ACE network is a 4D homogeneous coordinate offset (ΔX, ΔY, ΔZ, w), rather than directly outputting 3D scene coordinates. Here, ΔX, ΔY, and ΔZ represent the offset of a 3D point relative to the camera center, and w is a scale factor used to convert the offset from homogeneous coordinates to Euclidean coordinates.

[0189] In the ACE architecture, the position decoder converts the 4D homogeneous coordinate offsets output by each sub-network into 3D coordinates in the world coordinate system. This position encoder performs coordinate transformation based on the camera center coordinates.

[0190] For example, the position decoder converts the homogeneous offset into three-dimensional coordinates in the world coordinate system using the following formula (4):

[0191]

[0192] Among them, c x ,c y ,c z It uses the camera center coordinates, and by using relative offsets instead of absolute coordinates, it reduces the pressure on the network to directly predict large-scale coordinates (especially in large scenes).

[0193] The original ACE decoding method uses a fixed camera center during model training. The coordinates of this fixed camera center are derived from the SLAM or Structure from Motion (SfM) results during the mapping phase. Because it relies on a fixed camera center, the predicted coordinates tend to concentrate near the fixed camera center, forming a unimodal prior limitation.

[0194] To address the limitations of a single-peak prior, this embodiment clusters the camera poses to obtain K camera centers, meaning each sub-network has one camera center. Accordingly, during model training, the output of each sub-network includes not only the coordinate offset but also the weight of each camera center, with the sum of the weights of the K camera centers equal to 1. Optionally, the weight of each camera center is the weight assigned to each sub-network based on the training samples.

[0195] When the position decoder performs position decoding based on the coordinate offset, it uses a weighted operation based on the coordinates and weights of the K camera centers to obtain the target camera center. That is, the coordinates of each camera center are multiplied by the corresponding weight and then added together to obtain the target camera center. Then, the target camera center is substituted into formula (4) to calculate the three-dimensional coordinates.

[0196] This location decoding method can dynamically adjust the coordinates of the camera center based on image features, so that the predicted scene coordinates naturally follow the expansion of the camera's field of view, avoiding the limitation of a single peak, and can better adapt to the positioning needs of large scenes, thereby improving the model's positioning accuracy and scene adaptability.

[0197] The training method provided in this embodiment first pre-trains K sub-networks of the ACE network based on a scene image set. Then, it generates a first training set based on the scene image set and the feature extraction network, which includes multiple first training samples. The first training set is used to train the target ACE network, which includes a feature extraction network and a hybrid expert module. The hybrid expert module includes a gating network and K expert units, which are the K pre-trained sub-networks. The gating network determines the probability of a training sample being assigned to each sub-network based on the image features in the training samples, and determines the weight corresponding to each sub-network based on the probability. The output of the target ACE network is a fused scene coordinate obtained by weighting the scene coordinates output by the K sub-networks and their corresponding weights. In this method, the gating network can learn global spatial partitioning features based on the local features of the input, and selects a suitable sub-network based on the local features of the input and the global spatial partitioning features, thereby improving the accuracy of sub-network selection and thus improving the accuracy of scene coordinate prediction and camera pose localization.

[0198] Based on the foregoing, Embodiment 2 of this application provides a training method for a scene coordinate regression network. The same content is described above and will not be repeated in this embodiment. Figure 4 This is a flowchart of the training method for the scene coordinate regression network provided in Embodiment 2 of this application, as follows: Figure 4 As shown, the training method for this scene coordinate regression network includes the following steps.

[0199] S201. Divide the scene image set into K sub-image sets according to the camera pose of the images, where K is an integer greater than or equal to 2.

[0200] S202. Using K sub-image sets, K sub-networks of the ACE network are pre-trained respectively. The sub-networks are used to predict the scene coordinates of pixels based on the input image features.

[0201] S203. Generate a second training set based on the scene image set and the feature extraction network. The second training set includes multiple second training samples.

[0202] In one implementation, for each scene image in the scene image set, the scene image is divided into M image blocks; image features of each of the M image blocks are extracted by a feature extraction network; M second training samples are generated based on the image features of the M image blocks, wherein each second training sample includes the image features of an image block, the pixel coordinates of the pixels in the image block, and the camera pose of the scene image to which the image block belongs.

[0203] After performing the above processing on all images in the scene image set, a complete sample set corresponding to the scene image set is generated. This complete sample set is also called the buffer corresponding to the scene image set. Then, training samples are collected from the complete sample set generated from the scene image set using a randomization method to form the training set for the current batch. The second training set in this step can be the training set of any batch.

[0204] The process of generating the second training set is the same as the process of generating the sub-training sets of each sub-network in step S102, so it will not be repeated here. The difference is that the second training set is generated based on the complete set of scene images, while the sub-training sets of the sub-networks are generated based on the sub-image sets of the set of scene images.

[0205] S204. Fix the parameters of the K sub-networks and pre-train the gating network of the target ACE network using the second training set. The target ACE network includes a feature extraction network and a hybrid expert module. The hybrid expert module includes a gating network and K expert units, and the K expert units are K sub-networks.

[0206] In the training method described in Example 1, the training process of the target ACE network includes two sub-stages: (1) the pre-training stage of the K sub-networks of the ACE network; and (2) the joint training stage of the gating network and the sub-networks.

[0207] Unlike the training method in Example 1, the training process of the target ACE network in this example includes three sub-stages: (1) pre-training stage of K sub-networks of the ACE network; (2) pre-training stage of the gating network; and (3) joint training stage of the gating network and sub-networks.

[0208] The pre-training phase of the K subnetworks of the ACE network refers to the training phase described in steps S101 and S102 of Embodiment 1, and the joint training phase of the gated network and subnetworks refers to the training phase described in steps S103 and S104 of Embodiment 1.

[0209] In this embodiment, the training method adds a pre-training stage for the gating network before the joint training stage of the gating network and sub-networks. The main purpose of adopting a phased learning strategy for the gating network (i.e., first using gating network pre-training and then using joint training) is to first establish the basic selection capabilities of the gating network through pre-training, and then use the joint training method of the gating network and sub-networks to allow the two to optimize together.

[0210] During the pre-training phase of the gated network, K sub-network parameters are fixed. These K sub-network parameters are the same as those learned during the pre-training phase of the K sub-networks of the ACE network. With the sub-network parameters fixed, the learning objective of the gated network is simplified to "selecting the optimal sub-network based on the output error of the existing sub-networks." The parameters of the sub-networks and the gated network are decoupled, allowing the gated network to learn in a static reference frame first (the sub-network outputs, though imperfect, are fixed). This avoids the mutual drag caused by the simultaneous randomization of the parameters of both networks, enabling the gated network to quickly establish a basic mapping between features and the optimal sub-network.

[0211] In both the pre-training and joint training phases of the gating network, the function of the gating network is the same: to determine the probability of a training sample being assigned to each sub-network based on the image features in the training samples, and to determine the weights of each sub-network based on the probability.

[0212] In the pre-training phase of the gating network, the classification task is achieved by predicting the indices of sub-networks. Its loss function is constructed based on the reprojection error of the sub-network outputs; that is, the sub-network that minimizes the reprojection error of the current image is preferentially selected as the correct classification result. The goal of this phase is to allow the gating network to initially learn the differences in feature distributions among different sub-regions.

[0213] The pre-training phase of the gated network also needs to consider the load balancing problem of the hybrid expert modules. In this embodiment, the same load balancing method can be used for load balancing in the pre-training and joint training phases of the gated network.

[0214] In one implementation, a gating network is used to determine the probability of a training sample being assigned to each sub-network based on image features in the training samples. The probability of each sub-network is then adjusted based on its learnable bias term, and the weights of each sub-network are determined according to the adjusted probabilities. The bias term of each sub-network is dynamically learnable, as detailed in Embodiment 1.

[0215] S205. Generate a first training set based on the scene image set and the feature extraction network. The first training set includes multiple first training samples.

[0216] S206. Use the first training set to jointly train the gating network and K sub-networks of the target ACE network.

[0217] After the pre-training of the gated network is completed, the parameters of each sub-network are no longer fixed, and the joint training phase begins. During the joint training phase, the gated network and the sub-networks are optimized collaboratively in an end-to-end manner: the gated network dynamically selects the appropriate sub-network based on the input image features, while the sub-networks continuously optimize the coordinate prediction accuracy through reprojection loss.

[0218] The training method in this embodiment first fixes the sub-network parameters and pre-trains the gated network. Then, based on the pre-training results of the gated network, the gated network and K sub-networks are jointly optimized. Since the gated network has formed a preliminary selection strategy through pre-training, there is no need to learn the selection logic from scratch during joint training. This makes the optimization direction of the gated network and the sub-networks more consistent due to the preliminary selection strategy obtained from pre-training, reducing repeated oscillations in parameter adjustment (for example, the gate will not frequently switch selections due to small changes in the factor network parameters), thereby accelerating the overall convergence speed of the target ACE network.

[0219] In one implementation, both the pre-training and joint training phases of the gating network employ learnable bias terms from each sub-network to correct the probabilities of each sub-network, thereby achieving load balancing of the hybrid expert modules. Based on the foregoing, Embodiment 3 of this application provides a training method for a scene coordinate regression network, used to illustrate the load balancing method for the target ACE network. The same content is described above and will not be repeated in this embodiment.

[0220] Figure 5 This is a flowchart of the training method for the scene coordinate regression network provided in Embodiment 3 of this application. This embodiment uses the joint training phase of a gated network as an example for illustration. The load balancing in the pre-training phase of the gated network is the same as that in the joint training phase, and will not be repeated. Figure 5 As shown, the training method for this scene coordinate regression network includes the following steps.

[0221] S301. Generate a first training set based on the scene image set and the feature extraction network. The first training set includes multiple first training samples.

[0222] S302. Input the image features in the first training sample into the gating network, and determine the probability of the first training sample being assigned to each sub-network through the gating network.

[0223] The first training sample from the first training set is sequentially input into the gating network, which generates the probability that the first training sample is assigned to each sub-network. This probability can be a logarithmic probability.

[0224] S303. Adjust the probability of each subnetwork according to the bias term of each subnetwork.

[0225] In one implementation, the bias term of each subnetwork is added to the probability of that subnetwork to obtain the corrected probability of that subnetwork.

[0226] For example, the probability after subnetwork correction can be expressed by the following formula (5):

[0227]

[0228] Among them, f i Let b represent the image features of the i-th first training sample. j The bias term of the j-th subnetwork, b j ∈R K K represents the number of subnetworks (i.e. the number of experts), and the bias term is a learnable load balancing bias vector.

[0229] MLP(f i ) j This represents the probability that the i-th first training sample is assigned to the j-th sub-network. MLP indicates that the sub-network uses a multilayer perceptron. It can be understood that the sub-network is not limited to a multilayer perceptron, but can also be other neural networks. This represents the corrected probability that the i-th first training sample is assigned to the j-th subnetwork.

[0230] S304. Determine the weights of each subnetwork based on the corrected probabilities of the K subnetworks.

[0231] To ensure the differentiability of the training process, optionally, the Gumbel-based Softmax function (hereinafter referred to as the Gumbel-Softmax function) can be used to generate the weights corresponding to each sub-network.

[0232] The Gumbel-Softmax function is a technique used in deep learning to handle discrete variables. By approximating discrete sampling as a differentiable continuous process, it allows the model to use discrete variables during training while simultaneously being optimized through gradient descent.

[0233] In one implementation, generating the weights corresponding to the sub-networks based on the Gumbel-Softmax function may include the following steps: generating Gumbel noise for each sub-network; adding the Gumbel noise of each sub-network to the corrected probability of the corresponding sub-network to obtain the target probability of each sub-network; and calculating the weights corresponding to each sub-network using the Gumbel-Softmax function based on the target probabilities of the K sub-networks.

[0234] In one implementation, the weights corresponding to each sub-network are calculated using the following formula (6):

[0235]

[0236] Among them, w i,j This represents the weights assigned to the j-th subnetwork by the i-th training sample. G represents the corrected probability that the i-th training sample is assigned to the j-th subnetwork. i,j express The corresponding Gumbel noise, where τ is the temperature parameter. This represents the target probability of the i-th training sample being assigned to the j-th subnetwork.

[0237] S305. Input the first training sample into each sub-network to obtain the output of each sub-network. The output of each sub-network is the scene coordinates of the pixel.

[0238] During the training phase, continuous weights are used to maintain gradient information, so each subnetwork makes predictions on the first training sample.

[0239] S306. Based on the weights corresponding to the K sub-networks and the scene coordinates output by the K sub-networks respectively, the fused scene coordinates of the pixels are obtained by weighted processing.

[0240] For example, the scene coordinates of the pixel can be represented by the following formula (3):

[0241]

[0242] Among them, w i,j This represents the weight assigned to the j-th subnetwork by the i-th training sample, where j takes values ​​of 1, 2, ..., K. (Expert) j (f i() represents the scene coordinates output after the i-th training sample is processed by the j-th sub-network. This represents the fused scene coordinates corresponding to the i-th training sample.

[0243] S307. For each subnetwork, determine the long-term utilization rate of the subnetwork based on the weights corresponding to the subnetwork, and update the bias term of the subnetwork based on the long-term utilization rate and the target utilization rate.

[0244] The bias terms of the subnetwork are learnable parameters, and the bias terms of the subnetwork are updated independently of the parameters of the subnetwork and the parameters of the gated network.

[0245] Optionally, the long-term utilization rate of the sub-network is the exponential moving average (EMA) utilization rate.

[0246] EMA is a statistical method used to smooth time series data and track long-term trends. EMA calculates a dynamic average by giving new data higher weights and historical data exponentially decaying weights.

[0247] In one implementation, the long-term utilization of the sub-network is calculated using the following formula (7):

[0248]

[0249] Where α is the momentum factor, 0 < α < 1. α controls the decay rate of historical data. The closer α is to 1, the greater the influence of historical trends. For example, the value of α is 0.9.

[0250] N is the number of samples in the training set (i.e., the current batch). This represents the EMA utilization of the j-th network in the t-th iteration. This represents the weight assigned to the j-th subnetwork by the i-th training sample during the t-th iteration.

[0251] During model training, an epoch can contain several iterations. For example, the number of iterations = total number of samples / batch size.

[0252] In one implementation, the target utilization rate is set to 1 / K, where K is the number of subnetworks. Accordingly, updating the bias term of a subnetwork based on its long-term utilization rate and the target utilization rate can include the following steps: For each subnetwork, calculate the difference between its long-term and target utilization rates; multiply the utilization rate difference by the negative of the learning rate factor of the subnetwork's bias term to obtain the updated value of the subnetwork's bias term; add the updated value of the subnetwork's bias term to the updated value of the subnetwork's bias term to obtain the updated bias term of the subnetwork.

[0253] The method for updating the bias terms of the aforementioned subnetwork can be expressed by the following formula (8):

[0254]

[0255] in, u represents the long-term utilization of the j-th subnetwork at the t-th iteration. * Δb represents the target utilization rate. j Let λ represent the updated value of the bias term in the j-th subnetwork, and let λ represent the learning rate factor. This represents the bias term of the j-th subnetwork at the t-th iteration. This represents the bias term of the j-th subnetwork at the (t+1)-th iteration, i.e., the updated bias term of the j-th subnetwork.

[0256] S308. For each subnetwork, determine the reprojection loss of the subnetwork based on the pixel coordinates, the fused scene coordinates of the pixel, and the camera pose of the image to which the pixel belongs. Update the parameters of the subnetwork and the gated network based on the reprojection loss of the subnetwork.

[0257] The calculation of the reprojection loss of the subnetwork is described in the aforementioned content. Based on the reprojection loss of the K networks, the parameters of each subnetwork and the parameters of the gated network are updated.

[0258] Steps S301-S307 described above also apply to the pre-training phase of the gated network. The difference lies in the content updated based on the reprojection loss of the sub-network. During the pre-training phase, only the parameters of the gated network are updated based on the reprojection loss of the sub-network, not the parameters of the sub-networks. During the joint training phase, both the parameters of the gated network and the sub-networks are updated based on the reprojection loss of the sub-networks.

[0259] The training method in this embodiment introduces a load-balancing bias term for each sub-network. This bias term is a learnable dynamic parameter. Correspondingly, the gating network determines the probability of a training sample being assigned to each sub-network based on the image features in the training samples. The probability of each sub-network is adjusted according to its bias term, and the weight of each sub-network is determined based on the adjusted probability to achieve load balancing. This balancing method is applied to the forward computation process of the gating network, achieving balanced utilization of sub-networks without introducing an additional loss term. The target ACE network in this embodiment, by introducing a hybrid expert module, enables the gating network to implicitly learn global information. Based on this global information, it selects sub-networks using the input image features, improving the accuracy of sub-network selection and thus enhancing the localization accuracy of the scene coordinate regression-based localization method.

[0260] The training process of the target ACE network has been introduced above. The following section introduces the scene coordinate regression localization process based on the trained target ACE network.

[0261] Figure 6 This is a flowchart of the scene coordinate regression network localization method provided in Embodiment 4 of this application. In this embodiment, the target ACE network trained by the above training method is used for localization. The same content is referred to the foregoing content and will not be repeated here.

[0262] The method in this embodiment can be executed by a positioning device, which can be the same as or a different device from the aforementioned training device. In one application scenario, a dedicated training device trains the target ACE network, and after training, the target ACE network is stored in a fixed location (e.g., a server). The positioning device can then retrieve the target ACE network from this fixed location and install it. In another application scenario, the positioning device trains the target ACE network, and after training, the target ACE network is used for positioning.

[0263] This positioning device can be used on electronic devices such as mobile phones, personal computers, tablets, robots, VR devices, and aircraft. For example... Figure 6 As shown, the localization method of the scene coordinate regression network includes the following steps.

[0264] S401. Obtain the image to be located.

[0265] Positioning devices can capture images of the target location in real time using their own cameras. For example, robots or VR devices can capture images in real time using their own cameras and perform real-time positioning or map repositioning based on the captured images.

[0266] The positioning device can also receive images to be located sent by other devices, which may be images captured by cameras in a fixed area and transmitted to the positioning device.

[0267] S402. Using the image to be located and the target ACE network, determine the scene coordinates of M pixels in the image to be located in the scene. The target ACE network includes a feature extraction network and a hybrid expert module. The hybrid expert module includes a gating network and K expert units, each expert unit being a sub-network.

[0268] The target ACE network in this embodiment is a model trained using the aforementioned training method. The target ACE network includes a feature extraction network and a hybrid expert module. The hybrid expert module includes a gating network and K sub-networks.

[0269] The feature extraction network is used to divide the image to be located into M image blocks and extract the image features of each image block.

[0270] A gated network is used to determine the target subnetwork for each input information from K subnetworks based on the image features of each of the M input information. Each input information is input into the corresponding target subnetwork. The output of the target subnetwork includes the scene coordinates of the pixels in the input information in the scene. Each image block corresponds to one input information, and each input information includes the image features of an image block and the pixel coordinates of the pixels in the image block.

[0271] During the training of the ACE network, each training sample requires input to K sub-networks for prediction, resulting in K scene coordinates. These coordinates are then weighted according to their respective weights and outputs to obtain fused scene coordinates. Unlike the training process of the target ACE network, the inference process employs a hard selection strategy, choosing a target sub-network from the K sub-networks. This target sub-network then predicts the scene coordinates of the pixels based on image features from the input information.

[0272] A hard selection strategy refers to a gating network selecting only one optimal subnetwork (expert) from multiple subnetworks to handle the current input information, while other subnetworks do not participate in the computation. This strategy contrasts with "soft selection" (weighted fusion of the outputs of multiple subnetworks), and its core characteristic is a discrete, either-or selection. By employing this hard selection strategy, only one subnetwork is activated when using scene regression methods for large-scene localization inference, making the computational scale of large-scene localization comparable to that of small-scene localization, and improving the localization rate.

[0273] In one implementation, the scene coordinates of M pixels in the image to be located are determined using a target ACE network. This includes the following steps: dividing the image into M image blocks; extracting image features from each of the M image blocks using a feature extraction network; generating M input information based on the image features of the M image blocks; for each input information, inputting the image features into a gating network, and determining the probability of the input information being assigned to each sub-network using the gating network; correcting the probability of each sub-network based on its bias term; determining the weight of each sub-network based on the corrected probabilities of the K sub-networks; determining the sub-network with the largest weight as the target sub-network for this input information based on its weight; and processing the input information using the target sub-network to obtain the scene coordinates of the pixels in the input information within the scene.

[0274] The calculation methods for the probability and weight of each subnetwork are described in the aforementioned training process and will not be repeated here.

[0275] S403. Determine the camera pose of the image to be located based on the pixel coordinates of M pixels and the scene coordinates.

[0276] The pixel coordinates of M pixels and the scene coordinates constitute an M-to-2D-3D correspondence (also known as an M-to-2D-3D matching point). Based on the M-to-2D-3D correspondence, the camera pose of the image to be localized is determined.

[0277] Optionally, the PnP algorithm can be used to estimate M pairs of 2D-3D matching points to obtain the camera pose of the image to be localized. The PnP algorithm requires solving for the camera extrinsic parameters, namely the camera's rotation matrix and translation vector, using at least 3 pairs of 2D-3D matching points, thereby obtaining the camera's position and orientation in the world coordinate system based on the camera extrinsic parameters. Therefore, the value of M must be at least greater than 3.

[0278] In one implementation, abnormal pixels are first removed from the M pixels using the Random Sample Consensus (RANSAC) algorithm; based on the pixel coordinates of the remaining pixels and the scene coordinates, the PnP algorithm is used to estimate the initial camera pose of the image to be localized; the initial camera pose of the image to be localized is then optimized to obtain the camera pose of the image to be localized.

[0279] In practical applications, the M pairs of 2D-3D matching points obtained by the above method may contain some mismatched points (i.e., abnormal pixels). These mismatched points will seriously affect the accuracy of the PnP algorithm in solving the camera pose. Therefore, the RANSAC algorithm can be used to remove these mismatched points.

[0280] The camera pose estimated by the PnP algorithm is called the initial camera pose. The initial camera pose is subject to bias due to factors such as incorrect 2D-3D matching points and noisy data. To improve the accuracy of the camera pose, an iterative optimization method can be used to optimize the camera pose. The purpose of iterative optimization is to make the estimated camera pose value infinitely close to the true value, thereby reducing the positioning drift caused by the initial estimation error.

[0281] Commonly used iterative optimization methods include Bundle Adjustment (BA), which uses the least squares method to iteratively minimize the total reprojection error to obtain the optimal camera pose.

[0282] In this embodiment, an image to be located is acquired. Using the image to be located and a target ACE network, the scene coordinates of M pixels in the image to be located are determined. The target ACE network includes a feature extraction network and a hybrid expert module. The hybrid expert module includes a gating network and K expert units, each expert unit being a sub-network. The gating network is used to determine the target sub-network for each input information from the K sub-networks based on the image features of each of the M input information. Each input information is input into the corresponding target sub-network, and the output of the target sub-network includes the scene coordinates of the pixels in the input information. Based on the pixel coordinates of the M pixels and the scene coordinates, the camera pose of the image to be located is determined. In this method, the gating network selects only one optimal sub-network from multiple sub-networks to predict the scene coordinates. During localization, only one sub-network needs to be activated, making the computational scale of large-scene localization comparable to that of small-scene localization. Furthermore, the prediction results of this ACE network are more accurate, thereby improving localization accuracy.

[0283] It should be noted that in the method described in Embodiment 4, the gating network selects only one optimal subnetwork from multiple subnetworks to predict scene coordinates. In some embodiments of this application, K subnetworks can also be used to predict scene coordinates in parallel. Based on the scene coordinates output by the K subnetworks and the weights corresponding to the K subnetworks, the fused scene coordinates are calculated, and these fused scene coordinates are used as the scene coordinates of the pixels. The calculation method for the weights corresponding to the K subnetworks is the same as the method used by the target ACE network during the training phase, and will not be repeated here.

[0284] To facilitate better implementation of the scene coordinate regression network training method of the embodiments of this application, the embodiments of this application also provide a scene coordinate regression network training device. Figure 7 This is a schematic diagram of the training device for the scene coordinate regression network provided in Embodiment 5 of this application, as shown below. Figure 7 As shown, the training device 100 for the scene coordinate regression network may include:

[0285] The first training module 11 is used to divide the scene image set into K sub-image sets according to the camera pose of the image, and use the K sub-image sets to pre-train K sub-networks of the Accelerated Coordinate Encoding (ACE) network respectively. The sub-networks are used to predict the scene coordinates of the pixel points according to the input image features, where K is an integer greater than or equal to 2.

[0286] The second training module 12 is used to generate a first training set based on the scene image set and the feature extraction network. The first training set includes multiple first training samples. The first training set is used to train the target ACE network. The target ACE network includes the feature extraction network and a hybrid expert module. The hybrid expert module includes the gating network and K expert units. The K expert units are the K sub-networks. The gating network is used to determine the probability of a training sample being assigned to each sub-network based on the image features in the training samples, and to determine the weight corresponding to each sub-network based on the probability. The output of the target ACE network is a fused scene coordinate obtained by weighting the scene coordinates output by the K sub-networks and the weights corresponding to the K sub-networks respectively.

[0287] In one implementation, the device 100 further includes a third training module, used for:

[0288] Before the second training module 12 trains the target ACE network using the first training set, a second training set is generated based on the scene image set and the feature extraction network. The second training set includes multiple second training samples.

[0289] With the parameters of the K sub-networks fixed, the gating network of the target ACE network is pre-trained using the second training set.

[0290] In one implementation, the gated network is used to determine the probability of a training sample being assigned to each sub-network based on image features in the training samples, to correct the probability of each sub-network based on the learnable bias term of each sub-network, and to determine the weight corresponding to each sub-network based on the corrected probability.

[0291] In one implementation, the first training sample includes the pixel coordinates of a pixel, the image features of the image block containing the pixel, and the camera pose of the image to which the pixel belongs; correspondingly, the second training module 12 is specifically used for:

[0292] The image features in the first training sample are input into the gating network, and the probability of the first training sample being assigned to each sub-network is determined by the gating network.

[0293] The probability of each subnetwork is adjusted based on the bias term of each subnetwork.

[0294] Based on the corrected probabilities of the K sub-networks, determine the weights corresponding to each sub-network;

[0295] The first training sample is input into each sub-network to obtain the output of each sub-network, and the output of each sub-network is the scene coordinate of the pixel.

[0296] The fused scene coordinates of the pixel are obtained by weighting the weights corresponding to the K sub-networks and the scene coordinates output by the K sub-networks respectively.

[0297] For each subnetwork, the long-term utilization rate of the subnetwork is determined according to the weights corresponding to the subnetwork, and the bias term of the subnetwork is updated according to the long-term utilization rate and the target utilization rate.

[0298] For each subnetwork, the reprojection loss of the subnetwork is determined based on the pixel coordinates of the pixel, the fused scene coordinates of the pixel, and the camera pose of the image to which the pixel belongs. Based on the reprojection loss of the subnetwork, the parameters of the subnetwork and the parameters of the gated network are updated.

[0299] In one implementation, the second training sample includes the pixel coordinates of the pixel, the image features of the image block to which the pixel is located, and the camera pose of the image to which the pixel belongs; correspondingly, the third training module is specifically used for:

[0300] The image features in the second training sample are input into the gating network, and the probability of the second training sample being assigned to each sub-network is determined by the gating network.

[0301] The probability of each subnetwork is adjusted based on the bias term of each subnetwork.

[0302] Based on the corrected probabilities of the K sub-networks, determine the weights corresponding to each sub-network;

[0303] The second training sample is input into each sub-network to obtain the output of each sub-network, and the output of each sub-network is the scene coordinate of the pixel.

[0304] The fused scene coordinates of the pixel are obtained by weighting the weights corresponding to the K sub-networks and the scene coordinates output by the K sub-networks respectively.

[0305] For each subnetwork, the long-term utilization rate of the subnetwork is determined according to the weights corresponding to the subnetwork, and the bias term of the subnetwork is updated according to the long-term utilization rate and the target utilization rate.

[0306] For each subnetwork, the reprojection loss of the subnetwork is determined based on the pixel coordinates of the pixel, the fused scene coordinates of the pixel, and the camera pose of the image to which the pixel belongs. The parameters of the gated network are then updated based on the reprojection loss of the subnetwork.

[0307] In one implementation, determining the weights corresponding to each sub-network based on the corrected probabilities of the K sub-networks includes:

[0308] Generate Gumbel noise for each subnetwork;

[0309] Add the Gumbel noise of each subnetwork to the corrected probability of the corresponding subnetwork to obtain the target probability of each subnetwork.

[0310] Based on the target probabilities of the K sub-networks, the weights corresponding to each sub-network are calculated using the Gumbel-Softmax function.

[0311] In one implementation, the step of calculating the weights corresponding to each sub-network using the Gumbel-Softmax function based on the target probabilities of the K sub-networks includes:

[0312] The weights for each subnetwork are calculated using the following formula:

[0313]

[0314] Among them, w i,j This represents the weights assigned to the j-th subnetwork by the i-th training sample, z. i,j G represents the corrected probability that the i-th training sample is assigned to the j-th subnetwork. i,j express The corresponding Gumbel noise, where τ is the temperature parameter.

[0315] In one implementation, determining the long-term utilization rate of the sub-network based on the weights corresponding to the sub-network includes:

[0316] The long-term utilization rate of the sub-network is calculated using the following formula:

[0317]

[0318] Where α is the momentum factor and N is the number of samples in the training set. This represents the exponential moving average utilization of the j-th network at the t-th iteration. This represents the weight assigned to the j-th subnetwork by the i-th training sample during the t-th iteration.

[0319] In one implementation, the target utilization rate is set to 1 / K; updating the bias term of the sub-network based on its long-term utilization rate and the target utilization rate includes:

[0320] Calculate the difference between the long-term utilization rate and the target utilization rate of the sub-network;

[0321] Multiply the usage difference by the negative of the learning rate factor of the bias term of the sub-network to obtain the updated value of the bias term of the sub-network.

[0322] The updated bias term of the sub-network is obtained by adding the bias term of the sub-network to the updated bias term of the sub-network.

[0323] In one implementation, generating the first training set or the second training set based on the scene image set and the feature extraction network includes:

[0324] For each scene image in the scene image set, the scene image is divided into M image blocks;

[0325] The feature extraction network extracts the image features of each of the M image patches;

[0326] M training samples are generated based on the image features of the M image blocks, wherein each training sample includes the image features of an image block, the pixel coordinates of the pixels in the image block, and the camera pose of the scene image to which the image block belongs;

[0327] Samples are collected from the complete sample set generated from the scene image set using a randomization method to form the training set for the current batch.

[0328] In one implementation, the first training module 11 is specifically used for:

[0329] For each sub-image set, a corresponding sub-training set is generated based on the sub-image set and the feature extraction network, and the sub-training set includes multiple third training samples;

[0330] The sub-network is pre-trained using the sub-training set corresponding to the sub-image set.

[0331] In one implementation, the third training sample includes the pixel coordinates of the pixel, the image features of the image block where the pixel is located, and the camera pose of the image to which the pixel belongs;

[0332] The step of pre-training the sub-network corresponding to the sub-image set using the sub-training set includes:

[0333] The third training sample is input into the sub-network to predict the scene coordinates of the pixel.

[0334] The reprojection loss of the sub-network is determined based on the pixel coordinates, scene coordinates, and camera pose of the pixel.

[0335] The parameters of the subnetwork are updated based on the reprojection loss of the subnetwork.

[0336] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here.

[0337] This application also provides a positioning device for a scene coordinate regression network. Figure 8 This is a schematic diagram of the positioning device for the scene coordinate regression network provided in Embodiment Six of this application, as shown below. Figure 8 As shown, the localization device 200 for the scene coordinate regression network may include:

[0338] Acquisition module 21 is used to acquire the image to be located;

[0339] Scene coordinate determination module 22 is used to determine the scene coordinates of M pixels in the image to be located in the scene using the image to be located and the target ACE network trained by the scene coordinate regression network training device. The target ACE network includes a feature extraction network and a hybrid expert module. The hybrid expert module includes a gating network and K expert units, each expert unit being a sub-network. The feature extraction network is used to divide the image to be located into M image blocks and extract image features from each image block. The gating network is used to determine the target sub-network for each input information from the K sub-networks based on the image features of each of the M input information, and input each input information into the corresponding target sub-network. The output of the target sub-network includes the scene coordinates of the pixels in the input information in the scene. Each image block corresponds to one input information, and each input information includes the image features of an image block and the pixel coordinates of the pixels in the image block.

[0340] The pose determination module 23 is used to determine the camera pose of the image to be located based on the pixel coordinates of the M pixels and the scene coordinates.

[0341] In one implementation, the pose determination module 23 is specifically used for:

[0342] Abnormal pixels among the M pixels are removed using a random sampling consistency algorithm;

[0343] Based on the pixel coordinates of the remaining pixels and the scene coordinates, the initial camera pose of the image to be located is estimated using the n-point perspective PnP algorithm.

[0344] The initial camera pose of the image to be located is optimized to obtain the camera pose of the image to be located.

[0345] In one implementation, the scene coordinate determination module 22 is specifically used for:

[0346] The image to be located is divided into M image blocks;

[0347] The feature extraction network extracts the image features of each of the M image patches;

[0348] M input information is generated based on the image features of the M image blocks;

[0349] For each input information, the image features in the input information are input into the gating network, and the probability of the input information being assigned to each sub-network is determined by the gating network.

[0350] The probability of each subnetwork is adjusted based on the bias term of each subnetwork.

[0351] Based on the corrected probabilities of the K sub-networks, determine the weights corresponding to each sub-network;

[0352] Based on the weights corresponding to the K sub-networks, the sub-network with the largest weight is determined as the target sub-network used for the input information;

[0353] The target sub-network processes the input information to obtain the scene coordinates of the pixels in the input information within the scene.

[0354] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here.

[0355] The apparatuses 100 and 200 of this application embodiment have been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that these functional modules can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0356] This application also provides an electronic device. Figure 9 This is a schematic diagram of the structure of an electronic device provided in Embodiment 7 of this application, as shown below. Figure 9 As shown, the electronic device 300 may include:

[0357] The device includes a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, and a computer program stored on the memory 302 and executable on the processor. The processor 301 and the memory 302 are electrically connected. Those skilled in the art will understand that the electronic device structure shown in the figures does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0358] The processor 301 is the control center of the electronic device 300. It connects various parts of the electronic device 300 through various interfaces and lines. By running or loading software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it executes various functions of the electronic device 300 and processes data, thereby performing overall processing of the electronic device 300.

[0359] In this embodiment, the processor 301 in the electronic device 300 loads the instructions corresponding to the processes of one or more applications into the memory 302 according to the following steps, and the processor 301 runs the applications stored in the memory 302, thereby realizing the following training method for the scene coordinate regression network:

[0360] The scene image set is divided into K sub-image sets according to the camera pose of the images, where K is an integer greater than or equal to 2;

[0361] Using the K sub-image sets, K sub-networks of the Accelerated Coordinate Encoding (ACE) network are pre-trained respectively. The sub-networks are used to predict the scene coordinates of pixels based on the input image features.

[0362] A first training set is generated based on the scene image set and the feature extraction network, and the first training set includes multiple first training samples;

[0363] The target ACE network is trained using the first training set. The target ACE network includes the feature extraction network and a hybrid expert module. The hybrid expert module includes the gating network and K expert units, which are the K sub-networks. The gating network is used to determine the probability of a training sample being assigned to each sub-network based on the image features in the training samples, and to determine the weight corresponding to each sub-network based on the probability. The output of the target ACE network is a fused scene coordinate obtained by weighting the scene coordinates output by the K sub-networks and the weights corresponding to the K sub-networks.

[0364] Alternatively, processor 301 runs an application stored in memory 302 to implement the following scene coordinate regression network localization method:

[0365] Obtain the image to be located;

[0366] The target ACE network, trained using the training method of the image to be located and the scene coordinate regression network, determines the scene coordinates of M pixels in the image to be located in the scene. The target ACE network includes a feature extraction network and a hybrid expert module. The hybrid expert module includes a gating network and K expert units, each expert unit being a sub-network.

[0367] The feature extraction network is used to divide the image to be located into M image blocks and extract the image features of each image block respectively.

[0368] The gated network is used to determine the target sub-network for each input information from K sub-networks based on the image features of each input information in M ​​input information, and input each input information into the corresponding target sub-network. The output of the target sub-network includes the scene coordinates of the pixels in the input information in the scene. Each image block corresponds to one input information, and each input information includes the image features of an image block and the pixel coordinates of the pixels in the image block.

[0369] The camera pose of the image to be located is determined based on the pixel coordinates of the M pixels and the scene coordinates.

[0370] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0371] In some embodiments, the processor 301 may include, but is not limited to: a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0372] In some embodiments of this application, the memory 302 includes, but is not limited to, volatile memory and / or non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0373] Optional, such as Figure 9As shown, the electronic device 300 also includes: a display screen 303, a radio frequency circuit 304, an audio circuit 305, an input unit 306, a power supply 307, a camera 308, and an inertial sensor 309. The processor 301 is electrically connected to the display screen 303, the radio frequency circuit 304, the audio circuit 305, the input unit 306, the power supply 307, the camera 308, and the inertial sensor 309. Those skilled in the art will understand that... Figure 9 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0374] Display screen 303 is used to display images, videos, etc. Display screen 303 includes a display panel, which can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of electronic devices. These graphical user interfaces can be composed of graphics, text, icons, videos, and any combination thereof.

[0375] Optionally, the display screen 303 may also include a touch panel, and correspondingly, the display screen 303 may be a touch display screen. The touch panel can be used to collect user touch operations on or near it (such as user operations using fingers, styluses, or any suitable object or accessory on or near the touch panel), and generate corresponding operation instructions, which then execute the corresponding program. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 301, and can also receive and execute commands from the processor 301. The touch panel may cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 301 to determine the type of touch event, and then the processor 301 provides corresponding visual output on the display panel according to the type of touch event. In this embodiment, the touch panel and the display panel can be integrated into the touch display screen to realize input and output functions. However, in some embodiments, the touch panel and the display panel can be implemented as two separate components to perform input and output functions. That is, the touch display screen can also be used as part of the input unit 306 to perform input functions.

[0376] The radio frequency circuit 304 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other electronic devices, and to transmit and receive signals with network devices or other electronic devices.

[0377] Audio circuitry 305 can be used to provide an audio interface between a user and an electronic device via a speaker and a microphone. Audio circuitry 305 converts received audio data into electrical signals, transmits them to the speaker, and the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 305, converted back into audio data, and then processed by processor 301 before being transmitted via radio frequency circuitry 304 to, for example, another electronic device, or output to memory 302 for further processing. Audio circuitry 305 may also include an earphone jack to facilitate communication between peripheral headphones and electronic devices.

[0378] The input unit 306 can be used to receive input numbers, characters, or object feature information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0379] Power supply 307 supplies power to various components of electronic device 300. Optionally, power supply 307 can be logically connected to processor 301 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 307 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other components. The power management system can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance).

[0380] The camera 308 is used to acquire images and assist electronic devices in training and localization of scene coordinate regression networks. The number of cameras 308 can be one or more, and the type of camera 308 can be a fisheye camera, a regular camera, or other types of cameras. This application does not impose any restrictions on this.

[0381] The inertial sensor 309 is used to collect inertial measurement data (such as angular velocity, acceleration, etc.) of the device. The electronic device 300 can calculate the attitude change of the device based on the inertial measurement data. The inertial sensor 309 can be an IMU, which is a sensor that integrates an accelerometer and a gyroscope (some devices also include a magnetometer). The IMU can be a six-axis IMU or a nine-axis IMU, and no specific limitation is made here.

[0382] When the electronic device 300 is a VR device, the IMU and camera 308 are the core components for achieving accurate positioning and motion tracking of the VR device. The two work together to solve the problem of "how to map the user's position and posture in virtual space in real time". The IMU is used to measure instantaneous posture, but it will drift after long-term use; the camera 308 is used to measure absolute position, but it is affected by light and occlusion, and its response speed is relatively slow. When the two are combined, the system will use the absolute position of the camera 308 to calibrate the accumulated error of the IMU, and at the same time use the fast response of the IMU to fill the delay or occlusion gap of the camera 308, ultimately achieving low latency, high accuracy, and drift-free motion tracking.

[0383] although Figure 9 As not shown in the diagram, the electronic device 300 may also include a wireless fidelity module, a Bluetooth module, and other sensors.

[0384] It should be understood that the various components in the electronic device 300 are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0385] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0386] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the corresponding flow in the user position control method in the virtual scene of this application embodiment. For simplicity, further details are omitted here.

[0387] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0388] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0389] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A training method for a scene coordinate regression network, characterized in that, The method includes: The scene image set is divided into K sub-image sets according to the camera pose of the images, where K is an integer greater than or equal to 2; Using the K sub-image sets, K sub-networks of the accelerated coordinate encoding network are pre-trained respectively. The sub-networks are used to predict the scene coordinates of pixels based on the input image features. A first training set is generated based on the scene image set and the feature extraction network, and the first training set includes multiple first training samples; The target acceleration coordinate encoding network is trained using the first training set. The target acceleration coordinate encoding network includes the feature extraction network and a hybrid expert module. The hybrid expert module includes a gating network and K expert units, where the K expert units are the K sub-networks. The gating network is used to determine the probability of a training sample being assigned to each sub-network based on the image features in the training samples, and to determine the weight corresponding to each sub-network based on the probability. The output of the target acceleration coordinate encoding network is a fused scene coordinate obtained by weighting the scene coordinates output by the K sub-networks and their corresponding weights. Before training the target acceleration coordinate encoding network using the first training set, the following steps are also included: A second training set is generated based on the scene image set and the feature extraction network, and the second training set includes multiple second training samples. The parameters of the K sub-networks are fixed, and the gating network of the target acceleration coordinate encoding network is pre-trained using the second training set; The gated network is used to determine the probability of a training sample being assigned to each sub-network based on the image features in the training samples, to correct the probability of each sub-network based on the learnable bias term of each sub-network, and to determine the weight of each sub-network based on the corrected probability. The first training sample includes the pixel coordinates of the pixel, the image features of the image block where the pixel is located, and the camera pose of the image to which the pixel belongs; The step of training the target acceleration coordinate encoding network using the first training set includes: The image features in the first training sample are input into the gating network, and the probability of the first training sample being assigned to each sub-network is determined by the gating network. The probability of each subnetwork is adjusted based on the bias term of each subnetwork. Based on the corrected probabilities of the K sub-networks, determine the weights corresponding to each sub-network; The first training sample is input into each sub-network to obtain the output of each sub-network, and the output of each sub-network is the scene coordinate of the pixel. The fused scene coordinates of the pixel are obtained by weighting the weights corresponding to the K sub-networks and the scene coordinates output by the K sub-networks respectively. For each subnetwork, the long-term utilization rate of the subnetwork is determined according to the weights corresponding to the subnetwork, and the bias term of the subnetwork is updated according to the long-term utilization rate and the target utilization rate. For each subnetwork, the reprojection loss of the subnetwork is determined based on the pixel coordinates of the pixel, the fused scene coordinates of the pixel, and the camera pose of the image to which the pixel belongs. Based on the reprojection loss of the subnetwork, the parameters of the subnetwork and the parameters of the gated network are updated.

2. The method according to claim 1, characterized in that, The second training sample includes the pixel coordinates of the pixel, the image features of the image block where the pixel is located, and the camera pose of the image to which the pixel belongs; The step of fixing the parameters of the K sub-networks and pre-training the gated network of the target acceleration coordinate encoding network using the second training set includes: The image features in the second training sample are input into the gating network, and the probability of the second training sample being assigned to each sub-network is determined by the gating network. The probability of each subnetwork is adjusted based on the bias term of each subnetwork. Based on the corrected probabilities of the K sub-networks, determine the weights corresponding to each sub-network; The second training sample is input into each sub-network to obtain the output of each sub-network, and the output of each sub-network is the scene coordinate of the pixel. The fused scene coordinates of the pixel are obtained by weighting the weights corresponding to the K sub-networks and the scene coordinates output by the K sub-networks respectively. For each subnetwork, the long-term utilization rate of the subnetwork is determined according to the weights corresponding to the subnetwork, and the bias term of the subnetwork is updated according to the long-term utilization rate and the target utilization rate. For each subnetwork, the reprojection loss of the subnetwork is determined based on the pixel coordinates of the pixel, the fused scene coordinates of the pixel, and the camera pose of the image to which the pixel belongs. The parameters of the gated network are then updated based on the reprojection loss of the subnetwork.

3. The method according to claim 1 or 2, characterized in that, The step of determining the weight corresponding to each sub-network based on the corrected probabilities of the K sub-networks includes: Generate Gumbel noise for each subnetwork; Add the Gumbel noise of each subnetwork to the corrected probability of the corresponding subnetwork to obtain the target probability of each subnetwork. Based on the target probabilities of the K sub-networks, the weights corresponding to each sub-network are calculated using the Softmax function.

4. The method according to claim 3, characterized in that, The step of calculating the weights corresponding to each sub-network using the Softmax function based on the target probabilities of the K sub-networks includes: The weights for each subnetwork are calculated using the following formula: in, This represents the weights assigned to the j-th subnetwork by the i-th training sample. This represents the corrected probability that the i-th training sample is assigned to the j-th subnetwork. express The corresponding Gumbel noise, This refers to the temperature parameter.

5. The method according to claim 1 or 2, characterized in that, The step of determining the long-term utilization rate of the sub-network based on the weights corresponding to the sub-network includes: The long-term utilization rate of the sub-network is calculated using the following formula: in, Here, is the momentum factor, and N is the number of samples in the training set. This represents the exponential moving average utilization of the j-th network at the t-th iteration. This represents the weight assigned to the j-th subnetwork by the i-th training sample during the t-th iteration.

6. The method according to claim 1 or 2, characterized in that, The target utilization rate is set to 1 / K; The step of updating the bias term of the sub-network based on the long-term utilization rate and target utilization rate of the sub-network includes: Calculate the difference between the long-term utilization rate and the target utilization rate of the sub-network; Multiply the usage difference by the negative of the learning rate factor of the bias term of the sub-network to obtain the updated value of the bias term of the sub-network. The updated bias term of the sub-network is obtained by adding the bias term of the sub-network to the updated bias term of the sub-network.

7. The method according to any one of claims 1-2, characterized in that, Generating the first training set or the second training set based on the scene image set and the feature extraction network includes: For each scene image in the scene image set, the scene image is divided into M image blocks; The feature extraction network extracts the image features of each of the M image patches; M training samples are generated based on the image features of the M image blocks, wherein each training sample includes the image features of an image block, the pixel coordinates of the pixels in the image block, and the camera pose of the scene image to which the image block belongs; Samples are collected from the complete sample set generated from the scene image set using a randomization method to form the training set for the current batch.

8. The method according to any one of claims 1-2, characterized in that, The step of using the K sub-image sets to pre-train K sub-networks of the accelerated coordinate encoding network includes: For each sub-image set, a corresponding sub-training set is generated based on the sub-image set and the feature extraction network, and the sub-training set includes multiple third training samples; The sub-network is pre-trained using the sub-training set corresponding to the sub-image set.

9. The method according to claim 8, characterized in that, The third training sample includes the pixel coordinates of the pixel, the image features of the image block where the pixel is located, and the camera pose of the image to which the pixel belongs; The step of pre-training the sub-network corresponding to the sub-image set using the sub-training set includes: The third training sample is input into the sub-network to predict the scene coordinates of the pixel. The reprojection loss of the sub-network is determined based on the pixel coordinates, scene coordinates, and camera pose of the pixel. The parameters of the subnetwork are updated based on the reprojection loss of the subnetwork.

10. A localization method using a scene coordinate regression network, characterized in that, The method includes: Obtain the image to be located; Using the image to be located and the target accelerated coordinate encoding network trained by any one of claims 1-9, the scene coordinates of M pixels in the image to be located are determined in the scene, wherein the target accelerated coordinate encoding network includes a feature extraction network and a hybrid expert module, and the hybrid expert module includes a gating network and K expert units, each expert unit being a sub-network; The feature extraction network is used to divide the image to be located into M image blocks and extract the image features of each image block respectively. The gated network is used to determine the target sub-network for each input information from K sub-networks based on the image features of each input information in M ​​input information, and input each input information into the corresponding target sub-network. The output of the target sub-network includes the scene coordinates of the pixels in the input information in the scene. Each image block corresponds to one input information, and each input information includes the image features of an image block and the pixel coordinates of the pixels in the image block. The camera pose of the image to be located is determined based on the pixel coordinates of the M pixels and the scene coordinates.

11. The method according to claim 10, characterized in that, Determining the camera pose of the image to be located based on the pixel coordinates of the M pixels and the scene coordinates includes: Abnormal pixels among the M pixels are removed using a random sampling consistency algorithm; Based on the pixel coordinates of the remaining pixels and the scene coordinates, the initial camera pose of the image to be located is estimated using the n-point perspective PnP algorithm. The initial camera pose of the image to be located is optimized to obtain the camera pose of the image to be located.

12. The method according to claim 10, characterized in that, The step of determining the scene coordinates of M pixels in the image to be located within the scene using the target acceleration coordinate encoding network trained by the method described in any one of claims 1-9 includes: The image to be located is divided into M image blocks; The feature extraction network extracts the image features of each of the M image patches; M input information is generated based on the image features of the M image blocks; For each input information, the image features in the input information are input into the gating network, and the probability of the input information being assigned to each sub-network is determined by the gating network. The probability of each subnetwork is adjusted based on the bias term of each subnetwork. Based on the corrected probabilities of the K sub-networks, the weights to which the input information is assigned to each sub-network are determined; Based on the weights corresponding to the K sub-networks, the sub-network with the largest weight is determined as the target sub-network used for the input information; The target sub-network processes the input information to obtain the scene coordinates of the pixels in the input information within the scene.

13. A training device for a scene coordinate regression network, characterized in that, include: The first training module is used to divide the scene image set into K sub-image sets according to the camera pose of the images, and use the K sub-image sets to pre-train K sub-networks of the accelerated coordinate encoding network. The sub-networks are used to predict the scene coordinates of pixels based on the input image features, where K is an integer greater than or equal to 2. The second training module is used to generate a first training set based on the scene image set and the feature extraction network. The first training set includes multiple first training samples. The first training set is used to train the target acceleration coordinate encoding network. The target acceleration coordinate encoding network includes the feature extraction network and a hybrid expert module. The hybrid expert module includes a gating network and K expert units. The K expert units are the K sub-networks. The gating network is used to determine the weights assigned to each sub-network based on the image features in the training samples. The output of the target acceleration coordinate encoding network is a fused scene coordinate obtained by weighting the scene coordinates output by the K sub-networks according to their respective weights. The third training module is used before training the target acceleration coordinate encoding network using the first training set. A second training set is generated based on the scene image set and the feature extraction network, and the second training set includes multiple second training samples. The parameters of the K sub-networks are fixed, and the gating network of the target acceleration coordinate encoding network is pre-trained using the second training set; The gated network is used to determine the probability of a training sample being assigned to each sub-network based on the image features in the training samples, to correct the probability of each sub-network based on the learnable bias term of each sub-network, and to determine the weight of each sub-network based on the corrected probability. The first training sample includes the pixel coordinates of the pixel, the image features of the image block where the pixel is located, and the camera pose of the image to which the pixel belongs; The step of training the target acceleration coordinate encoding network using the first training set includes: The image features in the first training sample are input into the gating network, and the probability of the first training sample being assigned to each sub-network is determined by the gating network. The probability of each subnetwork is adjusted based on the bias term of each subnetwork. Based on the corrected probabilities of the K sub-networks, determine the weights corresponding to each sub-network; The first training sample is input into each sub-network to obtain the output of each sub-network, and the output of each sub-network is the scene coordinate of the pixel. The fused scene coordinates of the pixel are obtained by weighting the weights corresponding to the K sub-networks and the scene coordinates output by the K sub-networks respectively. For each subnetwork, the long-term utilization rate of the subnetwork is determined according to the weights corresponding to the subnetwork, and the bias term of the subnetwork is updated according to the long-term utilization rate and the target utilization rate. For each subnetwork, the reprojection loss of the subnetwork is determined based on the pixel coordinates of the pixel, the fused scene coordinates of the pixel, and the camera pose of the image to which the pixel belongs. Based on the reprojection loss of the subnetwork, the parameters of the subnetwork and the parameters of the gated network are updated.

14. A positioning device for a scene coordinate regression network, characterized in that, include: The acquisition module is used to acquire the image to be located. A scene coordinate determination module is used to determine the scene coordinates of M pixels in the image to be located in the scene using the image to be located and a target accelerated coordinate encoding network trained by any one of claims 1-9. The target accelerated coordinate encoding network includes a feature extraction network and a hybrid expert module. The hybrid expert module includes a gating network and K expert units, each expert unit being a sub-network. The feature extraction network is used to divide the image to be located into M image blocks and extract image features from each image block. The gating network is used to determine the target sub-network for each input information from the K sub-networks based on the image features of each of the M input information, and input each input information into the corresponding target sub-network. The output of the target sub-network includes the scene coordinates of the pixels in the input information in the scene. Each image block corresponds to one input information, and each input information includes the image features of an image block and the pixel coordinates of the pixels in the image block. The pose determination module is used to determine the camera pose of the image to be located based on the pixel coordinates of the M pixels and the scene coordinates.

15. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store a computer program, the processor being used to invoke and run the computer program stored in the memory to perform the method of any one of claims 1 to 9, or to perform the method of any one of claims 10 to 12.

16. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as claimed in any one of claims 1 to 9, or to perform the method as claimed in any one of claims 10 to 12.

Images