Relative pose estimation network training method, relative pose estimation method, camera absolute pose estimation method, device, equipment, medium and program product

Abstract

The invention relates to a relative pose estimation network training method, a relative pose estimation method, a camera absolute pose estimation method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: determining a reference image and a neighbor image of the reference image; extracting features of the reference image by using a deep network to obtain a first depth feature; using a pose network to extract relative pose features of the reference image and the neighbor image to obtain a first pose feature; inputting the first depth feature and the first pose feature into a fusion network to obtain a first fusion feature, and processing the first fusion feature through an estimation network to obtain a first relative pose; and determining a loss value based on the first relative pose, and training the relative pose estimation network based on the loss value. By adopting the method, the depth information can be introduced into the estimation of the relative pose, so that the relative pose with a scale is obtained, and the accuracy of the estimation of the relative pose is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a relative pose estimation network training method, a relative pose estimation method, a camera absolute pose estimation method, an apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Structure-from-motion (SfM) has always been an important research area in computer vision. SfM includes three key steps: feature extraction and matching, image first relative pose estimation, and global pose estimation. Based on the different strategies used in the third step, SfM can be divided into two categories, incremental SfM and global SfM. Incremental SfM usually starts with two to three "seed" views and gradually adds more images to the model. In order to mitigate the cumulative error, a bundle adjustment (BA) is performed after every few pictures are added. As the number of pictures increases, the complexity of the BA problem also increases, resulting in more and more time spent on BA, which ultimately affects computational efficiency.

[0003] Different from incremental SfM, global SfM estimates all camera poses simultaneously and only requires one BA, making it more efficient. The pose estimation in global SfM consists of two modules: rotation average and translation average, which calculate the global rotation and translation respectively. In the noise-free case,

[0004] The translational average follows the constraints, namely:

[0005] c j -c i =d ij R i v ij

[0006] Among them, c i and c j is the camera center, R i is the rotation from the ith camera to the world coordinates, v ij is the unit vector representing the relative translation from the jth camera to the ith camera, d ij =‖c j -c i ‖2 is the scale of the relative translation. However, the relative translation calculated from two images using the traditional five-point method only contains the direction but not the scale, i.e., d ijis unknown. This scale ambiguity is one of the main challenges of global SfM because it creates inconsistencies between the measurement space (the direction of the relative translation) and the solution space (the camera position). When the camera moves in a straight line, the impact of scale singularities becomes particularly significant, resulting in large errors in the relative pose estimation. Summary of the invention

[0007] Based on this, it is necessary to provide a relative pose estimation network training method, relative pose estimation method, camera absolute pose estimation method, device, computer equipment, computer-readable storage medium and computer program product that can address the above-mentioned technical problems.

[0008] In a first aspect, the present application provides a relative pose estimation network training method, wherein the relative pose estimation network includes a deep network, a pose network, a fusion network, and an estimation network, and the method includes:

[0009] Determining a reference image and neighboring images of the reference image;

[0010] Extracting features of the reference image using the deep network to obtain a first deep feature;

[0011] Extracting relative pose features of the reference image and the neighboring image using the pose network to obtain a first pose feature;

[0012] Inputting the first depth feature and the first posture feature into the fusion network to obtain a first fusion feature, and processing the first fusion feature through the estimation network to obtain a first relative posture;

[0013] A loss value is determined based on the first relative pose, and the relative pose estimation network is trained based on the loss value.

[0014] In one embodiment, the step of extracting the relative pose features of the reference image and the neighboring image using the pose network to obtain the first pose feature includes:

[0015] Extracting a first feature corresponding to the reference image through a first feature extraction module of the pose network, and extracting a second feature corresponding to the neighboring image through a second feature extraction module of the pose network; the first feature extraction module and the second feature extraction module share weights;

[0016] Through the decoder of the pose network, attention processing is performed based on the first feature and the second feature to obtain the first pose feature; wherein the decoder includes a self-attention layer and a cross-attention layer.

[0017] In one embodiment, inputting the first depth feature and the first posture feature into the fusion network to obtain a first fusion feature includes:

[0018] The first posture feature is processed by the self-attention layer of the fusion network, and the processed first posture feature and the first depth feature are fused by the cross-attention layer of the fusion network to obtain a first fusion feature.

[0019] In one embodiment, the processing the first fused feature by the estimation network to obtain the first relative pose includes:

[0020] Processing the first fused feature by a first sub-network of the estimation network to obtain a first relative translation;

[0021] Processing the first fused feature by a second sub-network of the estimation network to obtain a first relative rotation;

[0022] A first relative position and posture is obtained based on the first relative translation and the first relative rotation.

[0023] In one embodiment, the method further comprises:

[0024] Decoding based on the first depth feature to obtain a first depth feature map;

[0025] The determining the loss value according to the first relative posture comprises:

[0026] Based on the first depth feature map and the first relative pose, the reference image is projected onto the neighboring image to obtain a target image, and based on a photometric error value between the target image and the reference image, a loss value is obtained.

[0027] In one embodiment, the method further comprises:

[0028] Decoding based on the first depth feature map to obtain a mask, where the mask is used to represent the weight of each pixel in the reference image, and the weight of the pixel at the location of the fixed object in the reference image is higher than the weight of the pixel of the moving object and the texture-free area in the reference image;

[0029] Before the photometric error value between the target image and the reference image, the method further includes:

[0030] Obtaining an SSIM error of each pixel based on a pixel block corresponding to each pixel in the reference image and a first error weight;

[0031] Based on the corresponding pixels of the reference image and the target image and the second error weight, an L1 norm error of each pixel is obtained;

[0032] A photometric error value between the target image and the reference image is obtained based on the SSIM error, the L1 norm error, and the mask.

[0033] In one embodiment, the method further comprises:

[0034] Determine at least one of an edge-based first depth feature map smoothing loss value based on the first depth feature map and a pose consistency loss value based on the first relative pose;

[0035] The method for obtaining the edge-based first depth map smoothing loss value includes:

[0036] Regularizing the first depth feature map to obtain a regularized first depth feature map;

[0037] Weighting the gradient of the first depth feature map based on the gradient of the regular first depth feature map to obtain an edge-based first depth map smoothing loss value;

[0038] The method for obtaining the pose consistency loss value includes:

[0039] Inputting the neighbor image into the deep network to obtain a second deep feature, and obtaining a second deep feature map based on the second deep feature;

[0040] Swapping the reference image and the neighbor image to input the input position of the pose network, and processing the reference image and the neighbor image through the pose network to obtain a second pose feature;

[0041] Inputting the second depth feature and the second pose feature into a fusion network to obtain a second fusion feature, and obtaining a second relative pose based on the second fusion feature;

[0042] generating a pose consistency loss value based on the first relative pose and the second relative pose;

[0043] The training of the relative pose estimation network based on the loss value includes:

[0044] Generate a target loss value according to at least one of the edge-based first depth feature map smoothing loss value and the first relative pose-determined pose consistency loss value and the photometric error value;

[0045] The deep network, the pose network, the fusion network, and the estimation network are trained based on the target loss value.

[0046] In one embodiment, before extracting the relative pose features of the reference image and the neighboring image using the pose network to obtain the first pose feature, the method further includes:

[0047] Get the initial image;

[0048] Determining a rotation matrix, wherein the rotation matrix is ​​generated based on a randomly obtained target rotation axis and a target rotation angle;

[0049] Rotate the initial image based on the rotation matrix to obtain a rotated image;

[0050] Inputting the initial image and the rotated image into a pose network to obtain a relative rotation;

[0051] The pose network is pre-trained based on the relative rotation and the rotation matrix.

[0052] In one of the embodiments, determining a reference image and a neighboring image of the reference image comprises:

[0053] Determine a sample reference panoramic image and a sample neighboring panoramic image corresponding to the sample reference panoramic image;

[0054] Performing perspective projection on the sample reference panoramic image to obtain a reference image;

[0055] Perform perspective projection on the sample neighbor panoramic image to obtain a neighbor image.

[0056] In a second aspect, the present application also provides a relative pose estimation method, the method comprising:

[0057] Determine a reference panoramic image and neighboring panoramic images of the reference panoramic image;

[0058] Performing perspective projection on the reference panoramic image to obtain a reference image to be processed, and performing perspective projection on the neighboring panoramic image to obtain a neighboring image to be processed;

[0059] Inputting the reference image to be processed into a deep network in a trained relative pose estimation network to obtain a third deep feature;

[0060] Inputting the reference image to be processed and the neighboring image to be processed into the pose network in the trained relative pose estimation network to obtain a third pose feature;

[0061] The third depth feature and the third pose feature are input into the fusion network in the trained relative pose estimation network to obtain a third fusion feature, and the third fusion feature is processed by the estimation network in the trained relative pose estimation network to obtain an estimated relative pose carrying scale information.

[0062] In one embodiment, inputting the reference image and the neighboring image into a pose network in a trained relative pose estimation network to obtain a third pose feature includes:

[0063] Extracting a fifth feature corresponding to the reference image through a first feature extraction module of the pose network, and extracting a sixth feature corresponding to the neighboring image through a second feature extraction module of the pose network; the first feature extraction module and the second feature extraction module share weights;

[0064] Through the decoder of the posture network, attention processing is performed based on the fifth feature and the sixth feature to obtain a third posture feature; wherein the decoder includes a self-attention layer and a cross-attention layer.

[0065] In one embodiment, the step of inputting the third depth feature and the third pose feature into a fusion network in the trained relative pose estimation network to obtain a third fusion feature comprises:

[0066] The third posture feature is processed by the self-attention layer of the fusion network, and the processed third posture feature and the third depth feature are fused by the cross-attention layer of the fusion network to obtain a third fusion feature.

[0067] In one embodiment, the processing of the third fused feature by an estimated network in the trained relative pose estimation network to obtain an estimated relative pose includes:

[0068] Processing the first fused feature through a first estimation network to obtain a second relative translation;

[0069] Processing the first fused feature through a second estimation network to obtain a second relative rotation;

[0070] A first relative position and posture is obtained based on the second relative translation and the second relative rotation.

[0071] In a third aspect, the present application further provides a method for estimating an absolute position and posture of a camera, the method comprising:

[0072] Obtaining an estimated relative pose of each element carrying scale information obtained by the relative pose estimation method described in any one of the above embodiments;

[0073] Obtaining an absolute rotation based on the estimated relative pose;

[0074] Constructing an optimization model based on the estimated relative pose, the absolute rotation and the average translation constraints, and obtaining constraint conditions of the optimization model;

[0075] Iteratively solving the optimization model based on the constraint conditions to obtain an absolute translation;

[0076] The absolute position and posture of the camera is obtained based on the absolute translation and the absolute rotation.

[0077] In one embodiment, constructing an optimization model based on the estimated relative pose and the translation average constraint includes:

[0078] Based on the estimated relative pose and translation average constraint, the following optimization model is constructed:

[0079]

[0080] in, is a set of image pairs, t i is the translation vector from the world coordinate system to the i-th camera, that is, the absolute translation of the i-th camera, t j is the translation vector from the world coordinate system to the i-th camera, that is, the absolute translation of the j-th camera, t ij is the translation vector from the i-th camera to the j-th camera, λ ij is the scale of the relative translation to be obtained, s ij is the initial scale carried by the first relative translation of the estimated network output, R ij is the second relative rotation obtained from the estimation network, is the weight of each error, and δ is a fixed value.

[0081] In one embodiment, the obtaining of the constraint condition of the optimization model includes at least one of the following:

[0082] Determining a value range of a scale of the relative translation to be obtained based on an initial scale carried by the first relative translation, and obtaining a first constraint condition based on the value range;

[0083] A position of at least one of the cameras is determined as a second constraint.

[0084] In a fourth aspect, the present application further provides a relative pose estimation network training device, wherein the relative pose estimation network includes a deep network, a pose network, a fusion network and an estimation network, and the device includes:

[0085] A first image determination module, used to determine a reference image and neighboring images of the reference image;

[0086] A first depth feature extraction module, configured to extract features of the reference image using the deep network to obtain a first depth feature;

[0087] A first pose feature extraction module, used to extract relative pose features of the reference image and the neighboring image using the pose network to obtain a first pose feature;

[0088] A first fusion module, configured to input the first depth feature and the first posture feature into the fusion network to obtain a first fusion feature, and process the first fusion feature through an estimation network to obtain a first relative posture;

[0089] A training module is used to determine a loss value based on the first relative pose, and to estimate the relative pose network based on the loss value.

[0090] In a fifth aspect, the present application further provides a relative pose estimation device, the device comprising:

[0091] A second image determination module is used to determine a reference panoramic image and a neighboring panoramic image of the reference panoramic image; perform perspective projection on the reference panoramic image to obtain a reference image to be processed, and perform perspective projection on the neighboring panoramic image to obtain a neighboring image to be processed;

[0092] A second deep feature extraction module, used for inputting the reference image to be processed into a deep network in a trained relative pose estimation network to obtain a third deep feature;

[0093] A second pose feature extraction module, used for inputting the reference image to be processed and the neighboring image to be processed into the pose network in the trained relative pose estimation network to obtain a third pose feature;

[0094] The first relative pose estimation module is used to input the third depth feature and the third pose feature into the fusion network in the trained relative pose estimation network to obtain a third fusion feature, and process the third fusion feature through the estimation network in the trained relative pose estimation network to obtain an estimated relative pose carrying scale information. The reference image to be processed and the neighboring image to be processed are input into the pose network in the trained relative pose estimation network to obtain the third pose feature.

[0095] In a sixth aspect, the present application further provides a camera absolute pose estimation device, the device comprising:

[0096] A second relative pose estimation module, used to obtain the estimated relative poses of each carrying scale information obtained by the relative pose estimation device described in any one of the above;

[0097] an absolute rotation determination module, configured to obtain an absolute rotation based on the estimated relative pose;

[0098] A model building module, used to build an optimization model based on the estimated relative pose, the absolute rotation and the average translation constraints, and obtain the constraint conditions of the optimization model;

[0099] An absolute translation determination module, used for iteratively solving the optimization model based on the constraint conditions to obtain an absolute translation;

[0100] The camera absolute position and posture determination module is used to obtain the camera absolute position and posture based on the absolute translation and the absolute rotation.

[0101] In a seventh aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0102] In an eighth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above method when executed by a processor.

[0103] In a ninth aspect, the present application also provides a computer program product, comprising a computer program, which implements the steps of the above method when executed by a processor.

[0104] The above-mentioned relative pose estimation network training method, relative pose estimation method, camera absolute pose estimation method, device, computer equipment, computer-readable storage medium and computer program product extract the first depth feature of the reference image through a deep network, and also process the reference image and the neighboring image through the pose network to obtain the first pose feature, and then fuse the first depth feature and the first pose feature to obtain the first fused feature, thereby adding the depth feature to the pose estimation, and then processing the first fused feature through the estimation network to obtain the first relative pose, which can introduce depth information into the estimation of the relative pose, thereby obtaining a relative pose with scale, thereby improving the accuracy of the relative pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0106] Figure 1 A diagram showing an application environment of a relative pose estimation network training method in one embodiment;

[0107] Figure 2 Schematic diagram of a flow chart of a relative pose estimation network training method in one embodiment;

[0108] Figure 3 is a schematic diagram of the network structure of a relative pose estimation network in one embodiment;

[0109] Figure 4 A schematic diagram of the structure of a deep network in one embodiment;

[0110] Figure 5 Schematic diagram of the structure of a posture network in one embodiment;

[0111] Figure 6 A schematic diagram of the structure of a fusion network in an embodiment;

[0112] Figure 7 is a schematic diagram of perspective projection in one embodiment;

[0113] Figure 8 1 is a flow chart of a relative pose estimation method in one embodiment;

[0114] Fig. 9 1 is a flow chart of a method for estimating absolute pose of a camera in one embodiment;

[0115] Fig.10 is a structural block diagram of a relative pose estimation network training device in one embodiment;

[0116] Fig.11 is a structural block diagram of a relative posture estimation device in one embodiment;

[0117] Fig.12 is a structural block diagram of a camera absolute pose estimation device in one embodiment;

[0118] Fig.13 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0119] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0120] The relative pose estimation network training method, relative pose estimation method, and camera absolute pose estimation method provided in the embodiments of the present application can be applied to Figure 1 In the application environment shown.

[0121] Each camera can capture panoramic images and send them to the background for processing. The background is relative to the camera and can be a terminal or a server. For example, in the field of autonomous driving, the background can be a vehicle terminal, which is not specifically limited here.

[0122] In order to enable the background to obtain the relative pose with scale information when performing calculations to restore the structure, the present application proposes a relative pose estimation network, which estimates the relative pose with scale between two panoramic images.

[0123] The relative pose estimation network includes a depth network, a posture network, a fusion network and an estimation network. The relative pose estimation network can be used to introduce depth information into the estimation of the relative pose, thereby obtaining a relative pose with scale.

[0124] In this application, the relative pose estimation network is trained in a self-supervised manner, and the relative pose between panoramic images is estimated by the trained relative pose estimation network. Subsequently, the SfM process is obtained based on the relative pose. For details, please refer to the following.

[0125] The camera may be located in various terminals 102, and the terminals 102 may be, but are not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, projection devices, smart vehicle terminals, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0126] In an exemplary embodiment, Figure 2 As shown in FIG, a relative pose estimation network training method is provided, and the method is applied to Figure 1 Taking the server in as an example, the relative pose estimation network includes a depth network, a pose network, a fusion network and an estimation network, including the following steps 202 to 210. Among them:

[0127] S202: Determine a reference image and neighboring images of the reference image.

[0128] The reference image is obtained by projecting the sample reference panoramic image, and the neighbor image is obtained by projecting the sample neighbor global image, wherein the sample reference panoramic image and the sample neighbor panoramic image are acquired by cameras that are adjacent or close in distance.

[0129] In order to solve the problem of global image distortion, the present application pre-perspectively projects each sample reference panoramic image and each sample neighboring panoramic image. The specific perspective projection method can be selected based on needs, and is not specifically limited in the present application, but for clarity, a perspective projection method is given below.

[0130] S204: Using a deep network to extract features of a reference image to obtain a first deep feature.

[0131] Combination Figure 3 As shown, Figure 3 The figure is a schematic diagram of the network structure of a relative pose estimation network in an embodiment, wherein the relative pose estimation network includes a depth network, a pose network, a fusion network and an estimation network, wherein the depth network is used to extract the depth features of the reference image.

[0132] In some optional embodiments, the deep network can be based on the transformer structure and combined with different self-attention layers. In other embodiments, the structure of the deep network can be other, which is not specifically limited here. In this application, since the purpose is to obtain a more accurate relative pose, a lightweight network structure is selected as much as possible to avoid the problem that a complex network structure causes the deep network to fall into a local optimum during training.

[0133] Specifically, combined Figure 4 As shown, Figure 4 The figure is a schematic diagram of the structure of a deep network in an embodiment. In this embodiment, the input reference image is first converted into vectors through patch embedding, and then deep features are obtained through multiple consecutive self-attention layers. The obtained deep features are input into the fusion network to assist the pose network in relative pose estimation.

[0134] S206: Using the pose network to extract relative pose features of the reference image and the neighboring image to obtain the first pose feature.

[0135] The pose network is used to extract the pose features corresponding to the reference image and the neighboring images. In some optional embodiments, the pose network can be based on the transformer structure and combine different self-attention layers and cross-attention layers. In other embodiments, the structure of the pose network can be other, which is not specifically limited here.

[0136] Combination Figure 5 As shown, Figure 5The figure is a schematic diagram of the structure of a pose network in an embodiment. In this embodiment, the input of the pose network is two images. The pose network includes an encoding module and a decoder, wherein the encoding module includes a first feature extraction module and a second feature extraction module. The first feature extraction module and the second feature extraction module are respectively used to extract image features of a reference image and a neighboring image. The first feature extraction module and the second feature extraction module are similar to a deep network, and each extracts features through a self-attention layer. The decoder is used to fuse the features extracted by the first feature extraction module and the second feature extraction module to obtain pose features. The decoder includes several self-attention layers and cross-attention layers. The first feature extraction module and the second feature extraction module share weights, thereby improving the training speed and the generalization of the network.

[0137] In one of the optional embodiments, a pose network is used to extract relative pose features of a reference image and a neighboring image to obtain a first pose feature, including: extracting a first feature corresponding to the reference image through a first feature extraction module of the pose network, and extracting a second feature corresponding to the neighboring image through a second feature extraction module of the pose network; the first feature extraction module and the second feature extraction module share weights; and obtaining a first pose feature by performing attention processing based on the first feature and the second feature through a decoder of the pose network; wherein the decoder includes a self-attention layer and a cross-attention layer.

[0138] Specifically, the first feature extraction module and the second feature extraction module are parallel feature extraction modules, which do not interfere with each other. The reference image is passed through the first feature extraction module to obtain the first feature, and the neighboring image is passed through the second feature extraction module to obtain the second feature, and then the first feature and the second feature are processed by attention to obtain the posture feature.

[0139] S208: Input the first depth feature and the first pose feature into the fusion network to obtain a first fusion feature, and process the first fusion feature through the estimation network to obtain a first relative pose.

[0140] The fusion network is used to fuse the depth features and the pose features to add the depth features to the pose features, so that the relative pose with scale can be obtained later.

[0141] In some optional embodiments, the fusion network can be based on the transformer structure, combining different self-attention layers and cross-attention layers. In other embodiments, the structure of the fusion network can be other.

[0142] Specifically, combined Figure 6 As shown, Figure 6It is a structural diagram of a fusion network in an embodiment. In one optional embodiment, the first depth feature and the first pose feature are input into the fusion network to obtain a first fusion feature, including: processing the first pose feature through the self-attention layer of the fusion network, and fusing the processed first pose feature and the first depth feature through the cross-attention layer of the fusion network to obtain the first fusion feature.

[0143] In this embodiment, the fusion network includes multiple block coding modules, which correspond to the N layers in the deep network respectively, so as to block encode the N deep features input by the deep network and then input them into the cross-attention layer. The posture features are first extracted through the self-attention layer and then input into the cross-attention layer for fusion with the deep features. The fused features output by each cross-attention layer are used as the posture features input by the next self-attention layer.

[0144] In some optional embodiments, the estimation network may be based on the MLP structure. In other embodiments, the structure of the estimation network may be other.

[0145] In one of the optional embodiments, the first relative pose is obtained by processing the first fused feature through an estimation network, including: processing the first fused feature through a first subnetwork of the first estimation network to obtain a first relative translation; processing the first fused feature through a second subnetwork of the second estimation network to obtain a first relative rotation; and obtaining the first relative pose based on the first relative translation and the first relative rotation.

[0146] Combination Figure 3 As shown, the estimation network in the present application includes a first subnetwork and a second subnetwork, the first subnetwork is used to process the first fused feature to obtain a first relative translation, and the second subnetwork is used to process the first fused feature to obtain a first relative rotation.

[0147] The estimation network may be an MLP network, each of which includes 2 hidden layers, and each hidden layer has 1024 nodes. The difference between them is that the MLP output for estimating relative rotation is 9-dimensional because the rotation matrix contains 9 numbers, while the MLP output for estimating relative translation is 3-dimensional.

[0148] S210: Determine a loss value based on the first relative pose, and train a relative pose estimation network based on the loss value.

[0149] Among them, the loss value is determined based on the first relative pose of the output. In this application, after the loss value is determined, the deep network, pose network, fusion network and estimation network are trained to obtain the trained deep network, pose network, fusion network and estimation network, and these four networks are used as relative pose estimation networks.

[0150] The loss value in the present application may include a photometric loss value. In other embodiments, the loss value may also include at least one of a first depth feature map smoothing loss value based on an edge and a pose consistency loss value determined based on the first relative pose.

[0151] Optionally, the loss value is a weighted combination of a photometric loss value, a first depth feature map smoothing loss value based on edges, and a pose consistency loss value determined based on the first relative pose.

[0152] The above-mentioned relative pose estimation network training method extracts the first depth feature of the reference image through the deep network, and also processes the reference image and the neighboring image through the pose network to obtain the first pose feature, and then fuses the first depth feature and the first pose feature to obtain the first fused feature, thereby adding the depth feature to the pose estimation, and then processing the first fused feature through the estimation network to obtain the first relative pose, which can introduce depth information into the estimation of the relative pose, thereby obtaining a relative pose with scale, thereby improving the accuracy of the relative pose estimation.

[0153] In one of the optional embodiments, the method also includes: decoding based on the first depth feature to obtain a first depth feature map; determining a loss value based on the first relative pose, including: based on the first depth feature map and the first relative pose, projecting the reference image onto the neighboring image to obtain a target image, and obtaining a loss value based on a photometric error value between the target image and the reference image.

[0154] Among them, combined Figure 3 As shown, Figure 3 The deep features obtained by the deep network processing in the image are processed through two branches. One branch obtains the first deep feature map and mask through decoding. The decoding can be performed by a decoder of deconvolution and upsampling. The other branch enters the fusion network to assist the pose network in estimating the relative pose.

[0155] The first deep feature map is used to generate a loss value, and the training of the model is modeled as an optimization problem of minimizing the breadth error to achieve self-supervised training.

[0156] Specifically, based on the first depth feature map and the first relative pose, projecting the reference image onto the neighboring image to obtain the target image includes:

[0157] For the reference image I r For each pixel point p on the image, it can be projected to the target image using the following formula:

[0158] p′=π -1 (Rπ(p,d)+t)

[0159] Where (R, t) represents the relative pose between the reference image and the target image, obtained by the pose network, d represents the depth of the pixel, obtained by the depth network, that is, the depth value of each pixel in the first depth feature map. π(*) represents the transformation of a pixel from the image coordinate system to the camera coordinate system, π -1 (*) is the inverse transformation, that is, from the camera coordinate system to the image coordinate system. Since the projected p′ coordinate is not necessarily an integer, bilinear interpolation is used to calculate the color of each pixel based on the target image. When all pixels on the reference image are projected to the target image, the projected image, i.e., the target image I′, can be obtained. r Then calculate the target image I′ r and reference image I r The photometric error between them is taken as the loss value.

[0160] In one of the optional embodiments, the method further includes: decoding based on the first depth feature to obtain a mask, the mask is used to represent the weight of each pixel in the reference image, and the weight of the pixel at the location of the fixed object in the reference image is higher than the weight of the pixel of the moving object and the textureless area in the reference image. Based on the photometric error value between the target image and the reference image, it also includes: based on the pixel block corresponding to each pixel in the reference image and the first error weight, obtaining the SSIM error of each pixel; based on the corresponding pixels of the reference image and the target image and the second error weight, obtaining the L1 norm error of each pixel; based on the SSIM error, the L1 norm error and the mask, obtaining the photometric error value of the target image and the reference image.

[0161] The photometric error value of the present application includes SSIM error and L1 norm error. The specific expression of the photometric error value is:

[0162]

[0163] Among them, L ssim L1(p) and L1(p) represent the L1 norm error and SSIM error at the pixel point p, respectively. Their specific forms are:

[0164]

[0165] L1(p)=(1-α)‖I r (p)-I′ r (p) ‖1

[0166] Among them, W p represents the pixel block centered at pixel p in the reference image, W p′ represents a pixel block centered at pixel point p in the target image, and α is a weight used to balance the two errors, which is generally set to 0.85, and can also be set to other values ​​in other embodiments. SSIM(W p ,W p ′) is used to find the similarity between two pixel blocks.

[0167] M(p) represents the mask value of the pixel point p, and also represents the weight of different pixels. For moving objects and textureless areas, the weight will be relatively low; for fixed objects in the scene, the weight will be relatively high. In this way, the impact of moving objects on pose estimation can be reduced. The mask value is obtained by decoding the depth feature.

[0168] In the above embodiment, the mask value is a number in the range of [0-1], which represents the weight of each pixel when calculating the final loss. The weights of textureless areas and dynamic objects are lower, and the weights of static geometric structures in the scene are higher. The SSIM error and L1 norm error are weighted by the mask value to obtain an accurate photometric error value to achieve self-supervised training.

[0169] In one of the optional embodiments, the method further includes: determining at least one of an edge-based first depth feature map smoothing loss value based on the first depth feature map and a pose consistency loss value based on the first relative pose.

[0170] Thus, the relative pose estimation network is trained based on the loss value, including: generating a target loss value based on at least one of the edge-based first depth feature map smoothing loss value and the pose consistency loss value determined based on the first relative pose and the photometric error value; and training the relative pose estimation network based on the target loss value.

[0171] Among them, in order to make the trained network more accurate, this embodiment also introduces at least one of a first depth feature map smoothing loss value based on the edge and a posture consistency loss value determined based on the first relative posture.

[0172] The first depth feature map smoothing loss value based on the edge can make the depth feature map smoother and reduce the error. The pose consistency loss value determined based on the first relative pose can improve the accuracy of relative pose estimation.

[0173] The target loss value is the sum of the three loss values ​​and can be expressed as:

[0174]

[0175] The λ e and λ c Used to balance the losses of different items, is the target loss value, is the photometric error value, is the edge-based smoothing loss value of the first deep feature map, is the pose consistency loss value.

[0176] Optionally, a method for obtaining an edge-based depth map smoothing loss value includes: regularizing the first depth feature map to obtain a regularized first depth feature map; and weighting the gradient of the first depth feature map based on the gradient of the regularized first depth feature map to obtain an edge-based first depth map smoothing loss value.

[0177] The specific form of the edge-based first depth map smoothing loss value may be:

[0178]

[0179] Where D r The first depth feature map representing the reference image, Representative D r Note that only the depth of the reference image needs to be calculated, not the depth of the neighboring image. The meaning of this loss is to first regularize the first depth feature map corresponding to the reference image to obtain Then, the gradient of the first depth feature map is weighted according to the gradient of the reference image. The first depth feature map is required to be smooth at the location where the reference image changes slowly, and the first depth feature map is allowed to have sudden changes in the area where the color of the reference image changes dramatically. and represents the gradient of different directions of the reference image and the first depth feature map. Specifically, To find the gradient in the x direction of the first depth feature map after regularization, To find the gradient of the reference image in the x direction, To find the gradient in the y direction of the first depth feature map after regularization, To find the gradient in the y direction of the reference image.

[0180] In the above embodiment, the first depth feature map is required to be smooth at locations where the reference image changes smoothly, and the first depth feature map is allowed to mutate in areas where the color of the reference image changes dramatically. By introducing an edge-based depth feature map smoothing loss, the depth feature map is made smoother and the error is reduced.

[0181] The method for obtaining the pose consistency loss value includes: inputting the neighbor image into the deep network to obtain the second depth feature, and obtaining the second depth feature map based on the second depth feature; exchanging the reference image and the neighbor image to input the input position of the pose network, and processing the reference image and the neighbor image through the pose network to obtain the second pose feature; inputting the second depth feature and the second pose feature into the fusion network to obtain the second fusion feature, and obtaining the second relative pose based on the second fusion feature; generating the pose consistency loss value based on the first relative pose and the second relative pose.

[0182] The second relative pose is obtained by exchanging the reference image and the neighboring image. Assuming the first relative pose is (R1, t1), the two images are swapped and then input into the network, and the relative pose obtained is the second relative pose (R2, t2). In theory, these two relative poses should be exactly opposite, and the pose consistency loss is used to penalize this inconsistency. It can be expressed as:

[0183]

[0184] Where β is used to balance the two losses, ‖*‖ F represents the F norm of the matrix, R represents rotation, and t represents translation.

[0185] Introducing posture consistency loss in the above embodiment can improve the accuracy of posture consistency.

[0186] In one of the optional embodiments, before using the pose network to extract the relative pose features of the reference image and the neighboring image to obtain the first pose feature, it also includes: obtaining an initial image; determining a rotation matrix, which is generated based on a randomly obtained target rotation axis and a target rotation angle; rotating the initial image based on the rotation matrix to obtain a rotated image; inputting the initial image and the rotated image into the pose network to obtain a relative rotation; and pre-training the pose network based on the relative rotation and the rotation matrix.

[0187] In this embodiment, the pose network is first pre-trained for rotation, so that the pose estimation accuracy under large viewing angle differences can be improved by randomly rotating the image.

[0188] Specifically, in practical applications, the camera may be in a handheld shooting state, in which case the camera may rotate rapidly, resulting in large differences in viewing angles between adjacent images. When using ordinary pinhole images, this will result in too small a coverage area between images, making it impossible to correctly estimate the pose. Panoramic cameras are more robust to such rapid changes in viewing angle due to their omnidirectional field of view. However, due to such rapid changes, there will be large differences between images. During the network training phase, the estimates of depth and pose are inaccurate, and large image differences will cause network training to fall into a local optimum. In order to solve this problem, this embodiment provides a rotation pre-training strategy that utilizes the characteristics of panoramic images, specifically including:

[0189] For an initial image I r , the initial image is a panoramic image, and a rotation axis (r x ,r y ,r z ) and a rotation angle r d , we can get a rotation matrix R * The initial image can be rotated by this rotation matrix, and the initial image and the rotated image can be input into the pose network to estimate their relative rotation. Assuming that the estimated result is, that is, the relative rotation is R, we can use ‖RR * ‖ F As a loss to supervise the entire pose network, it can find matches between images under different perspectives. In most cases, the camera is generally rotated around the Y axis, so the constraint condition |r y |≥|r x |+|r z |.

[0190] In the above embodiment, the pose estimation accuracy under large viewing angle differences is improved by performing rotation pre-training on the pose network.

[0191] In one of the optional embodiments, determining a reference image and a neighboring image of the reference image includes: determining a sample reference panoramic image and a sample neighboring panoramic image corresponding to the sample reference panoramic image; performing perspective projection on the sample reference panoramic image to obtain a reference image; performing perspective projection on the sample neighboring panoramic image to obtain a neighboring image.

[0192] The image processed by the network in this application is a pinhole image, so the panoramic image needs to be projected first. The panoramic image is not used directly because most of the current visual pre-training models are trained on pinhole images. These pre-trained networks do not work well for panoramic images. Therefore, in order to use the existing pre-trained network model, the panoramic image is transformed into a pinhole image in this embodiment, such as Figure 7To convert the entire panoramic image into multiple perspective images, it is necessary to select multiple different viewing directions. For ease of understanding, this embodiment provides a method for generating a perspective image under a specified viewing direction, which specifically includes:

[0193] First, specify the field of view (FOV) of the perspective image, which ranges from [0°-360°] and is represented by f here.

[0194] Second, specify the resolution of the perspective image in pixels. Since the image is square, S is used here to represent the image size.

[0195] Third, specify the orientation of the perspective image center, that is, the coordinates of the central pixel in the spherical coordinate system of the panoramic image (θ c ,φ c ). where θ c Represents longitude, the value range is θ c ∈[-π,π],φ c Represents latitude, the value range is

[0196] Fourth, for each pixel in the image, assuming its coordinates are (x, y), transform its coordinates to obtain:

[0197] Fifth, calculate the angle between the pixel and the ray emitted from the center of the image c=arctanρ

[0198] Sixth, calculate the coordinates of the pixel in the spherical coordinate system

[0199]

[0200] Seventh, convert the spherical coordinates of the pixel into the pixel coordinates of the panoramic image, where W and H are the width and height of the panoramic image:

[0201]

[0202] Eighth, bilinear interpolation is used to obtain the pixel color according to the pixel coordinate position. Then steps 4 to 7 are repeated for all pixels to obtain the entire perspective projection image.

[0203] In the above embodiment, perspective projection images are used to solve the distortion problem of panoramic images, thereby making full use of the existing pre-trained model.

[0204] In an exemplary embodiment, Figure 8 As shown, a relative pose estimation method is provided, which is applied to Figure 1 The server in the example is used to illustrate, including the following steps 802 to 810. Among them:

[0205] S802: Determine a reference panoramic image and neighboring panoramic images of the reference panoramic image.

[0206] The reference panoramic image and the neighboring panoramic image are acquired by cameras that are adjacent or close in distance.

[0207] S804: Perform perspective projection on the reference panoramic image to obtain a reference image to be processed, and perform perspective projection on the neighboring panoramic image to obtain a neighboring image to be processed.

[0208] In order to solve the distortion problem of panoramic images, in the present application, the reference panoramic image is projected to obtain the reference image to be processed, and the neighboring panoramic image is projected to obtain the neighboring image to be processed. Optionally, the reference panoramic image is perspective projected to obtain the reference image to be processed, and the neighboring panoramic image is perspective projected to obtain the neighboring image to be processed, including: perspective projecting the reference panoramic image to obtain the reference image to be processed; perspective projecting the neighboring panoramic image to obtain the neighboring image to be processed.

[0209] The specific perspective projection method can be found above.

[0210] S806: Input the reference image to be processed into the deep network in the trained relative pose estimation network to obtain a third deep feature.

[0211] S808: Input the reference image to be processed and the neighboring image to be processed into the pose network in the trained relative pose estimation network to obtain a third pose feature.

[0212] S810: Input the third depth feature and the third pose feature into the fusion network in the trained relative pose estimation network to obtain the third fusion feature, and process the third fusion feature through the estimation network in the trained relative pose estimation network to obtain the estimated relative pose carrying scale information. The reference image to be processed and the neighboring image to be processed are input into the pose network in the trained relative pose estimation network to obtain the third pose feature.

[0213] Among them, the specific limitations of the deep network, pose network, fusion network and estimation network can be found above and will not be repeated here.

[0214] In one of the optional embodiments, the reference image to be processed and the neighboring image to be processed are input into a pose network in a trained relative pose estimation network to obtain a third pose feature, including: extracting the fifth feature corresponding to the reference image through a first feature extraction module of the pose network, and extracting the sixth feature corresponding to the neighboring image through a second feature extraction module of the pose network; the first feature extraction module and the second feature extraction module share weights; and obtaining the third pose feature by performing attention processing based on the fifth feature and the sixth feature through a decoder of the pose network; wherein the decoder includes a self-attention layer and a cross-attention layer.

[0215] In one of the optional embodiments, the third depth feature and the third pose feature are input into a fusion network in a trained relative pose estimation network to obtain a third fusion feature, including: processing the third pose feature through a self-attention layer of the fusion network, and fusing the processed third pose feature and the third depth feature through a cross-attention layer of the fusion network to obtain the third fusion feature.

[0216] In one of the optional embodiments, the third fused feature is processed through an estimation network to obtain an estimated relative pose, including: processing the first fused feature through a first subnetwork of the estimation network to obtain a second relative translation; processing the first fused feature through a second subnetwork of the estimation network to obtain a second relative rotation; and obtaining the first relative pose based on the second relative translation and the second relative rotation.

[0217] The above relative pose estimation method can introduce depth information into the relative pose estimation, thereby obtaining a relative pose with scale, thereby improving the accuracy of relative pose estimation.

[0218] In an exemplary embodiment, Fig. 9 As shown, a camera absolute pose estimation method is provided, which is applied to Figure 1 The server in the example is used to illustrate, including the following steps 902 to 910. Among them:

[0219] S902: Obtain the estimated relative poses of each component carrying scale information obtained based on the relative pose estimation method in any one of the above embodiments.

[0220] S904: Obtaining absolute rotation based on the estimated relative pose.

[0221] S906: constructing an optimization model based on the estimated relative pose, absolute rotation, and average translation constraints, and obtaining constraint conditions of the optimization model.

[0222] The translation average constraint is the constraint followed by the translation average in the absence of noise, namely:

[0223] c j -ci =d ij R i v ij

[0224] Among them, c i and c j is the camera center, R i is the rotation from the ith camera to the world coordinates, i.e., the absolute rotation, v ij is the unit vector representing the relative translation from the jth camera to the ith camera, d ij =‖c j -c i ‖2 is the scale of the relative translation.

[0225] Since there is noise in reality, we need to minimize the difference between the two, that is, minimize c j -c i -d ij R i v ij , and convert it to get t i -R ij t j -λ ij t ij , where t i is the translation vector from the world coordinate system to the i-th camera, that is, the absolute translation of the image, λ ij is the scale of the relative translation, R ij is a relative rotation.

[0226] Optionally, constructing an optimization model based on the estimated relative pose and the average translation constraint includes: constructing the following optimization model based on the estimated relative pose and the average translation constraint:

[0227]

[0228] in, is a set of image pairs, t i is the translation vector from the world coordinate system to the i-th camera, that is, the absolute translation of the i-th camera, t j is the translation vector from the world coordinate system to the i-th camera, that is, the absolute translation of the j-th camera, t ij is the translation vector from the i-th camera to the j-th camera, λ ij is the scale of the relative translation to be obtained, s ij is the initial scale carried by the first relative translation of the estimated network output, R ij is the second relative rotation obtained from the estimation network, is the weight of each error, which is the target residual, and δ is a fixed value.

[0229] Obtaining the constraint conditions of the optimization model includes at least one of the following: determining a value range of the scale of the relative translation to be obtained based on an initial scale carried by the first relative translation, and obtaining a first constraint condition based on the value range; determining the position of at least one of the cameras as a second constraint condition.

[0230] The first constraint is the range of the relative translation scale to be obtained, that is, st0.5s ij ≤λ ij ≤1.5s ij .

[0231] The second constraint is used to eliminate translation ambiguity, which can be t1 = (0, 0, 0) T . Among them, constraining the position of one of the cameras is to prevent the positions of all cameras from deviating during the optimization process. Because the constraints all come from the relative positions between cameras, and there is no constraint on the absolute positions, as long as the relative positions satisfy the relationship, the absolute positions of the cameras are arbitrary. For example, suppose there are three cameras, and their positions are (0, 0, 0), (1, 1, 0), and (1, 0, 1). Since only the relative position relationship is constrained, the optimization result can be (1000, 1000, 1000), (1001, 1001, 1000), and (1001, 1000, 1001). Therefore, in order to prevent the camera position from deviating too much from the origin of the coordinate system, in this embodiment, the absolute position of one camera is constrained to be (0, 0, 0), which is equivalent to using one point to fix the positions of all cameras.

[0232] S908: Iteratively solving the optimization model based on the constraint conditions to obtain an absolute translation.

[0233] S910: Obtain the absolute position and posture of the camera based on the absolute translation and the absolute rotation.

[0234] After the optimization model and constraints are determined, iterative optimization is performed. The entire optimization process generally iterates 2-3 times before convergence. In other embodiments, other numbers of iterations may also occur. For convenience, the weight is set to w in the first iteration. ij =1.

[0235] In addition, in the present application, only the translational average is improved, and the rotational average can be improved in any manner, which is not specifically limited here.

[0236] In the above embodiment, after the network obtains the relative pose, the global SfM can be used to obtain the absolute pose of the camera. Estimating the absolute pose is divided into two steps, rotation averaging and translation averaging. The rotation averaging uses the existing method, and the translation averaging is optimized by the above optimization model to improve the accuracy.

[0237] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0238] Based on the same inventive concept, the embodiments of the present application also provide a relative pose estimation network training device, a relative pose estimation device, and a camera absolute pose estimation device for implementing the relative pose estimation network training method, relative pose estimation method, and camera absolute pose estimation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more relative pose estimation network training devices, relative pose estimation devices, and camera absolute pose estimation device embodiments provided below can refer to the limitations of the relative pose estimation network training method, relative pose estimation method, and camera absolute pose estimation method above, and will not be repeated here.

[0239] In an exemplary embodiment, Fig.10 As shown, a relative pose estimation network training device is provided, the relative pose estimation network includes a depth network, a pose network, a fusion network and an estimation network, and the relative pose estimation network training device includes: a first image determination module 1001, a first depth feature extraction module 1002, a first pose feature extraction module 1003, a first fusion module 1004 and a training module 1005, wherein:

[0240] A first image determination module 1001 is used to determine a reference image and neighboring images of the reference image;

[0241] A first depth feature extraction module 1002, configured to extract features of a reference image using a deep network to obtain a first depth feature;

[0242] The first pose feature extraction module 1003 is used to extract the relative pose features of the reference image and the neighboring image using the pose network to obtain the first pose feature;

[0243] A first fusion module 1004 is used to input the first depth feature and the first pose feature into a fusion network to obtain a first fusion feature, and process the first fusion feature through an estimation network to obtain a first relative pose;

[0244] The training module 1005 is used to determine a loss value based on the first relative pose, and train the relative pose estimation network based on the loss value.

[0245] In one of the optional embodiments, the above-mentioned first pose feature extraction module 1003 is specifically used to extract the first feature corresponding to the reference image through the first feature extraction module of the pose network, and extract the second feature corresponding to the neighboring image through the second feature extraction module of the pose network; the first feature extraction module and the second feature extraction module share weights; through the decoder of the pose network, attention processing is performed based on the first feature and the second feature to obtain the first pose feature; wherein the decoder includes a self-attention layer and a cross-attention layer.

[0246] In one of the optional embodiments, the above-mentioned first fusion module 1004 is specifically used to process the first pose feature through the self-attention layer of the fusion network, and fuse the processed first pose feature and the first depth feature through the cross-attention layer of the fusion network to obtain the first fusion feature.

[0247] In one of the optional embodiments, the above-mentioned first fusion module 1004 is specifically used to process the first fused feature through the first sub-network of the estimation network to obtain a first relative translation; process the first fused feature through the second sub-network of the estimation network to obtain a first relative rotation; and obtain a first relative posture based on the first relative translation and the first relative rotation.

[0248] In one of the optional embodiments, the above-mentioned device further includes: a depth decoding module, configured to decode based on the first depth feature to obtain a first depth feature map;

[0249] The above-mentioned training module 1005 is specifically used to project the reference image onto the neighboring image to obtain the target image based on the first depth feature map and the first relative pose, and obtain the loss value based on the photometric error value between the target image and the reference image.

[0250] In one of the optional embodiments, the above-mentioned depth decoding module is also specifically used to decode based on the first depth feature to obtain a mask, and the mask is used to represent the weight of each pixel in the reference image, and the weight of the pixel at the location of the fixed object in the reference image is higher than the weight of the pixel of the moving object and the textureless area in the reference image.

[0251] The above-mentioned training module 1005 is specifically used to obtain the SSIM error of each pixel based on the pixel block corresponding to each pixel in the reference image and the first error weight; obtain the L1 norm error of each pixel based on the corresponding pixel points of the reference image and the target image and the second error weight; obtain the photometric error value of the target image and the reference image based on the SSIM error, L1 norm error and the mask.

[0252] In one of the optional embodiments, the training module 1005 is further specifically used to determine at least one of a first depth feature map smoothing loss value based on an edge based on the first depth feature map and a posture consistency loss value based on the first relative posture;

[0253] The training module 1005 includes:

[0254] The depth image smoothing loss value acquisition unit is used to regularize the first depth feature map to obtain a regularized first depth feature map; weight the gradient of the first depth feature map based on the gradient of the regularized first depth feature map to obtain an edge-based first depth map smoothing loss value.

[0255] A pose consistency loss value acquisition unit is used to input the neighbor image into the depth network to obtain a second depth feature, and obtain a second depth feature map based on the second depth feature; exchange the reference image and the neighbor image to input the input position of the pose network, and process the reference image and the neighbor image through the pose network to obtain a second pose feature; input the second depth feature and the second pose feature into the fusion network to obtain a second fusion feature, and obtain a second relative pose based on the second fusion feature; generate a pose consistency loss value based on the first relative pose and the second relative pose.

[0256] A training unit is used to generate a target loss value based on at least one of a first depth feature map smoothing loss value based on an edge and a pose consistency loss value determined based on a first relative pose, and a photometric error value; and train a deep network, a pose network, a fusion network, and an estimation network based on the target loss value.

[0257] In one of the optional embodiments, the above-mentioned device also includes: a rotation pre-training module, used to obtain an initial image; determine a rotation matrix, the rotation matrix is ​​generated based on a randomly obtained target rotation axis and a target rotation angle; rotate the initial image based on the rotation matrix to obtain a rotated image; input the initial image and the rotated image into a pose network to obtain a relative rotation; and pre-train the pose network based on the relative rotation and the rotation matrix.

[0258] In one of the optional embodiments, the above-mentioned first image determination module 1001 is specifically used to determine a sample reference panoramic image and a sample neighbor panoramic image corresponding to the sample reference panoramic image; perform perspective projection on the sample reference panoramic image to obtain a reference image; perform perspective projection on the sample neighbor panoramic image to obtain a neighbor image.

[0259] In an exemplary embodiment, Fig.11 As shown, a relative pose estimation device is provided, including: a second image determination module 1101, a second depth feature extraction module 1102, a second pose feature extraction module 1103 and a first relative pose estimation module 1104, wherein:

[0260] The second image determination module 1101 is used to determine a reference panoramic image and a neighboring panoramic image of the reference panoramic image; perform perspective projection on the reference panoramic image to obtain a reference image to be processed, and perform perspective projection on the neighboring panoramic image to obtain a neighboring image to be processed;

[0261] A second depth feature extraction module 1102 is used to input the reference image to be processed into a deep network in a trained relative pose estimation network to obtain a third depth feature;

[0262] A second pose feature extraction module 1103 is used to input the reference image to be processed and the neighboring image to be processed into a pose network in a trained relative pose estimation network to obtain a third pose feature;

[0263] The first relative pose estimation module 1104 is used to input the third depth feature and the third pose feature into the fusion network in the trained relative pose estimation network to obtain the third fusion feature, and process the third fusion feature through the estimation network in the trained relative pose estimation network to obtain the estimated relative pose carrying scale information. The reference image to be processed and the neighboring image to be processed are input into the pose network in the trained relative pose estimation network to obtain the third pose feature.

[0264] In one of the optional embodiments, the above-mentioned second pose feature extraction module 1103 is specifically used to extract the fifth feature corresponding to the reference image through the first feature extraction module of the pose network, and extract the sixth feature corresponding to the neighboring image through the second feature extraction module of the pose network; the first feature extraction module and the second feature extraction module share weights; through the decoder of the pose network, attention processing is performed based on the fifth feature and the sixth feature to obtain the third pose feature; wherein the decoder includes a self-attention layer and a cross-attention layer.

[0265] In one of the optional embodiments, the above-mentioned second relative pose estimation module 1201 is specifically used to process the third pose feature through the self-attention layer of the fusion network, and to fuse the processed third pose feature and the third depth feature through the cross-attention layer of the fusion network to obtain a third fusion feature.

[0266] In one of the optional embodiments, the above-mentioned second relative pose estimation module 1201 is specifically used to obtain a second relative translation by processing the first fused feature through the first subnetwork of the estimation network; obtain a second relative rotation by processing the first fused feature through the second subnetwork of the estimation network; and obtain the first relative pose based on the second relative translation and the second relative rotation.

[0267] In an exemplary embodiment, Fig.12 As shown, a camera absolute pose estimation device is provided, comprising: a second relative pose estimation module 1201, an absolute rotation determination module 1202, a model building module 120 / 3, an absolute translation determination module 1204 and a camera absolute pose determination module 1205, wherein:

[0268] A second relative pose estimation module 1201 is used to obtain an estimated relative pose carrying scale information obtained by the relative pose estimation device in any of the above embodiments;

[0269] An absolute rotation determination module 1202, configured to obtain an absolute rotation based on the estimated relative pose;

[0270] A model building module 1203 is used to build an optimization model based on the estimated relative pose, absolute rotation and translation average constraints, and obtain the constraint conditions of the optimization model;

[0271] An absolute translation determination module 1204 is used to iteratively solve the optimization model based on the constraint conditions to obtain an absolute translation;

[0272] The camera absolute position and posture determination module 1205 is used to obtain the camera absolute position and posture based on the absolute translation and the absolute rotation.

[0273] In one of the optional embodiments, the model building module 1203 is specifically used to build the following optimization model based on the estimated relative pose and the translation average constraint:

[0274]

[0275] in, is a set of image pairs, t i is the translation vector from the world coordinate system to the i-th camera, that is, the absolute translation of the i-th camera, t jis the translation vector from the world coordinate system to the i-th camera, that is, the absolute translation of the j-th camera, t ij is the translation vector from the i-th camera to the j-th camera, λ ij is the scale of the relative translation to be obtained, s ij is the initial scale carried by the first relative translation of the estimated network output, R ij is the second relative rotation obtained from the estimation network, is the weight of each error, and δ is a fixed value.

[0276] In one of the optional embodiments, the above-mentioned model building module 1203 is specifically used to obtain the constraint conditions of at least one optimization model: determine the value range of the scale of the relative translation to be obtained based on the initial scale carried by the first relative translation, and obtain the first constraint condition based on the value range; determine the position of at least one of the cameras as the second constraint condition.

[0277] Each module in the relative pose estimation network training device, relative pose estimation device, and camera absolute pose estimation device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0278] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.13 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a relative pose estimation network training method, a relative pose estimation method, and a camera absolute pose estimation method are implemented.

[0279] Those skilled in the art will understand that Fig.13The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0280] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0281] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0282] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0283] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0284] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0285] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0286] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A relative pose estimation network training method, characterized in that: The relative pose estimation network includes a depth network, a pose network, a fusion network and an estimation network, and the method includes: Determining a reference image and neighboring images of the reference image; Extracting features of the reference image using the deep network to obtain a first deep feature; Extracting relative pose features of the reference image and the neighboring image using the pose network to obtain a first pose feature; Inputting the first depth feature and the first posture feature into the fusion network to obtain a first fusion feature, and processing the first fusion feature through the estimation network to obtain a first relative posture; A loss value is determined based on the first relative pose, and the relative pose estimation network is trained based on the loss value.

2. The method according to claim 1, characterized in that The step of extracting the relative pose features of the reference image and the neighboring image by using the pose network to obtain the first pose feature includes: Extracting a first feature corresponding to the reference image through a first feature extraction module of the pose network, and extracting a second feature corresponding to the neighboring image through a second feature extraction module of the pose network; the first feature extraction module and the second feature extraction module share weights; Through the decoder of the pose network, attention processing is performed based on the first feature and the second feature to obtain the first pose feature; wherein the decoder includes a self-attention layer and a cross-attention layer.

3. The method according to claim 1, characterized in that The step of inputting the first depth feature and the first posture feature into the fusion network to obtain a first fusion feature includes: The first posture feature is processed by the self-attention layer of the fusion network, and the processed first posture feature and the first depth feature are fused by the cross-attention layer of the fusion network to obtain a first fusion feature.

4. The method according to claim 1, characterized in that The step of processing the first fusion feature by the estimation network to obtain a first relative pose includes: Processing the first fused feature by a first sub-network of the estimation network to obtain a first relative translation; Processing the first fused feature by a second sub-network of the estimation network to obtain a first relative rotation; A first relative position and posture is obtained based on the first relative translation and the first relative rotation.

5. The method according to claim 1, characterized in that The method further comprises: Decoding based on the first depth feature to obtain a first depth feature map; The determining the loss value according to the first relative posture comprises: Based on the first depth feature map and the first relative pose, the reference image is projected onto the neighboring image to obtain a target image, and based on a photometric error value between the target image and the reference image, a loss value is obtained.

6. The method according to claim 5, characterized in that The method further comprises: Decoding based on the first depth feature to obtain a mask, where the mask is used to represent the weight of each pixel in the reference image, and the weight of the pixel at the location of the fixed object in the reference image is higher than the weight of the pixel of the moving object and the textureless area in the reference image; Before the photometric error value between the target image and the reference image, the method further includes: Obtaining an SSIM error of each pixel based on a pixel block corresponding to each pixel in the reference image and a first error weight; Based on the corresponding pixels of the reference image and the target image and the second error weight, an L1 norm error of each pixel is obtained; A photometric error value between the target image and the reference image is obtained based on the SSIM error, the L1 norm error, and the mask.

7. The method according to claim 5, characterized in that The method further comprises: Determine at least one of an edge-based first depth feature map smoothing loss value based on the first depth feature map and a pose consistency loss value based on the first relative pose; The method for obtaining the edge-based first depth map smoothing loss value includes: Regularizing the first depth feature map to obtain a regularized first depth feature map; Weighting the gradient of the first depth feature map based on the gradient of the regular first depth feature map to obtain an edge-based first depth map smoothing loss value; The method for obtaining the pose consistency loss value includes: Inputting the neighbor image into the deep network to obtain a second deep feature, and obtaining a second deep feature map based on the second deep feature; Swapping the reference image and the neighbor image to input the input position of the pose network, and processing the reference image and the neighbor image through the pose network to obtain a second pose feature; Inputting the second depth feature and the second pose feature into a fusion network to obtain a second fusion feature, and obtaining a second relative pose based on the second fusion feature; generating a pose consistency loss value based on the first relative pose and the second relative pose; The training of the relative pose estimation network based on the loss value includes: Generate a target loss value according to at least one of the edge-based first depth feature map smoothing loss value and the first relative pose-determined pose consistency loss value and the photometric error value; The deep network, the pose network, the fusion network, and the estimation network are trained based on the target loss value.

8. The method according to any one of claims 1 to 7, characterized in that: Before the step of using the pose network to extract the relative pose features of the reference image and the neighboring image to obtain the first pose feature, the step further includes: Get the initial image; Determining a rotation matrix, wherein the rotation matrix is ​​generated based on a randomly obtained target rotation axis and a target rotation angle; Rotate the initial image based on the rotation matrix to obtain a rotated image; Inputting the initial image and the rotated image into a pose network to obtain a relative rotation; The pose network is pre-trained based on the relative rotation and the rotation matrix.

9. The method according to any one of claims 1 to 7, characterized in that: The determining of a reference image and a neighboring image of the reference image comprises: Determine a sample reference panoramic image and a sample neighboring panoramic image corresponding to the sample reference panoramic image; Performing perspective projection on the sample reference panoramic image to obtain a reference image; Perform perspective projection on the sample neighbor panoramic image to obtain a neighbor image.

10. A relative pose estimation method, characterized in that: The method comprises: Determine a reference panoramic image and neighboring panoramic images of the reference panoramic image; Performing perspective projection on the reference panoramic image to obtain a reference image to be processed, and performing perspective projection on the neighboring panoramic image to obtain a neighboring image to be processed; Inputting the reference image to be processed into a deep network in a trained relative pose estimation network to obtain a third deep feature; Inputting the reference image to be processed and the neighboring image to be processed into the pose network in the trained relative pose estimation network to obtain a third pose feature; The third depth feature and the third pose feature are input into the fusion network in the trained relative pose estimation network to obtain a third fusion feature, and the third fusion feature is processed by the estimation network in the trained relative pose estimation network to obtain an estimated relative pose carrying scale information.

11. The method according to claim 10, characterized in that The step of inputting the reference image and the neighboring image into a posture network in a trained relative posture estimation network to obtain a third posture feature comprises: Extracting a fifth feature corresponding to the reference image through a first feature extraction module of the pose network, and extracting a sixth feature corresponding to the neighboring image through a second feature extraction module of the pose network; the first feature extraction module and the second feature extraction module share weights; Through the decoder of the posture network, attention processing is performed based on the fifth feature and the sixth feature to obtain a third posture feature; wherein the decoder includes a self-attention layer and a cross-attention layer.

12. The method according to claim 10, characterized in that The step of inputting the third depth feature and the third pose feature into the fusion network in the trained relative pose estimation network to obtain a third fusion feature comprises: The third posture feature is processed by the self-attention layer of the fusion network, and the processed third posture feature and the third depth feature are fused by the cross-attention layer of the fusion network to obtain a third fusion feature.

13. The method according to claim 10, characterized in that The step of processing the third fused feature by an estimation network in the trained relative pose estimation network to obtain an estimated relative pose includes: Processing the first fused feature by a first sub-network of the estimation network to obtain a second relative translation; Processing the first fused feature by a second sub-network of the estimation network to obtain a second relative rotation; A first relative position and posture is obtained based on the second relative translation and the second relative rotation.

14. A camera absolute pose estimation method, characterized in that: The method comprises: Obtaining an estimated relative pose of each carrying scale information obtained by the relative pose estimation method according to any one of claims 10 to 13; Obtaining an absolute rotation based on the estimated relative pose; Constructing an optimization model based on the estimated relative pose, the absolute rotation and the average translation constraints, and obtaining constraint conditions of the optimization model; Iteratively solving the optimization model based on the constraint conditions to obtain an absolute translation; The absolute position and posture of the camera is obtained based on the absolute translation and the absolute rotation.

15. The method according to claim 14, characterized in that The optimization model is constructed based on the estimated relative pose and the translation average constraint, comprising: Based on the estimated relative pose and translation average constraint, the following optimization model is constructed: in, is a set of image pairs, t i is the translation vector from the world coordinate system to the i-th camera, that is, the absolute translation of the i-th camera, t j is the translation vector from the world coordinate system to the i-th camera, that is, the absolute translation of the j-th camera, t ij is the translation vector from the i-th camera to the j-th camera, λ ij is the scale of the relative translation to be obtained, s ij is the initial scale carried by the first relative translation of the estimated network output, R ij is the second relative rotation obtained from the estimation network, is the weight of each error, and δ is a fixed value.

16. The method according to claim 15, characterized in that The obtaining of the constraint condition of the optimization model comprises at least one of the following: Determining a value range of a scale of the relative translation to be obtained based on an initial scale carried by the first relative translation, and obtaining a first constraint condition based on the value range; A position of at least one of the cameras is determined as a second constraint.

17. A relative pose estimation network training device, characterized in that: The relative pose estimation network includes a depth network, a pose network, a fusion network and an estimation network, and the device includes: A first image determination module, used to determine a reference image and neighboring images of the reference image; A first depth feature extraction module, configured to extract features of the reference image using the deep network to obtain a first depth feature; A first pose feature extraction module, used to extract relative pose features of the reference image and the neighboring image using the pose network to obtain a first pose feature; A first fusion module, configured to input the first depth feature and the first posture feature into the fusion network to obtain a first fusion feature, and process the first fusion feature through an estimation network to obtain a first relative posture; A training module is used to determine a loss value based on the first relative pose, and to estimate the relative pose network based on the loss value.

18. A relative pose estimation device, characterized in that: The device comprises: A second image determination module is used to determine a reference panoramic image and a neighboring panoramic image of the reference panoramic image; perform perspective projection on the reference panoramic image to obtain a reference image to be processed, and perform perspective projection on the neighboring panoramic image to obtain a neighboring image to be processed; A second deep feature extraction module, used for inputting the reference image to be processed into a deep network in a trained relative pose estimation network to obtain a third deep feature; A second pose feature extraction module, used for inputting the reference image to be processed and the neighboring image to be processed into the pose network in the trained relative pose estimation network to obtain a third pose feature; The first relative pose estimation module is used to input the third depth feature and the third pose feature into the fusion network in the trained relative pose estimation network to obtain a third fusion feature, and process the third fusion feature through the estimation network in the trained relative pose estimation network to obtain an estimated relative pose carrying scale information. The reference image to be processed and the neighboring image to be processed are input into the pose network in the trained relative pose estimation network to obtain the third pose feature.

19. A camera absolute pose estimation device, characterized in that: The device comprises: A second relative pose estimation module, used to obtain the estimated relative poses of each carrying scale information obtained based on the relative pose estimation device according to claim 18; an absolute rotation determination module, configured to obtain an absolute rotation based on the estimated relative pose; A model building module, used to build an optimization model based on the estimated relative pose, the absolute rotation and the average translation constraints, and obtain the constraint conditions of the optimization model; An absolute translation determination module, used for iteratively solving the optimization model based on the constraint conditions to obtain an absolute translation; The camera absolute position and posture determination module is used to obtain the camera absolute position and posture based on the absolute translation and the absolute rotation.

20. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 16 are implemented.

21. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

22. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

Images