Training method, regression method, system and equipment of attitude regression model

Abstract

The invention relates to a training method of an attitude regression model. The training method comprises the following steps: acquiring a shot sample image and a composite image; the composite image is synthesized based on the shot sample image; respectively inputting the shot sample image and the synthetic image into an attitude regression model to obtain a first predicted attitude, a first image feature, a second predicted attitude and a second image feature; obtaining a first attitude regression loss according to the first predicted attitude and the first real label; obtaining discriminator loss according to the first image feature and the second image feature; obtaining generator loss according to the second image feature; obtaining a second attitude regression loss according to the second prediction attitude and the first real label; and obtaining total loss according to the first attitude regression loss, the discriminator loss, the generator loss and the second attitude regression loss, and optimizing parameters of the attitude regression model by using the total loss to obtain the attitude regression model. The method can improve the accuracy of attitude regression of the model in different scenes, and is applied to the technical field of computer vision.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a training method, regression method, system and equipment for a posture regression model. Background Art

[0002] The main limitation of existing pose regression models is that their positioning accuracy decreases significantly when used in new scenes that differ from those they were trained on. Under varying lighting conditions, such as switching from indoor artificial lighting to outdoor natural lighting, the model may be unable to accurately regress pose due to changes in image features caused by the changing lighting. Large changes in viewpoint can also lead to positioning errors and inability to accurately regress pose. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a training method, regression method, system and device for a posture regression model, which can improve the accuracy of posture regression of the model in different scenarios.

[0004] In one aspect, the present invention provides a method for training a posture regression model, comprising the following steps:

[0005] Acquire a captured sample image and a composite image; the composite image is synthesized based on the captured sample image;

[0006] Inputting the captured sample image into a posture regression model to obtain a first predicted posture and a first image feature; obtaining a first posture regression loss based on the first predicted posture and the first true label;

[0007] Inputting the synthesized image into a pose regression model to obtain a second predicted pose and a second image feature; obtaining a discriminator loss based on the first image feature and the second image feature; and obtaining a generator loss based on the second image feature;

[0008] A second posture regression loss is obtained according to the second predicted posture and the first true label; a total loss is obtained according to the first posture regression loss, the discriminator loss, the generator loss and the second posture regression loss, and the parameters of the posture regression model are optimized using the total loss to obtain a trained posture regression model.

[0009] Optionally, the composite image is synthesized by the following method:

[0010] Extracting at least one Gaussian distribution from the captured sample image; wherein a scene in the image is represented as a Gaussian distribution;

[0011] Adding disturbance to the Gaussian parameters of the Gaussian distribution to obtain new Gaussian parameters, and randomly combining the new Gaussian parameters to obtain a new Gaussian distribution;

[0012] Each new Gaussian distribution is converted into a sample feature, and the sample features are mapped according to the camera posture to obtain the composite image.

[0013] Optionally, the first predicted posture includes a translation posture and a rotation posture of the camera posture; and inputting the captured sample image into a posture regression model to obtain the first predicted posture specifically includes:

[0014] Extracting the output of the first specific layer in the posture regression model as a first sub-feature map, and extracting the output of the second specific layer in the posture regression model as a second sub-feature map; the first feature layer and the second feature layer are different network layers;

[0015] Perform translation regression on the first sub-feature map to obtain the translation posture; perform rotation regression on the second sub-feature map to obtain the rotation posture.

[0016] Optionally, performing translation regression on the first sub-feature map to obtain the translation posture; and performing rotation regression on the second sub-feature map to obtain the rotation posture specifically includes:

[0017] Generate translation tokens and rotation tokens;

[0018] splicing the first sub-feature map and the translation token to obtain a third sub-feature map, generating a first position code according to the third sub-feature map, and obtaining the translation posture according to the third sub-feature map and the first position code;

[0019] The second sub-feature map and the rotation token are concatenated to obtain a fourth sub-feature map, a second position code is generated according to the fourth sub-feature map, and the rotation posture is obtained according to the fourth sub-feature map and the second position code.

[0020] Optionally, the first predicted pose includes a translation pose and a rotation pose of the camera pose, and the first true label includes a translation label and a rotation label; the first pose regression loss is obtained according to the first predicted pose and the first true label, specifically including

[0021] Obtaining a translation loss according to the translation posture and the translation label; obtaining a rotation loss according to the rotation posture and the rotation label;

[0022] A weighted sum is performed on the translation loss and the rotation loss to obtain the first posture regression loss.

[0023] Optionally, obtaining a discriminator loss according to the first image feature and the second image feature specifically includes:

[0024] Inputting the first image feature and the second image feature into an adjustment layer respectively, and obtaining a first adjustment layer output and a second adjustment layer output respectively;

[0025] Inputting the first adjustment layer output and the second adjustment layer output into a discriminator respectively, to obtain a first discriminator output and a second discriminator output respectively;

[0026] The discriminator loss is obtained according to the first discriminator output and the second discriminator output.

[0027] Optionally, obtaining a second posture regression loss according to the second predicted posture and the first true label specifically includes:

[0028] Adding noise to the first true label to obtain a second true label;

[0029] The second posture regression loss is obtained according to the second true label and the second predicted posture.

[0030] On the other hand, the present invention provides a posture regression method, comprising the following steps:

[0031] The image to be predicted is obtained, and the image to be predicted is input into a trained posture regression model to obtain the actual predicted posture; the trained posture regression model is obtained by training using the training method described above.

[0032] On the other hand, the present invention provides a posture regression system, including a training module and a prediction module, wherein:

[0033] The training module is used to implement the training method described above;

[0034] The prediction module is used to implement the posture regression method described above.

[0035] On the other hand, the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the aforementioned method when executing the computer program.

[0036] The implementation of the present invention includes the following beneficial effects: first, a new synthetic image is synthesized on an actual sample image. The sample image and the synthetic image are respectively input into the model. A first pose regression loss is designed based on the pose predicted by the sample image to reduce the domain gap between the synthetic data and the real data. In addition to the final predicted pose, the present invention also considers the impact of the accuracy of the model's extracted image features on pose prediction. The present invention introduces a discriminator loss based on the first image features of the sample image and the second image features of the synthetic image. The discriminator loss is used to make the feature output of the real image 1, the feature output of the synthetic image 0, and the feature output of the generator for the synthetic image 1. The generator loss and the discriminator loss are used to form an adversarial relationship, enabling the model to learn how to extract features similar to the real image from the synthetic image. Finally, a second pose regression loss is designed based on the pose predicted by the synthetic image to further improve the model's generalization ability for unseen image and pose pairs. The aforementioned various losses are combined to form a final total loss, and the model is trained based on the total loss. Through the synergistic effect of adversarial training and multi-level regression loss, the model maintains the accuracy of pose prediction while improving its generalization ability for unknown scenes, thereby improving the model's generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flowchart of the steps of a posture regression model training method provided by the present invention;

[0038] Figure 2 This is a branch flow chart of a training method for a posture regression model provided by the present invention;

[0039] Figure 3 This is a synthesis flow chart of a synthetic image provided by the present invention;

[0040] Figure 4 This is a branch flow chart for obtaining a first predicted posture provided by the present invention;

[0041] Figure 5 This is a flowchart of the steps of posture regression provided by the present invention;

[0042] Figure 6 It is a branch flow chart for calculating the discriminator loss and the generator loss provided by the present invention;

[0043] Figure 7 It is a structural schematic diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are provided for ease of description only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted based on the understanding of those skilled in the art.

[0045] The terms used in the embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit the embodiments of the present application. The singular forms "a," "the," and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0046] When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0047] In some embodiments, as Figure 1-2 As shown, Figure 1 It is a flowchart of the steps of the training method of the posture regression model. Figure 2 The present invention provides a branch flow chart of a training method for a posture regression model, which includes the following steps:

[0048] S100: Acquire a sample image and a composite image.

[0049] The captured sample image represents a real image captured by a camera, and the synthesized image is synthesized based on the captured sample image.

[0050] Methods for synthesizing images include, but are not limited to, 3D Gaussian Splatting (3DGS) or other synthesis methods in the prior art.

[0051] S200: Input the captured sample image into a posture regression model to obtain a first predicted posture and a first image feature; and obtain a first posture regression loss based on the first predicted posture and the first true label.

[0052] The first predicted posture includes a translation posture and a rotation posture; the first true label includes a translation label and a rotation label.

[0053] The posture regression model includes a backbone network and a posture regression network. The backbone network is used to extract image features. The output of the backbone network is a multi-dimensional feature map, namely the first image feature, which usually contains rich local and global information of the image; the posture regression network is used to perform posture regression based on image features to obtain the predicted posture.

[0054] Specifically, the captured sample image is input into the posture regression model, and the feature output by the last layer of the backbone network is used as the first image feature; the features of the captured sample image are extracted by a specific layer of the backbone network as the first sub-feature map and the second sub-feature map, and the posture regression network performs posture regression based on the first sub-feature map and the second sub-feature map to obtain the first predicted posture.

[0055] The translation loss is obtained according to the translation posture and translation label; the rotation loss is obtained according to the rotation posture and rotation label; the translation loss and rotation loss are weighted summed to obtain the first posture regression loss.

[0056] S300: Input the synthesized image into a posture regression model to obtain a second predicted posture and a second image feature; obtain a discriminator loss based on the first image feature and the second image feature; and obtain a generator loss based on the second image feature.

[0057] The second predicted posture includes a translation posture and a rotation posture, which is different from the first predicted posture including the translation posture and the rotation posture in that the second predicted posture is a posture predicted based on the synthetic image.

[0058] The discriminator may be, but is not limited to, a discriminator in the prior art; the generator may be, but is not limited to, a generator in the prior art.

[0059] Specifically, the posture regression model processes the synthetic image to obtain the second predicted posture and the second image features in a process similar to that of obtaining the first predicted posture and the first image features, except that the input is a synthetic image.

[0060] The first image feature and the second image feature are respectively input into the adjustment layer and then into the discriminator, and the discriminator loss is obtained based on the output of the discriminator. The second image feature is input into the generator, and the generator loss is obtained based on the output of the generator.

[0061] S400. Obtain a second posture regression loss based on the second predicted posture and the first true label; obtain a total loss based on the first posture regression loss, the discriminator loss, the generator loss, and the second posture regression loss, and use the total loss to optimize the parameters of the posture regression model to obtain a trained posture regression model.

[0062] Specifically, noise is added to the first true label to obtain a second true label. A second pose regression loss is obtained based on the second predicted pose and the second true label. A weighted sum of the first pose regression loss, the discriminator loss, the generator loss, and the second pose regression loss is performed to obtain a total loss. During the parameter optimization of the pose regression model, the discriminator and generator losses are alternately optimized using the discriminator and generator losses.

[0063] The total loss is used to optimize the parameters of the pose regression model. The optimization method includes but is not limited to back gradient propagation.

[0064] Repeat steps S100-S400, and after every 20 epochs, synthesize a new synthetic image online based on the captured sample image. Using random mixing weights ω, adjust the appearance of the synthetic image to give it a different style and detail when rendered, thereby generating a synthetic image Isyn with a new true pose label Psyn. The synthetic image Isyn is generated by adjusting the rendered image to simulate the appearance changes in different scenes.

[0065] Based on the new synthetic image-true pose label pair (Isyn, Psyn) and the captured sample image, the aforementioned steps S100-S40 are performed until the total loss meets the preset value, the number of iterations reaches the preset number, or the total loss on the validation set no longer decreases, thereby obtaining a trained pose regression model.

[0066] In some embodiments, as Figure 3 As shown, Figure 3 This is a flow chart of synthesizing a synthesized image. In connection with the above embodiment, the synthesized image in step S100 is synthesized by the following method:

[0067] S110 , extracting at least one Gaussian distribution from the captured sample image.

[0068] The captured sample image contains multiple scenes, such as a car or a road. Each scene in the image is represented by a Gaussian distribution. Each Gaussian distribution describes the local area of a scene element and can include several Gaussian parameters, where several represents a value greater than 1. These Gaussian parameters can effectively capture the scene's geometric and appearance information.

[0069] The Gaussian parameters can be, but are not limited to, the position, rotation, scale, color, or opacity of each Gaussian distribution in space. During the image synthesis process, these Gaussian parameters of the captured sample images are adjusted to generate synthetic images with different perspectives, appearances, and dynamic characteristics.

[0070] Assume that, in the 3DGS framework, the K scenes in the adjusted sample images are represented by K Gaussian distributions. The Gaussian parameters of each Gaussian distribution are defined as follows:

[0071] Position (μ): The spatial position of each Gaussian distribution is expressed as a 3D coordinate (x, y, z). The positions of K Gaussian distributions are shown in Equation (1):

[0072] μ∈R K×3 (1)

[0073] Where each row is a 3D position (coordinate in 3D space) of a Gaussian distribution, and R represents a real number.

[0074] Rotation (q): Each Gaussian distribution rotation is represented by a quaternion to avoid the gimbal deadlock problem. The rotation matrix q contains the rotation information of a Gaussian distribution, as shown in Equation (2):

[0075] q∈R K×4 (2)

[0076] Each row is a 4-dimensional quaternion (qw, qx, qy, qz), where qw represents the cosine of the rotation angle, and qx, qy, and qz represent the product of the x, y, and z (unit vector) components of the rotation axis and the sine of the rotation angle, respectively, together representing the rotation information of the Gaussian distribution.

[0077] Scaling (s): The scaling of each Gaussian distribution is expressed as a scale factor in three directions, as shown in Equation (3):

[0078] s∈R K×3 (3)

[0079] Each row is a Gaussian-distributed 3D scaling factor (sx, sy, sz), where sx, sy, and sz represent the scaling ratios along the x-axis, y-axis, and z-axis, respectively.

[0080] Color (y): The color of each Gaussian distribution is a 16x3 RGB matrix, which represents the color characteristics of the Gaussian distribution, as shown in formula (4):

[0081] y∈R K×16×3 (4)

[0082] Among them, each Gaussian distribution has 16 different color values, and each color value represents the color change under different lighting conditions.

[0083] Opacity (α): The transparency of each Gaussian distribution is expressed as a scalar, usually between 0 and 1, as shown in Equation (5):

[0084] α∈R K (5)

[0085] Each element represents a transparency value of a Gaussian distribution.

[0086] S120. Add disturbance to the Gaussian parameters of the Gaussian distribution to obtain new Gaussian parameters, and randomly combine the new Gaussian parameters to obtain a new Gaussian distribution.

[0087] Specifically, perturbations can be added to one or more Gaussian parameters of the Gaussian distribution to obtain new Gaussian parameters. For example, position perturbation: adding position perturbations to the spatial position of the Gaussian distribution to generate synthetic images of different spatial positions. Rotation perturbation: by randomly perturbing the quaternions of the Gaussian distribution, different perspective changes can be simulated. Scaling perturbation: perturbing the scaling factor of the Gaussian distribution to generate scenes with different scales. Color perturbation: by perturbing the color of the Gaussian distribution, different lighting and texture changes can be simulated. Transparency perturbation: perturbing the opacity to simulate different scene light transmittance effects.

[0088] The new Gaussian parameters are then randomly combined to form a new Gaussian distribution. For example, the perturbed spatial position is combined with other unperturbed Gaussian parameters to obtain the Gaussian distribution of the composite image; or the perturbed spatial position and perturbed color are combined with other unperturbed Gaussian parameters to obtain the Gaussian distribution of the composite image.

[0089] These perturbations ensure diversity in the synthesized images, enabling the generation of rich training data.

[0090] S130 , converting each new Gaussian distribution into sample features, mapping the sample features according to the camera posture, and obtaining a composite image.

[0091] Specifically, in order to improve the quality of the synthesized image, the present invention also introduces dynamic appearance modeling. In this step, the scene (Gaussian distribution) is extracted from the captured sample image, and a learnable sampler (S) is used to assign independent appearance features to each Gaussian distribution. Specifically, each new Gaussian distribution is input into the sampler S, and the sampler S provides a 2-dimensional sample feature for each Gaussian distribution, representing the appearance characteristics of the scene. The feature can be but is not limited to texture, lighting or other visual information. This enables the synthesized image to better reflect the dynamic changes in the real scene, such as lighting changes and material changes.

[0092] In pose regression, the model aims to compute the camera pose from the image. To simulate the motion blur effect in real scenes, the present invention optimizes the camera motion blur process (i.e., the inverse of the scene motion modeled based on the camera pose) when synthesizing new synthetic images. Specifically, the scene motion is simulated by a Gaussian position transformation T, where T refers to the rotational and translational poses of the camera. This transformation belongs to the 3D rigid transformation (SE(3) group): T∈SE(3).

[0093] T describes the relative motion between the camera and the scene. This motion blur modeling can effectively reduce the motion blur effect in the synthesized image, making the final synthesized data more realistic and natural. SE(3) is a rigid transformation group in 3D space, which includes translation and rotation operations. The present invention adjusts the transformation T of the Gaussian position and randomly selects T from SE(3) to map the sample features to obtain a synthesized image, thereby reducing the impact of the blur effect on image quality.

[0094] These synthetic images can provide high-quality training data for the pose regression model, filling the gaps in the actual dataset and improving the generalization ability and adaptability of the pose regression model.

[0095] In this embodiment, if Figure 4 As shown, Figure 4 This is a branch flow chart for obtaining a first predicted posture. Step S200 inputs the captured sample image into a posture regression model to obtain the first predicted posture, specifically including:

[0096] S210 , extracting the output of the first specific layer in the posture regression model as a first sub-feature map, and extracting the output of the second specific layer in the posture regression model as a second sub-feature map.

[0097] The pose regression model can be, but is not limited to, a PoseTransformer model (pose estimation transformer). The first specific layer can be a layer in the backbone network, such as (third layer reduction_3); the second specific layer can be a layer in the backbone network, such as (fourth layer reduction_4). It is worth noting that the first feature layer and the second feature layer are different network layers.

[0098] The output image feature of the last layer of the backbone network of the PoseTransformer model can be expressed as E(I)∈R H ×W×C , which can be used for adversarial between the generator and the discriminator, where H and W are the height and width of the feature map, respectively, and C is the number of channels.

[0099] The first sub-feature map: Taking the third layer reduction_3 as an example, it can be expressed as Among them, Ht and Wt are the height and width of the feature map of this layer, and Ct is the number of channels.

[0100] The second sub-feature map: Taking the fourth layer reduction_4 as an example, it can be expressed as Among them, Hr, Wr, and Cr are the height, width, and number of channels of the feature map used for rotation regression.

[0101] S220: Perform translation regression on the first sub-feature map to obtain a translation posture; perform rotation regression on the second sub-feature map to obtain a rotation posture.

[0102] like Figure 5 As shown, Figure 5 This is a flowchart of the steps of posture regression, step S220 specifically includes:

[0103] S221. Generate a translation token and a rotation token.

[0104] Among them, translation token and rotation token translation token is of dimension The feature vector of , combined with the backbone network structure design, is a learnable parameter used to concatenate with the flattened sub-feature map to encode the global information related to the translation.

[0105] Translation Token: Rotating Tokens:

[0106] S222: Concatenate the first sub-feature map and the translation token to obtain a third sub-feature map, generate a first position code according to the third sub-feature map, and obtain a translation posture according to the third sub-feature map and the first position code.

[0107] Specifically, the first sub-feature map is flattened, which is expressed as Flatten(F t (I))∈R t W t ×C t , concatenate the flattened first sub-feature map and the translated token to obtain the third sub-feature map As shown in formula (6):

[0108]

[0109] Among them, Cat(.) represents the splicing operation, Flatten(F t (I)) and Trans merged into

[0110] After the splicing is completed, it is necessary to The first position encoding (PE) is added to the information in order to capture the spatial position information in the feature. The dimension of the first position encoding is consistent with the dimension of the concatenated feature, that is, The role of position encoding is to add position information to each feature, thereby enhancing the model's understanding of spatial structure. The concatenated third sub-feature map and the first position encoding are input into the Multi-Head Self-Attention (MSA) module for processing. The self-attention mechanism captures long-range dependencies by calculating the similarity between features, enhancing the model's global understanding ability. As shown in Equation (7):

[0111]

[0112] in, Represents the features output by the MSA module.

[0113] The features output by the MSA module are input into a fully connected feed-forward network (FFN) for further processing. The FFN includes two linear layers with ReLU activation functions, which can enhance the nonlinear expression ability of the features. As shown in formula (8):

[0114]

[0115] Among them, LN stands for layer normalization, which can improve the stability of the network training process. Represents the features of FNN output.

[0116] The features output by the FNN are input into the posture regression network for posture regression to obtain the translation posture. The MAS module and the FNN module can extract global information and long-range dependencies in the image and reduce the noise impact of local features.

[0117] S223: Concatenate the second sub-feature map and the rotation token to obtain a fourth sub-feature map, generate a second position code according to the fourth sub-feature map, and obtain the rotation posture according to the fourth sub-feature map and the second position code.

[0118] Specifically, step S223 is similar to step S222, except that the second sub-feature map is flattened first. In formulas (6)-(8), the first sub-feature map is replaced by the second sub-feature map, the translation token is replaced by the rotation token, and the first position code is replaced by the second position code. The final result is a rotation posture.

[0119] In this embodiment, in step S200, the first posture regression loss is obtained according to the first predicted posture and the first true label, specifically including

[0120] S230 , obtaining a translation loss according to the translation posture and the translation label; and obtaining a rotation loss according to the rotation posture and the rotation label.

[0121] Specifically, the translation loss is obtained according to the translation pose and translation label, as shown in formula (9):

[0122]

[0123] Among them, L t represents the translation loss, t represents the translation posture, Indicates a translation label.

[0124] The rotation loss is obtained according to the rotation pose and rotation label, as shown in formula (10):

[0125]

[0126] Among them, L r represents the rotation loss, r represents the rotation attitude, Represents a rotated label.

[0127] S230 , performing weighted summation on the translation loss and the rotation loss to obtain a first posture regression loss.

[0128] Specifically, two learning parameters s are introduced in the first pose regression loss t and s r , which is used to balance the optimization of translation and rotation. The weighted sum can be, but is not limited to, the formula (11):

[0129]

[0130] in, represents the first pose regression loss, and exp(.) represents the exponential function with the natural constant e as the base.

[0131] In this embodiment, if Figure 6 As shown, Figure 6 This is a branch flow chart for calculating the discriminator loss and the generator loss. In step S300, the discriminator loss is obtained according to the first image feature and the second image feature, specifically including:

[0132] S310 , inputting the first image feature and the second image feature into an adjustment layer respectively, and obtaining a first adjustment layer output and a second adjustment layer output respectively.

[0133] The adjustment layer includes Conv layer, ReLU layer and BN layer. The first image feature: F t (i), the second image feature F t (I′), both are features output by the last layer of the backbone network.

[0134] Specifically, the first image feature is input into the adjustment layer to obtain the first adjustment layer output Adj(F t(I)), input the second image feature into the adjustment layer to obtain the second adjustment layer output Adj′(F t (I′)).

[0135] S320 , respectively inputting the first adjustment layer output and the second adjustment layer output into the discriminator to obtain a first discriminator output and a second discriminator output.

[0136] The discriminator D is tasked with distinguishing the features of the captured sample image (real image) from those of the synthesized image. The discriminator can employ a common convolutional neural network (CNN), including convolutional layers (Conv layers), batch normalization layers (BN layers), and ReLU activation functions. A spectral normalization layer can also be introduced to stabilize training. The generator can employ a network structure similar to that of a CNN, including transposed convolutional layers, BN layers, and activation functions. The discriminator and generator structures in existing models such as Deep Convolutional Generative Adversarial Networks (DCGAN) and WGAN (WassersteinGAN) can serve as reference.

[0137] Specifically, the output of the first adjustment layer is input to the discriminator to obtain the first discriminator output D(Adj(F t (I))), the second adjustment layer output is input to the discriminator to obtain the second discriminator output D(Adj′(F t (I′))).

[0138] S330: Obtain a discriminator loss according to the first discriminator output and the second discriminator output.

[0139] Specifically, the discriminator loss can be, but is not limited to, as shown in formula (12):

[0140]

[0141] Among them, L Dis represents the discriminator loss, Represents the distribution P of all captured sample images data Next, we find the expectation of the sample image I, E P It means finding the expectation of the synthetic image I′ under the distribution P of all synthetic images.

[0142] The goal of the discriminator D is to use the discriminator loss to make the output of the first image feature 1 and the output of the second image feature 0. The goal of the generator G is to make the output of the discriminator D for the second image feature as close to 1 as possible, attempting to deceive the discriminator D and make it difficult for it to distinguish between the synthesized image and the captured sample image. In this way, the characteristics of the synthesized image will gradually approach those of the real image, thereby improving the generalization ability of the model.

[0143] According to the second image feature F t (I′) gets the generator loss, as shown in formula (13):

[0144]

[0145] Among them, L Gen represents the generator loss, E Y represents the expected value of the discriminator's output Y (i.e., the probability value that the discriminator determines that the synthetic image is the captured sample image), where Y is the discriminator's output result for the synthetic image I'. In this embodiment, in step S400, the second pose regression loss is obtained based on the second predicted pose and the first true label, specifically including:

[0146] S410: Add noise to the first true label to obtain a second true label.

[0147] The noise includes but is not limited to common types such as Gaussian noise and uniform noise. The second true label includes the translation label after adding noise and the rotation label after adding noise.

[0148] Specifically, to translate the label Add a translation noise δt to generate the translation label after adding noise Rotate label Add a rotation noise δr to generate the rotation label after adding noise

[0149] S420 , obtaining a second posture regression loss according to the second true label and the second predicted posture.

[0150] The second predicted posture includes a translation posture t′ and a rotation posture r′.

[0151] Specifically, the second posture regression loss process is obtained according to the second true label and the second predicted posture as follows: replace t in formula (9) with t′, Replace Replace r in formula (10) with r′, Replace The second posture regression loss is calculated according to formula (11):

[0152] Based on the calculation process of the aforementioned losses, in step S400, the total loss is obtained according to the first pose regression loss, the discriminator loss, the generator loss and the second pose regression loss, which can be shown as formula (14):

[0153]

[0154] Among them, L total Represents the total loss, β1, β2, and β3 are the weight parameters of each loss term, which are used to balance the impact of different loss terms.

[0155] In some embodiments, the present invention provides a posture regression method, comprising the following steps:

[0156] Obtain the image to be predicted, input the image to be predicted into the trained posture regression model, and obtain the actual predicted posture.

[0157] The trained pose regression model is obtained by the aforementioned training method. The trained pose regression model includes the backbone network and the pose regression network, excluding the aforementioned adjustment layer, discriminator, and generator.

[0158] Specifically, the image to be predicted is input into the trained posture regression model to obtain the actual predicted posture. S210-S220 and its sub-steps are similar, except that the input is the image to be predicted.

[0159] In some embodiments, the present invention provides a posture regression system, including a training module and a prediction module, wherein:

[0160] Training module, used to implement the previous training method;

[0161] The prediction module is used to implement the previous posture regression method.

[0162] The present invention also has the following beneficial effects:

[0163] First, a new synthetic image is synthesized on the actual sample image. The sample image and the synthetic image are input into the model separately. A first pose regression loss is designed based on the pose predicted by the sample image to reduce the domain gap between the synthetic data and the real data. In addition to the final predicted pose, the present invention also considers the impact of the accuracy of the model's extracted image features on the pose prediction. The present invention introduces a discriminator loss based on the first image features of the sample image and the second image features of the synthetic image. The discriminator loss is used to make the feature output of the real image 1 and the feature output of the synthetic image 0. The generator loss is used to compete with the discriminator loss, allowing the model to learn how to extract features similar to the real image from the synthetic image. Finally, a second pose regression loss is designed based on the pose predicted by the synthetic image to further improve the model's generalization ability for unseen image and pose pairs. The aforementioned various losses are combined to form a final total loss, and the model is trained based on the total loss to improve the model's generalization ability.

[0164] The present invention generates new synthetic images and poses online every 20 epochs, uses the new poses as real labels for training, and increases the samples of training data. The diversity and quality of synthetic images are improved through Gaussian sputtering and SE (3) groups. The generator and discriminator are used for adversarial training, and synthetic images and perturbations are added to the labels, so that the model can accurately handle complex situations such as illumination changes and perspective changes. In scenes with drastic illumination changes, the model can accurately regress the camera pose by learning the features under different illumination conditions in the synthetic images, thus avoiding positioning errors caused by illumination changes. In cases where the perspective changes greatly, the model can use the Pose Transformer's ability to model long-distance dependencies and the rich perspective information in the synthetic data to accurately estimate the camera pose, thereby improving the accuracy and stability of positioning.

[0165] In addition, the present invention provides an example of a Gaussian distribution:

[0166] Assume that the image contains three Gaussian distributions, and the parameters of each Gaussian distribution are as follows:

[0167] Position (μ):

[0168]

[0169] This means that the first Gaussian distribution is located at the origin, the second Gaussian distribution is located at (1, 2, 3), and the third Gaussian distribution is located at (-1, 0.5, 2.5).

[0170] Rotation(q):

[0171]

[0172] This means that the rotation quaternion of the first Gaussian distribution represents a 90-degree rotation around the y-axis, the second Gaussian distribution represents no rotation, and the third Gaussian distribution represents a 45-degree rotation around both x, y, and z.

[0173] Zoom(s):

[0174]

[0175] This means that the first Gaussian distribution is scaled to the original size, the second Gaussian distribution is scaled to half the x-axis, and the third Gaussian distribution has different scaling in all directions.

[0176] Color (y):

[0177]

[0178] This means that each Gaussian distribution has 16 color variations, representing different lighting conditions.

[0179] Opacity (α):

[0180]

[0181] This means that the first Gaussian distribution is completely opaque, the second Gaussian distribution has some transparency, and the third Gaussian distribution is relatively transparent.

[0182] The present invention also provides a specific example of translation regression:

[0183] The size of the image input to the backbone network is 224×224, and the backbone network used is ResNet-50.

[0184] The backbone network outputs the first image feature: the output feature map of the backbone network has a size of 56×56×2048.

[0185] Translation regression: Extract the first sub-feature map F from the third layer reduction_3 t (I)∈R 56×56×512 .

[0186] Features after splicing: The flattened features are Flatten(F t (I))∈R 3136×512 , the feature size after splicing is (3136+1)×512, that is, R 3137×512 .

[0187] First position code: first position code PE∈R 3137×512 .

[0188] Multi-head self-attention: After being processed by the multi-head self-attention module, the features are obtained

[0189] Fully connected feedforward network: Input FFN, after processing, get the output of the fully connected feedforward network, the size is R 3137×512 .

[0190] The translation pose is then regressed based on the output of the fully connected feedforward network.

[0191] It is worth noting that the present invention can be applied to visual positioning, including but not limited to aircraft, autonomous driving scenarios, and augmented reality, to achieve camera posture regression.

[0192] For example, in the field of autonomous driving, the method of the present invention can provide more accurate positioning information for vehicles, improving the safety and reliability of autonomous driving systems. By accurately determining the vehicle's position and posture on the road, autonomous driving systems can better plan driving routes and avoid collisions. In the field of robotic navigation, the method of the present invention can enable robots to more accurately perceive their own position and surroundings in complex environments, improving navigation accuracy and autonomous decision-making capabilities. In logistics handling robots, it can more accurately grasp goods and improve logistics efficiency. In the field of augmented reality, the present invention can more accurately overlay virtual objects on real scenes, enhancing the immersion and interactivity of augmented reality applications. In quality inspection, it can more accurately detect product defects and improve product quality. In the field of intelligent security, it can more accurately identify the position and posture of target objects, improving the monitoring capabilities of security systems. In the medical field, the method of the present invention also has potential application value. In surgical navigation, it can provide doctors with more accurate surgical instrument position information, improving the success rate of operations.

[0193] The beneficial effects of the present invention are verified by experiments:

[0194] 1. Experimental Setup

[0195] This experiment uses representative datasets such as 7-Scenes, Cambridge Landmarks, and MARS. The 7-Scenes dataset contains indoor scenes, such as offices and kitchens. The data is characterized by relatively fixed scenes, but with certain lighting changes and perspective diversity. During processing, the sample images were normalized to unify the data scale and ensure the stability of model training. The Cambridge Landmarks dataset covers complex outdoor scenes, such as churches and colleges, with large lighting changes and complex scene structures. For this dataset, image enhancement operations such as adjusting brightness and contrast were performed during the data preprocessing stage to increase data diversity and improve the model's adaptability to different lighting conditions. The MARS dataset focuses on autonomous driving scenarios, which include challenges such as moving objects and frequent changes in lighting. During processing, the data was labeled and classified so that the model can better learn the features of different scenarios.

[0196] The experiment used median translation error (unit: cm) and rotation error (unit: °) as the main evaluation metrics. The median translation error effectively reflects the average position deviation of the model during positioning, avoiding the influence of individual extreme values. The rotation error measures the difference between the rotation angle predicted by the model and the true value. Using these two metrics, the positioning accuracy of the model can be comprehensively and accurately evaluated. At the same time, to more comprehensively evaluate the model performance, the average error, maximum error, and minimum error are also reported, showing the performance of the model in different scenarios from different perspectives.

[0197] 2. Experimental Procedure

[0198] During the data synthesis phase, 3DGS technology is used to upgrade RGB image sequences to 3D Gaussian sputtering representations. First, the parameters of 3DGS are set according to the characteristics and requirements of the scene, including the number, position, rotation, and scaling of Gaussian distributions. By adjusting these Gaussian parameters, synthetic images with different perspectives and appearances are generated. When dealing with motion blur, the Deblur-GS method is used to model camera motion and optimize the transformation of Gaussian positions to produce clearer synthetic images. During data preprocessing, the synthetic images and captured sample images (real images) are normalized so that their pixel values are in the range [0, 1]. The images are also resized to meet the input requirements of the model.

[0199] The model was trained using the pretrained Efficient-B0 backbone network on NVIDIA RTX A6000 GPUs with a batch size of B = 12. The Adam optimizer was used, with the learning rate initialized to 1×10⁻4. Early stopping was employed during training with a patience value of 200, stopping training when the validation loss stopped decreasing within 200 epochs to prevent overfitting. Every 50 epochs, if the validation loss remained stable, the learning rate was reduced to 0.95. The weights for each loss were set to β1 = 1, β2 = 1, and β3 = 0.7. Following the DFNet approach, an additional triplet loss was introduced with the same weights as β1. Every 20 epochs, appearance and pose augmentation techniques were used to enrich the training data by randomly generating the same number of views as the training samples. The model typically converged after approximately 1000 epochs.

[0200] During model testing, the trained model is applied to a real-world scenario. First, a test image is fed into the pose regression model, which outputs an initial predicted pose. 3DGS technology is then used to render an RGB-D image based on the initial predicted pose. Next, the MASt3R method is combined to obtain 2D-3D correspondences. A RANSAC-PnP (Consensus-Perspective-n-Point) operation is performed to optimize the initial predicted pose and obtain the final, accurate pose. Throughout the testing process, the performance of the pose regression model is evaluated and analyzed strictly in accordance with the requirements of the actual application scenario.

[0201] 3. Experimental Results

[0202] Experimental results show that the method of the present invention has achieved excellent positioning accuracy on different data sets. On the 7-Scenes dataset, compared with the previous single-frame absolute pose regression (APR) method, the average translation error of the pose regression method of the present invention is reduced by 50% (from 0.10cm to 0.05cm), and the rotation error is reduced by 21.91% (from 3.24° to 2.51°). On the Cambridge Landmarks dataset, compared with other single-frame absolute pose regression methods, the method of the present invention shows significant performance advantages in all scenes, with both translation error and rotation error improved by more than 30%. On the MARS dataset, the average positioning error of the RAP method of the present invention is 45cm / 0.78°, which is significantly better than the baseline PoseNet and DFNet methods. After one optimization, the RAPref method further reduces the outdoor positioning error to below 10cm.

[0203] In some embodiments, such as Figure 7 As shown, Figure 71 is a structural diagram of an electronic device provided by the present invention. The present invention also provides an electronic device, which includes a processor 10 and a memory 11, wherein the memory 11 stores a computer program, and when the processor 10 executes the computer program, it implements any one of the methods described in the above method embodiments.

[0204] Among them, the memory is a non-transient computer-readable storage medium that can be used to store non-transient software programs and non-transient computer executable programs. The memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory optionally includes a remote memory remotely arranged relative to the processor, and these remote memories can be connected to the processor via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0205] It is understood that all or some steps, systems in the disclosed method above can be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components can be implemented as software by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those of ordinary skill in the art, the term computer storage medium is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data) and is volatile and non-volatile, removable and non-removable media. Computer storage media includes but is not limited to RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or can be used to store desired information and any other medium that can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0206] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A training method for a posture regression model, characterized in that: The following steps are involved: Acquire a captured sample image and a composite image; the composite image is synthesized based on the captured sample image; Inputting the captured sample image into a posture regression model to obtain a first predicted posture and a first image feature; obtaining a first posture regression loss based on the first predicted posture and the first true label; Inputting the synthesized image into a pose regression model to obtain a second predicted pose and a second image feature; obtaining a discriminator loss based on the first image feature and the second image feature; and obtaining a generator loss based on the second image feature; A second posture regression loss is obtained according to the second predicted posture and the first true label; a total loss is obtained according to the first posture regression loss, the discriminator loss, the generator loss and the second posture regression loss, and the parameters of the posture regression model are optimized using the total loss to obtain a trained posture regression model.

2. The method according to claim 1, characterized in that The composite image is synthesized by the following method: Extracting at least one Gaussian distribution from the captured sample image; wherein a scene in the image is represented as a Gaussian distribution; Adding disturbance to the Gaussian parameters of the Gaussian distribution to obtain new Gaussian parameters, and randomly combining the new Gaussian parameters to obtain a new Gaussian distribution; Each new Gaussian distribution is converted into a sample feature, and the sample features are mapped according to the camera posture to obtain the composite image.

3. The method according to claim 1, characterized in that The first predicted posture includes a translation posture and a rotation posture of the camera posture; and inputting the captured sample image into a posture regression model to obtain the first predicted posture specifically includes: Extracting the output of the first specific layer in the posture regression model as a first sub-feature map, and extracting the output of the second specific layer in the posture regression model as a second sub-feature map; the first feature layer and the second feature layer are different network layers; Perform translation regression on the first sub-feature map to obtain the translation posture; perform rotation regression on the second sub-feature map to obtain the rotation posture.

4. The method according to claim 3, characterized in that performing translation regression on the first sub-feature map to obtain the translation posture; Performing rotation regression on the second sub-feature map to obtain the rotation posture specifically includes: Generate translation tokens and rotation tokens; splicing the first sub-feature map and the translation token to obtain a third sub-feature map, generating a first position code according to the third sub-feature map, and obtaining the translation posture according to the third sub-feature map and the first position code; The second sub-feature map and the rotation token are concatenated to obtain a fourth sub-feature map, a second position code is generated according to the fourth sub-feature map, and the rotation posture is obtained according to the fourth sub-feature map and the second position code.

5. The method according to claim 1, wherein The first predicted pose includes the translation pose and rotation pose of the camera pose, and the first real label includes a translation label and a rotation label; the first pose regression loss is obtained according to the first predicted pose and the first real label, specifically including Obtaining a translation loss according to the translation posture and the translation label; obtaining a rotation loss according to the rotation posture and the rotation label; A weighted sum is performed on the translation loss and the rotation loss to obtain the first posture regression loss.

6. The method according to claim 1, characterized in that Obtaining a discriminator loss according to the first image feature and the second image feature specifically includes: Inputting the first image feature and the second image feature into an adjustment layer respectively, and obtaining a first adjustment layer output and a second adjustment layer output respectively; Inputting the first adjustment layer output and the second adjustment layer output into a discriminator respectively, to obtain a first discriminator output and a second discriminator output respectively; The discriminator loss is obtained according to the first discriminator output and the second discriminator output.

7. The method according to claim 1, characterized in that The obtaining of a second posture regression loss according to the second predicted posture and the first true label specifically includes: Adding noise to the first true label to obtain a second true label; The second posture regression loss is obtained according to the second true label and the second predicted posture.

8. A posture regression method, characterized in that: The following steps are involved: Acquire an image to be predicted, input the image to be predicted into a trained posture regression model to obtain an actual predicted posture; the trained posture regression model is trained by the training method according to any one of claims 1 to 7.

9. A posture regression system, characterized in that: It includes training module and prediction module, among which, The training module is used to implement the training method according to any one of claims 1 to 7; The prediction module is used to implement the posture regression method described in claim 8.

10. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

Images