Template matching posture determination method, system and electronic device

Abstract

The present invention discloses a template matching pose determination method, system and electronic device, and relates to the technical field of pose determination. The template matching pose estimation model constructed by the present invention includes: a query network, a template network, an encoder, a decoder, a segmentation decoder and a discriminator, wherein the decoder, the segmentation decoder and the discriminator are only used in the training stage. The present invention generates encoding vectors of the query image and the template respectively through a network, and determines the pose of the target in the query image by comparing the query encoding vector with the template encoding vector for similarity. The template is a plurality of different rotation matrices obtained by rendering an isometric view around the target, and the template encoding vectors are generated through the template network and the encoder to form a template library. Each encoding vector in the template library corresponds to its rotation matrix. The present invention adopts a joint loss function composed of reconstruction loss, metric loss, segmentation loss and mutual information loss in the training stage, which can simultaneously improve the accuracy and robustness of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of posture determination, and in particular to a template matching posture determination method, system and electronic equipment. Background Art

[0002] Estimating the pose of an object from an RGB image—that is, its position and orientation relative to the camera coordinate system—is a challenging task in computer vision. This task is widely used in numerous application scenarios, such as virtual reality, augmented reality, autonomous driving, robotics, and aircraft control. In recent years, pose estimation methods based on deep learning have garnered widespread attention. Compared to traditional methods, neural networks can learn more robust feature representations, far surpassing manual methods in pose estimation accuracy and applicability.

[0003] Template matching methods use neural networks to map high-dimensional images into a low-dimensional embedding space. The image's position in the embedding space is represented by a vector. During the training phase, the goal is to position target images with the same pose closer together (lowering their position) in the embedding space, while moving target images with different poses further apart. During the testing phase, the pose of the query image's target is determined by comparing the similarity between the query image's representation vector and the representation vectors in a template library. Template matching methods are mainly categorized into metric-based and reconstruction-based methods. Previous metric-based methods used encoders constructed using convolutions and trained using triplet and bigram losses. However, triplet and bigram losses can only utilize a pair of positive and negative examples, making it difficult to constrain the distance between other negative examples and the positive, resulting in slow convergence and poor performance. Some methods have proposed using the InfoNCE loss function to train the network. This function improves the shortcomings of the triplet loss by utilizing all negative examples in a batch, resulting in better matching accuracy and robustness. Reconstruction-based methods use an encoder and a decoder. The encoder maps the image into the embedding space, while the decoder reverse-maps the representation vector back to the image space. Reconstruction-based methods constrain representation vectors through a reconstruction loss. During training, gradients must pass through the decoder before being propagated to the representation vectors. This makes it difficult to maintain the distance between the representation vectors of positive and negative instances, especially for instances with high similarity. Adding images with occluded objects to the training set can improve the accuracy of these methods in these situations, but they are still insufficient for complex and changing scenes.

[0004] In summary, how to improve the accuracy and robustness of template matching pose estimation under complex background occlusion has become a technical problem that needs to be solved urgently in this field. Summary of the Invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a template matching posture determination method, system and electronic equipment.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A template matching pose determination method, comprising:

[0008] Train the network model according to the loss function;

[0009] Sampling the target isometric view to obtain a rotation matrix for constructing a template library, and constructing the template library through a template network and an encoder; the template library includes a template encoding vector, a rotation matrix corresponding to the template encoding vector, and a 2D bounding box of the target in the rendering view;

[0010] Get the query image and generate the query encoding vector through the query network and encoder;

[0011] Compare the cosine similarity between the query encoding vector and the encoding vector in the template library;

[0012] Determine the target pose in the query image; the target state is the rotation matrix corresponding to the template encoding vector that has the highest similarity to the query encoding vector;

[0013] The translation vector of the target is obtained according to the 2D bounding box of the query image and the parameters of the 2D bounding box camera of the rendering view.

[0014] Optionally, the query network includes: five stages;

[0015] Among them, before each stage, convolution with a kernel of 2×2 and a stride of 2 is used for downsampling;

[0016] In the first two stages, the residual module is used to extract features, and in the last two stages, the multi-head attention module is used to extract deep features;

[0017] In the fifth stage, a 1×1 convolution is used to reduce the dimension and output the deep feature map, and then the MLP module is used to output the deep feature map as a low-dimensional feature vector.

[0018] Optionally, the query network takes the query image as input to generate a feature vector; the encoder takes the feature vector as input to generate a query encoding vector; after organizing the rotation matrix into a one-dimensional vector, the feature vector is input into the template network to generate a feature vector, and the feature vector generated by the template network is passed through the encoder to generate a template encoding vector; the decoder is used to reconstruct the template encoding vector into a rotation matrix; the metric loss acts between the query encoding vector and the template variable vector; the reconstruction loss acts between the reconstructed rotation matrix and the input rotation matrix; the segmentation loss and the mutual information loss are used to enhance the local features of the target; the segmentation decoder takes the output feature map of the third stage of the query network as input, upsamples stage by stage, and outputs a segmentation mask of the target; the segmentation decoder corresponds to the network layers of the first three stages of the query network;

[0019] The discriminator includes a local discriminant network and a global discriminant network; the local discriminant network is used to expand the query encoding vector to the same size as the local feature map, and then splice the expanded query encoding vector and the local feature map. If the expanded local features and the expanded query encoding vector come from the same query, the discriminator output is maximized; if the expanded local features and the expanded query encoding vector do not come from the same query, the discriminator output is minimized; the global discriminant network reduces the dimension of the local feature map to a global vector through a convolution operation, and then splices the global vector and the query encoding vector into a vector to determine whether the global vector and the query encoding vector come from the same query. If the global vector and the query encoding vector come from the same query, the discriminator output is maximized; if the global vector and the query encoding vector do not come from the same query, the discriminator output is minimized.

[0020] Optionally, the template network includes four layers of perceptrons; and the encoder includes two layers of perceptrons.

[0021] Optionally, a mean square error function is used as the reconstruction loss; the reconstruction loss is:

[0022]

[0023] Among them, L rec Represents the reconstruction loss value, y rot represents the rotation matrix, Represents the reconstructed rotation matrix, N represents the amount of data, MSE() represents the mean square error function, Represents the rotation matrix of the i-th data, Represents the rotation matrix after reconstruction of the i-th data;

[0024] N-pair loss is used as the metric loss; the metric loss is:

[0025]

[0026] Among them, x i represents the i-th input target image, Represents the rotation matrix of the i-th data, z i Represents the input target image x i The query encoding vector generated by the query network and encoder, Represents the rotation matrix The template representation vector generated by the template network and encoder is The template representation vector generated by the rotation matrix of the jth data, j≠i, Represents the N-pair loss function value, T represents transposition;

[0027] The OHEM function is used as the segmentation loss; the segmentation loss is:

[0028]

[0029] Among them, top_k() means taking the first k maximum values, CE() means the cross entropy loss function, y mask represents the annotated segmentation mask, represents the segmentation mask output by the network, L seg Represents the segmentation loss value;

[0030] The mutual information loss is:

[0031]

[0032] in, represents the mutual information loss value, ω1, ω2, φ, ψ, γ represent the network parameters of the local discriminator, global discriminator, query network, template network and encoder respectively, represents the JS mutual information estimation, z glb Represents the global feature generation network output value, z glb =Γ′(M), Γ′() represents a global feature generation network inside the global discriminator, M represents the feature map, w m ×h m represents the number of dimensions, z represents the template encoding vector, Take the maximum value function, α and β are hyper parameters, m (i) Indicates dimension c m The i-th eigenvector of .

[0033] Optionally, the loss function of the target pose matching model is:

[0034]

[0035] Among them, λ1, λ2 and λ3 are hyperparameters, L N-pair is the measurement loss value.

[0036] Optionally, determining the pose of the target in the query image specifically includes:

[0037] Performing a cosine similarity comparison between the query encoding vector and the template encoding vector of the template image in the template library to obtain a cosine similarity comparison result;

[0038] The posture of the target in the template image corresponding to the minimum cosine similarity comparison result is used as the posture of the target in the query image.

[0039] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0040] The template matching pose determination method provided by the present invention constructs a target pose matching model comprising a query network, a template network, an encoder, a decoder, a segmentation decoder, and a discriminator. This method can accurately match a query image with a template image to obtain the target pose. Furthermore, by employing a loss function comprising a reconstruction loss, a metric loss, a segmentation loss, and a mutual information loss as the loss function of the target pose matching model, the present invention can further improve the accuracy of target pose matching while also enhancing the robustness of template pose matching.

[0041] To implement the template matching pose determination method provided above, the present invention also provides the following two implementation structures:

[0042] A template matching pose determination system is applied to the above-mentioned template matching pose determination method; the system comprises:

[0043] Model training module, used to train the network model according to the loss function;

[0044] A template library construction module is used to sample the target isometric view to obtain the rotation matrix for constructing the template library, and construct the template library through the template network and the encoder; the template library includes the template encoding vector, the rotation matrix corresponding to the template encoding vector, and the 2D bounding box of the target in the rendering view;

[0045] The image acquisition module is used to obtain the query image and generate the query encoding vector through the query network and encoder;

[0046] A cosine similarity comparison module is used to compare the cosine similarity between the query encoding vector and the encoding vector in the template library;

[0047] A target posture determination module is used to determine the target posture in the query image; the target state is the rotation matrix corresponding to the template encoding vector with the highest similarity to the query encoding vector;

[0048] The target translation vector determination module is used to obtain the target translation vector according to the 2D bounding box of the query image and the parameters of the 2D bounding box camera of the rendering view.

[0049] An electronic device, comprising:

[0050] memory for storing computer programs;

[0051] A processor is connected to the memory and is used to call and execute the computer program to implement the above-mentioned template matching posture determination method.

[0052] Optionally, the memory is a computer-readable storage medium.

[0053] Since the technical effects achieved by the two implementation structures provided above in the present invention are the same as the technical effects achieved by the template matching posture determination method provided above in the present invention, they will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 Flowchart of the template matching pose determination method provided by the present invention;

[0056] Figure 2 A schematic diagram of the structure of the query network provided by the present invention;

[0057] Figure 3 A partial schematic diagram describing the constraint coding vector in the target pose matching model provided by the present invention;

[0058] Figure 4 A partial schematic diagram describing constrained local features in the target pose matching model provided by the present invention;

[0059] Figure 5 This is a network diagram of the actual measurement phase provided by the present invention. DETAILED DESCRIPTION

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0061] The purpose of the present invention is to provide a template matching posture determination method, system and electronic equipment, which can improve the accuracy and robustness of template posture matching.

[0062] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] like Figure 1 As shown, the template matching pose determination method provided by the present invention includes:

[0064] Step 100: Train the network model according to the loss function.

[0065] Step 101 samples the target isometric view to obtain a rotation matrix for constructing a template library, and constructs the template library through a template network and an encoder. The template library includes a template encoding vector, a rotation matrix corresponding to the template encoding vector, and a 2D bounding box of the target in the rendered view.

[0066] Step 102: Obtain a query image and generate a query encoding vector through the query network and encoder.

[0067] Step 103: Compare the cosine similarity between the query encoding vector and the encoding vectors in the template library.

[0068] Step 104: Determine the target pose in the query image. The target state is the rotation matrix corresponding to the template encoding vector that has the highest similarity to the query encoding vector.

[0069] Step 105: Determine the translation vector of the target in the query image, wherein the translation vector of the target is obtained based on the 2D bounding box of the query image and the parameters of the 2D bounding box camera of the rendering view.

[0070] Since the core of the template matching posture determination method provided by the present invention lies in the construction process of the target posture matching model in step 101, the advantages and implementation process of the method provided by the present invention are explained based on the specific construction process of the target posture matching model.

[0071] The six network structures of the target pose matching model provided by this invention are: the query network takes an image as input, and the template network takes a rotation matrix as input; both networks output feature vectors of the same size. The encoder takes the feature vector as input and outputs a low-dimensional encoding vector. The decoder reconstructs the encoder into a rotation matrix. The segmentation decoder outputs a segmentation mask for the image. The discriminator takes a deep feature of the query network and the encoding vector as input and maximizes the average mutual information estimate between the two. The query network, template network, and encoder are used in the training and testing phases, while the decoder, segmentation decoder, and discriminator are used only in the training phase.

[0072] like Figure 2 As shown, the query network used in the present invention is mainly divided into five stages (i.e. Figure 2In the process of downsampling, the convolution kernel is used to extract features with a kernel size of 2×2 and a stride of 2 before each stage. To ensure efficiency, a residual module is used to extract features in the first two stages, and a multi-head attention module is used to extract deep features in the last two stages. After the fifth stage, a 1×1 convolution is used to reduce the dimension and output a deep feature map. By constraining the mutual information between the feature map and the template representation vector, the response of the local target area is enhanced. The final multilayer perceptron (MLP) module outputs the high-dimensional feature map as a low-dimensional feature vector.

[0073] Figure 2 In the query network, the input image is propagated from top to bottom, and the final output feature vector has a dimension of 1024. A 2×2 convolution with a stride of 2 reduces the input resolution at each stage. Residual and self-attention modules are used for feature extraction.

[0074] The template network uses the rotation matrix corresponding to the target in the image to generate the template vector. The template network consists of 4 MLP layers, as shown in Table 1, and ultimately outputs a 1024-dimensional feature vector.

[0075] The encoder takes the image feature vector (i.e., the feature vector output by the query network) and the template feature vector (i.e., the feature vector output by the template network) as input and generates a 128-dimensional representation vector. The encoder consists of two layers of MLP.

[0076] The decoder takes the representation vector as input and reconstructs the corresponding pose.

[0077] The segmentation decoder segmentation network and the first three stages of the query network are mirror-symmetric, and the network layers of each stage are the same.

[0078] The discriminator consists of a local discriminant network and a global discriminant network. The local discriminant network first expands the encoding vector to the same size as the local feature map, then concatenates the expanded encoding with the local feature map. If the local features and the encoded features come from the same query, the discriminator output is maximized; otherwise, it is minimized. The global discriminant network first reduces the dimensionality of the local feature map to a global vector through a convolution operation, then concatenates it with the encoding vector into a single vector, and then determines whether it comes from the same query.

[0079] Table 1. Six network structures of target pose matching model

[0080]

[0081]

[0082] Furthermore, the specific construction process of the loss function of the target pose matching model is as follows:

[0083] (1) Reconstruction loss is used to reconstruct the pose The difference between the pose and the real pose y is calculated based on this, and the mean square error function is used as the reconstruction loss. The reconstruction loss is:

[0084]

[0085] Among them, L rec Represents the reconstruction loss value, y rot represents the rotation matrix of the annotation, Represents the reconstructed rotation matrix, N represents the amount of data in a training batch, i represents the i-th data, MSE() represents the mean square error function, Represents the rotation matrix of the i-th data, Represents the rotation matrix after reconstruction of the i-th data.

[0086] (2) Metric loss directly constrains the representation vector. For a positive instance, it is hoped that the distance between its generated encoding vector and the corresponding template encoding vector can be reduced, and the distance between it and other template encoding vectors can be increased. Based on this, N-pair loss is used as the metric loss. The metric loss is:

[0087]

[0088] Among them, x i represents the i-th input target image, Represents the rotation matrix of the i-th data, z i Represents the input target image x i The query encoding vector generated by the query network and encoder, Represents the rotation matrix The template representation vector generated by the template network and encoder is The template representation vector generated by the rotation matrix of the jth data, j≠i, Represents the N-pair loss function value, and T represents transposition.

[0089] (3) Segmentation loss is used to enhance the features of the target in the deep layer of the query network and reduce the features of the background and occlusion. Based on this, the OHEM function is used as the segmentation loss. The segmentation loss is:

[0090]

[0091] Among them, top_k() means taking the first k maximum values, CE() means the cross entropy loss function, y mask represents the annotated segmentation mask, represents the segmentation mask output by the network, L seg Represents the segmentation loss value.

[0092] (4) Mutual information loss. The proposed method uses the adversarial learning method to estimate the mutual information between local features and encoding vectors and maximize the average mutual information between the two. Based on this, the JS mutual information estimation is used here, which is:

[0093]

[0094] in, represents the distribution, E p [] represents the expectation of distribution p, express The expected value of the distribution, sp(*) is the softplus function, sp(*)=log(1+e * ), T ω : It is the discriminant function defined by the neural network, that is, the discriminant network, ψ, ω represent the encoding network and the discriminant network parameters respectively, where x, y represent the image and the rotation matrix respectively.

[0095] The above mutual information estimation is used to constrain the relationship between local and global. First, the input image is encoded into a feature map M. It is the output of the last convolutional module of the query network. The feature map consists of w m ×h m The dimension is c m The characteristic vector of m (i) Indicates dimension c m The template network Ψ and encoder Υ take the rotation matrix y as input, generate the template encoding vector z, define the mutual information estimator between the local feature vector and the template representation vector, and constrain the local and global consistency by maximizing the mutual information.

[0096]

[0097] w1,φ, γ represents the network parameters of the local discriminator, query network, template network and encoder respectively, represents the learned optimal local discriminator, query network, template network and encoder network parameters, The definition form is the same as (4), and the table is composed of w1,φ, The local mutual information estimator defined by γ. At the same time, in order to ensure the global characteristics of local features, the global discriminator is used to maximize the mutual information between global features and template encoding vectors:

[0098]

[0099] Among them, z glb =Γ′(M), Γ′() is a global feature generation network located inside the global discriminator, w2 represents the global discriminator network parameters, The definition is the same as (4), which means that w2,φ, The global mutual information estimator defined by γ. Then the total mutual information constraint function is:

[0100]

[0101] The total loss function is:

[0102]

[0103] Among them, α, β, λ1, λ2, and λ3 are hyperparameters to balance various losses.

[0104] Based on the above description, the initial target pose matching model is constructed. In order to ensure that the final model has the best estimation accuracy and robustness, the initial target pose matching model constructed above needs to be trained and tested. The specific training and testing process is as follows:

[0105] Step 1: Data Preparation. To train the network, you need training data consisting of an RGB image of the target, its rotation matrix, translation vector, and segmentation mask. The data volume should be ≥ 50K. The RGB image size should be 128×128.

[0106] Step 2: Network training. The network structure in the training phase is as follows Figure 3 and Figure 4 As shown, Figure 3 and Figure 4 Describe the part of constrained encoding vector and the part of constrained local features respectively. Figure 3 As shown in the figure, the training image generates a feature vector through the query network, and the encoder uses the feature vector as input to generate a coding vector; the rotation matrix corresponding to the image is organized into a one-dimensional vector and sent to the template network to generate a feature vector, which is then passed through the encoder to generate a template coding vector; the decoder reconstructs the template coding vector into a rotation matrix. The metric loss acts between the query coding vector and the template variable vector to shorten the distance between the two, and the reconstruction loss acts between the reconstructed rotation matrix and the input rotation matrix to reduce the difference between the reconstruction and the input to maintain the structure of the embedding space. Figure 4As shown in the figure, the local features of the target are enhanced through segmentation loss and mutual information loss. The segmentation decoder takes the output feature map of the third stage of the query network as input, upsamples it stage by stage, and outputs the segmentation mask of the target. The segmentation decoder corresponds to the network layers of the first three stages of the query network. The mutual information loss maximizes the mutual information between the local feature map output by the last convolution of the query network and the template encoding vector in an adversarial learning manner. The discriminator is divided into a local discriminant network and a global discriminant network. The local discriminant network first expands the encoding vector to the same size as the local feature map, and then concatenates the expanded encoding with the local feature map. If the local feature and the encoding feature come from the same query, the discriminator output is maximized, otherwise it is minimized. The global discriminant network first reduces the dimensionality of the local feature map to a global vector through a convolution operation, and then concatenates it with the encoding vector into a vector, and finally determines whether it comes from the same query.

[0107] The training parameters are set as follows: Adam optimizer is used to optimize all network parameters simultaneously, the initial learning rate is set to 0.00015, and the number of iterations for a single target is set to 50k. The model is trained on 8 2080Ti processors, and the batch size is set to 256. The training and test images are cropped from the original images according to the segmentation mask, and the cropped image size is w = h = 1.4*max(w mask ,h mask ), where w mask ,h mask are the width and height of the object mask, respectively. The cropped image is then scaled to 128 × 128. α, β, λ1, λ2, and λ3 are set to 1, 1, 1, 0.1, and 1, respectively.

[0108] Step 3: Create a template library. The template encoding vector is generated by the template network and encoder. Similar to previous methods, the template library consists of 2562 equidistant viewpoints around the target, with 36 in-plane rotations added to each viewpoint, for a total of 2562 × 36 = 92232 viewpoints. Unlike previous methods, the proposed method directly uses rotation matrices to generate the template encoding vector.

[0109] Step 4: Pose matching. Figure 5 This is a network diagram for the actual measurement phase of the present invention. First, a template library is constructed using a template network and an encoder. The template library stores the rotation matrix and the resulting encoding vectors. A query image passes through the query network and encoder to generate a query encoding vector. The query encoding vector is then compared for similarity with the encoding vectors in the template library. The rotation matrix corresponding to the most similar encoding vector in the template library is the rotation matrix of the target in the query image.

[0110] Among them, the image passes through the query network and the encoding network to generate the query encoding vector z query, and compare the cosine similarity with the template encoding vector in the template library:

[0111]

[0112] The pose corresponding to the template vector with the smallest query vector is taken as the estimated result.

[0113] Step 5: Estimation of the translation vector. The 2D bounding box of the rendered view is saved in the codebook. During testing, the diagonal length of the 2D bounding box || bb is calculated. syn,i ||, and at the same time calculate the diagonal of the minimum bounding box ||bb by segmenting the image area query ||, according to the ratio between the two, the z-axis distance can be calculated according to perspective projection imaging:

[0114]

[0115] Indicates the Z-axis position of the query image, f real and f syn They are the actual acquisition camera focal length and the rendering view focal length, bb real represents the diagonal length of the target in the query image, represents the diagonal length of the target in the template view that is most similar to the query image, t syn,z Indicates the view's Z-axis position.

[0116] According to the target center and the rendering view, we can get:

[0117]

[0118] Among them, bb (·),c is the homogeneous coordinate form of the center of the 2D bounding box, represents the translation vector from the render view target center to the query image target center, and They represent the inverse of the camera intrinsic parameter matrix corresponding to the query image and the rendered view, respectively. The 3D translation vector of the query image is:

[0119]

[0120] in, Represents the translation vector of the rendering view's Z axis, The Z coordinate value of the render view.

[0121] Furthermore, the proposed method is compared with previously proposed reconstruction-based and metric loss-based template matching pose estimation methods on the Occlusion LineMOD dataset. The Occlusion LineMOD dataset contains eight objects with varying degrees of occlusion. The training set images are computer-generated and divided into 50 scenes, each containing approximately 1K images annotated with position, pose, and mask labels.

[0122] The pose error of the most commonly used ADD evaluation method. For the estimated pose of the object model without indistinguishable views Relative true pose Calculated as the average error of the object surface points:

[0123]

[0124] If the object has symmetric views, it is calculated as the average distance to the nearest model point:

[0125]

[0126] As shown in Table 2, the ADD error for the eight objects in the reported dataset is less than 10% of the recall rate at the target scale, demonstrating that the proposed method significantly outperforms the simple measurement and reconstruction methods. The test results for the measurement method and reconstruction method 1 were obtained using the same training parameters as the proposed method, while the results for reconstruction method 2 were obtained using publicly available data from previous methods.

[0127] Table 2 Occlusion LineMOD dataset test results

[0128] Measurement methods Reconstruction method 1 Reconstruction Method 2 Proposed method ape 0.097 0.063 0.023 0.211 can 0.065 0.146 0.09 0.402 cat 0.053 0.064 0.07 0.187 driller 0.08 0.17 0.055 0.485 duck 0.039 0.05 0.111 0.117 eggbox* 0.061 0.078 0.211 0.094 glue* 0.229 0.257 0.264 0.393 holepuncher 0.045 0.070 0.1 0.19 mean 0.084 0.112 0.116 0.260

[0129] In order to prove the effectiveness of the proposed loss function, an ablation test of the loss function was performed:

[0130] Based on the designed network structure, different loss functions are used to train the model. As shown in Table 3, when all four loss functions are used, the average performance of the model reaches the best.

[0131] Table 3 Performance comparison of different loss functions

[0132]

[0133]

[0134] To implement the template matching pose determination method provided above, the present invention also provides the following two implementation structures:

[0135] A template matching pose determination system is applied to the above-mentioned template matching pose determination method; the system comprises:

[0136] Model training module, used to train the network model according to the loss function;

[0137] The template library construction module is used to sample the target isometric view to obtain the rotation matrix for constructing the template library, and construct the template library through the template network and encoder; the template library contains the template encoding vector, the rotation matrix corresponding to the template encoding vector, and the 2D bounding box of the target in the rendering view.

[0138] The image acquisition module is used to obtain the query image and generate the query encoding vector through the query network and encoder;

[0139] A cosine similarity comparison module is used to compare the cosine similarity between the query encoding vector and the encoding vector in the template library;

[0140] The target pose determination module is used to determine the target pose in the query image. The target state is the rotation matrix corresponding to the template encoding vector with the highest similarity to the query encoding vector.

[0141] The target translation vector determination module is used to obtain the target translation vector according to the 2D bounding box of the query image and the parameters of the 2D bounding box camera of the rendering view.

[0142] An electronic device, comprising:

[0143] memory for storing computer programs;

[0144] A processor is connected to the memory and is used to call and execute the computer program to implement the above-mentioned template matching posture determination method.

[0145] In addition, when the computer program in the above-mentioned memory is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

[0146] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0147] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A template matching pose determination method, characterized in that: include: Train the network model according to the loss function; Sampling the target isometric view to obtain the rotation matrix for constructing the template library, and then constructing the template library through the template network and encoder; The template library includes a template encoding vector, a rotation matrix corresponding to the template encoding vector, and a 2D bounding box of the target in the rendering view; Get the query image and generate the query encoding vector through the query network and encoder; Compare the cosine similarity between the query encoding vector and the encoding vector in the template library; Determine the pose of the object in the query image; The target pose is the rotation matrix corresponding to the template encoding vector that has the highest similarity to the query encoding vector; Obtain the target's translation vector based on the query image's 2D bounding box and the rendering view's 2D bounding box camera's parameters; The query network takes the query image as input to generate a feature vector; the encoder takes the feature vector as input to generate a query encoding vector; after the rotation matrix is organized into a one-dimensional vector, it is input into the template network to generate a feature vector, and the feature vector generated by the template network is passed through the encoder to generate a template encoding vector; the decoder is used to reconstruct the template encoding vector into a rotation matrix; the metric loss acts between the query encoding vector and the template variable vector; the reconstruction loss acts between the reconstructed rotation matrix and the input rotation matrix; the segmentation loss and the mutual information loss are used to enhance the local features of the target; The segmentation decoder takes the output feature map of the third stage of the query network as input, upsamples the image stage by stage, and outputs the segmentation mask of the target; the segmentation decoder corresponds to the network layers of the first three stages of the query network; The discriminator includes a local discriminant network and a global discriminant network; the local discriminant network is used to expand the query encoding vector to the same size as the local feature map, and then splice the expanded query encoding vector and the local feature map. If the expanded local features and the expanded query encoding vector come from the same query, the discriminator output is maximized; if the expanded local features and the expanded query encoding vector do not come from the same query, the discriminator output is minimized; the global discriminant network reduces the dimension of the local feature map to a global vector through a convolution operation, and then splices the global vector and the query encoding vector into a vector to determine whether the global vector and the query encoding vector come from the same query. If the global vector and the query encoding vector come from the same query, the discriminator output is maximized; if the global vector and the query encoding vector do not come from the same query, the discriminator output is minimized.

2. The template matching pose determination method according to claim 1, characterized in that: The query network includes: five stages; Among them, before each stage, convolution with a kernel of 2×2 and a stride of 2 is used for downsampling; In the first two stages, the residual module is used to extract features, and in the last two stages, the multi-head attention module is used to extract deep features; In the fifth stage, a 1×1 convolution is used to reduce the dimension and output the deep feature map, and then the MLP module is used to output the deep feature map as a low-dimensional feature vector.

3. The template matching pose determination method according to claim 1, characterized in that: The template network includes four layers of perceptrons; the encoder includes two layers of perceptrons.

4. The template matching pose determination method according to claim 1, characterized in that: The mean square error function is used as the reconstruction loss; the reconstruction loss is: Among them, L rec Represents the reconstruction loss value, y rot represents the rotation matrix, Represents the reconstructed rotation matrix, N represents the amount of data, MSE() represents the mean square error function, Represents the rotation matrix of the i-th data, Represents the rotation matrix after reconstruction of the i-th data; N-pair loss is used as the metric loss; the metric loss is: Among them, x i represents the i-th input target image, Represents the rotation matrix of the i-th data, z i Represents the input target image x i The query encoding vector generated by the query network and encoder, Represents the rotation matrix The template representation vector generated by the template network and encoder is The template representation vector generated by the rotation matrix of the jth data, j≠i, Represents the N-pair loss function value, T represents transposition; The OHEM function is used as the segmentation loss; the segmentation loss is: Among them, top_k() means taking the first k maximum values, CE() means the cross entropy loss function, y mask represents the annotated segmentation mask, represents the segmentation mask output by the network, L seg Represents the segmentation loss value; The mutual information loss is: in, represents the mutual information loss value, ω1, ω2, φ, ψ, γ represent the network parameters of the local discriminator, global discriminator, query network, template network and encoder respectively, represents the JS mutual information estimation, z glb Represents the global feature generation network output value, z glb =Γ′(M), Γ′() represents a global feature generation network inside the global discriminator, M represents the feature map, w m ×h m represents the number of dimensions, z represents the template encoding vector, Take the maximum value function, α and β are hyper parameters, m (i) represents the i-th eigenvector of dimension cm.

5. The template matching pose determination method according to claim 1, characterized in that: The loss function of the target pose matching model is: Among them, λ1, λ2 and λ3 are hyperparameters, L N-pair is the measurement loss value.

6. The template matching pose determination method according to claim 1, characterized in that: Determine the pose of the object in the query image, specifically including: Performing a cosine similarity comparison between the query encoding vector and the template encoding vector of the template image in the template library to obtain a cosine similarity comparison result; The posture of the target in the template image corresponding to the minimum cosine similarity comparison result is used as the posture of the target in the query image.

7. A template matching pose determination system, characterized in that: The method for determining a template matching pose according to any one of claims 1 to 6 is applied; the system comprises: Model training module, used to train the network model according to the loss function; A template library construction module is used to sample the target isometric view to obtain the rotation matrix for constructing the template library, and construct the template library through the template network and the encoder; the template library includes the template encoding vector, the rotation matrix corresponding to the template encoding vector, and the 2D bounding box of the target in the rendering view; The image acquisition module is used to obtain the query image and generate the query encoding vector through the query network and encoder; A cosine similarity comparison module is used to compare the cosine similarity between the query encoding vector and the encoding vector in the template library; A target pose determination module is used to determine the target pose in the query image; the target pose is the rotation matrix corresponding to the template encoding vector that has the highest similarity to the query encoding vector; The target translation vector determination module is used to obtain the target translation vector according to the 2D bounding box of the query image and the parameters of the 2D bounding box camera of the rendering view.

8. An electronic device, characterized in that: include: memory for storing computer programs; A processor, connected to the memory, is used to call and execute the computer program to implement the template matching pose determination method as described in any one of claims 1-6.

9. The electronic device according to claim 8, wherein: The memory is a computer-readable storage medium.

Images