A method and system for estimating target pose

Abstract

The present invention proposes a target pose estimation method, comprising the following steps: acquiring image data of a target object to be identified; fusing a multi-branch structural block of multi-scale convolution to construct a pose estimation model; inputting the image data of the target object to be identified into the pose estimation model to obtain a pose estimation value of the target object; processing the target pose estimation value by adopting a hybrid voxel grid method to obtain the volume density and color information of the voxel, and inputting the volume density and color information of the voxel into a neural radiation field model to obtain a rendering key frame; acquiring a real-time observed true value of the target object, and calculating the loss between the rendering key frame and the real-time observed true value of the target object according to a photometric loss function to obtain a loss value; the method constructs a pose estimation model by fusing the multi-branch structural block of multi-scale convolution, can replace the convolution structure of a traditional detection network and construct a pose estimation network branch, and realizes high-precision and high-efficiency pose estimation value inference.

Description

Technical Field

[0001] The present invention relates to the field of positioning technology, and in particular to a target posture estimation method and system. Background Art

[0002] In the fields of intelligent navigation, unmanned driving and augmented reality, accurately estimating the position and direction of the target, namely pose estimation, is a basic and key technology. With the maturity of the Beidou satellite navigation system and the development of computer vision technology, the target pose estimation algorithm based on the fusion of Beidou and computer vision has become a research hotspot.

[0003] A method and system for regional surface displacement monitoring based on Beidou and computer vision, with publication number CN115236716A, discloses surface displacement monitoring of monitoring points in a monitoring area, wherein first-level monitoring points, second-level monitoring points and third-level monitoring points are set in the monitoring area, the number of first-level monitoring points is at least three and each first-level monitoring point is provided with a receiver integrating a first target, the third-level monitoring point is provided with a third target, and the second-level monitoring point is provided with a camera and the field of view of the camera covers all the first targets and the third targets; the method comprises: obtaining Beidou positioning signals received by all receivers, and calculating each Beidou positioning signal according to all Beidou positioning signals. The method comprises the steps of: obtaining a first three-dimensional coordinate of each first target; obtaining an image taken by a camera, and calculating the two-dimensional coordinates of each first target and a third target in the image coordinate system according to the image; and calculating the second three-dimensional coordinate of the camera according to all the first three-dimensional coordinates and the two-dimensional coordinates of all the first targets in the image coordinate system; calculating the third three-dimensional coordinate of the third target according to the first three-dimensional coordinate of the first target, the two-dimensional coordinate of the first target in the image coordinate system, and the two-dimensional coordinate of the third target in the image coordinate system; and calculating the surface displacement change of all the first-level monitoring points, the second-level monitoring points, and the third-level monitoring points according to the first three-dimensional coordinates, the second three-dimensional coordinates, and the third three-dimensional coordinates measured twice before and after.

[0004] Although the Beidou system provides accurate positioning services in open areas, in certain areas or specific environments, such as urban canyons, indoors or other satellite signal-blocked environments, the Beidou Satellite Positioning System (BDS) signal may be interfered with or blocked, and the signal error will increase significantly, affecting the positioning accuracy. On the other hand, computer vision can provide the relative position information of the target in an environment lacking GPS signals by analyzing the images captured by the camera. Although computer vision is more accurate in a local range, it relies on the quality of the image and the extraction of feature points, and is easily affected by problems such as illumination changes, occlusion, and lack of feature points, which in turn leads to deviations in the positioning position, thereby reducing the accuracy of target positioning. Summary of the invention

[0005] In view of this, the present invention proposes a target pose estimation method and system, which integrates the positioning algorithm of Beidou and the implicit three-dimensional model of the neural radiation field, and can effectively improve the positioning accuracy in complex scenes.

[0006] The technical solution of the present invention is implemented as follows: In a first aspect, the present invention provides a method for estimating a target pose, comprising the following steps:

[0007] S1, obtaining image data of a target object to be identified;

[0008] S2, a multi-branch structure block that integrates multi-scale convolutions to build a pose estimation model;

[0009] S3, inputting the image data of the target object to be identified into the pose estimation model to obtain the pose estimation value of the target object;

[0010] S4, using a hybrid voxel grid method to process the target pose estimation value, obtain the volume density and color information of the voxel, and input the volume density and color information of the voxel into the neural radiation field model to obtain a rendering key frame;

[0011] S5, obtaining the real-time observation true value of the target object, and calculating the loss between the rendered key frame and the real-time observation true value of the target object according to the photometric loss function to obtain a loss value;

[0012] S6, iteratively training through the back propagation algorithm to update the parameters of the neural radiation field model to obtain the optimal volume density result;

[0013] S7, based on the optimal body density result, the corresponding posture is solved.

[0014] On the basis of the above technical solution, preferably, in step S2, a multi-branch structure block of multi-scale convolution is integrated to construct a pose estimation model, which includes the following sub-steps:

[0015] S21, a multi-branch structure block integrating multi-scale convolution, sets a 4th-order tensor convolution kernel F, whose input channel number is C, output channel number is D, and kernel size is K×K, then the convolution kernel F satisfies:

[0016] F∈R D×H×W ;

[0017] S22, the multi-branch structure block receives a C-channel feature image I as input and outputs a D-channel feature image M, where

[0018] The feature image I satisfies: I∈R C×H×W , the feature image M satisfies:

[0019] S23, add a bias term b∈RD , and define the bias term as REP(b), then the expression is:

[0020]

[0021] S24, convolution operation is performed on the C channel feature map I and the D channel feature map M, and a bias term is added to obtain the output feature map M expression:

[0022]

[0023] S25, construct a pose estimation model, which is expressed as:

[0024]

[0025] In the formula, C is the number of input channels, K is the convolution kernel size, and X(c,h,w)∈R K×K It is represented as the sliding window of the coordinate (h, w) in the feature map M for the c-th channel of the input feature map I, F j,c,u,v represents the cth input channel, jth output channel, the weight of the convolution kernel in the uth row and vth column, b j is the bias of the jth output channel.

[0026] On the basis of the above technical solution, preferably, before step S3, it also includes iterative training of the pose estimation model to obtain the trained pose estimation model, wherein the training includes the following sub-steps:

[0027] Obtain a training data set, which includes a two-dimensional bounding box used to mark the position of the target object in the image, the true value of the target label, and the corresponding pose information, and the pose information includes quaternion and depth information;

[0028] The pose estimation model at least includes an original reasoning branch and a new reasoning branch, the original reasoning branch is used to predict the two-dimensional bounding box of the target object, and the new reasoning branch is used to predict the pose information of the target object;

[0029] The training data set is input into the pose estimation model. The pose estimation model outputs the prediction result of the 2D detection box and the pose estimation value. The loss of the original reasoning branch is calculated. The grid area and prior anchor box corresponding to the true value of each target label are found in the new reasoning branch. The loss between the predicted value and the true value of the target pose information is calculated to obtain the loss of the new reasoning branch.

[0030] Based on the original reasoning branch loss and the new reasoning branch loss, back-propagation optimization is performed to complete the training of the pose estimation model.

[0031] On the basis of the above technical scheme, preferably, in step S3, the image data of the target object to be identified is input into the pose estimation model to obtain the pose estimation value of the target object, wherein the image data of the target object to be identified is input into the pose estimation model, and the prediction result of the two-dimensional bounding box of the target object is output through the original reasoning branch to obtain the grid position of each target on the feature map and the corresponding prior anchor frame, and the corresponding quaternion and depth information are output based on the new reasoning branch, and the pose estimation value of the target object is obtained according to the output of the corresponding quaternion and depth information.

[0032] On the basis of the above technical solution, preferably, when the pose estimation model is back-propagation optimized, it also includes obtaining the initial positioning value of the target object based on the Beidou satellite positioning system BDS, and rendering a group of sparse random pixels according to the target positioning initial value.

[0033] On the basis of the above technical solution, preferably, in step S4, a hybrid voxel grid method is used to process the target pose estimation value to obtain the volume density and color information of the voxels, and the volume density and color information of the voxels are input into the neural radiation field model to obtain a rendering key frame, which includes the following sub-steps:

[0034] S41, determining the resolution and size of the voxel grid according to the size and reconstruction accuracy of the target object, and creating an initialization voxel grid;

[0035] S42, acquiring multi-view images of the target object, and for each view image, mapping image pixels to a three-dimensional space according to the target pose estimation value, and traversing a voxel grid along rays emitted from all pixels;

[0036] S43, obtaining color information of voxels using an interpolation method according to color information of adjacent pixels, fusing depth values ​​of the voxels in combination with multi-view images of the target object, and obtaining volume density of the voxels based on the depth information;

[0037] S44, constructing a training neural radiation field model, inputting the volume density and color information of the voxels into the neural radiation field model, and obtaining a rendering key frame.

[0038] Based on the above technical solution, preferably, the photometric loss function calculation expression in step S5 is:

[0039]

[0040] In the formula, To render a keyframe, C[r] is the real-time observed true value of the target object, r represents a ray passing through the image plane, and R represents the set of rays passing through the image plane.

[0041] In a second aspect, the present invention further provides a target pose estimation system, the system comprising:

[0042] An acquisition module, used to acquire image data of a target object to be identified;

[0043] Build a model module to fuse the multi-branch structure blocks of multi-scale convolution and build a pose estimation model;

[0044] A prediction module, used to input the image data of the target object to be identified into the pose estimation model to obtain the pose estimation value of the target object;

[0045] A mapping module is used to process the target pose estimation value by adopting a hybrid voxel grid method to obtain the volume density and color information of the voxels, and input the volume density and color information of the voxels into the neural radiation field model to obtain a rendering key frame;

[0046] A calculation module is used to obtain the real-time observation true value of the target object, and calculate the loss between the rendered key frame and the real-time observation true value of the target object according to the photometric loss function to obtain the loss value;

[0047] The optimization training module is used to iteratively train the neural radiation field model through the back-propagation algorithm to obtain the optimal volume density result;

[0048] The solution module is used to output the corresponding pose based on the optimized trained neural radiation field model.

[0049] In a third aspect, the present invention also provides a terminal device, comprising: a memory, a processor, and a target pose estimation method program stored in the memory and executable on the processor, wherein the target pose estimation method program is configured to implement the steps of the target pose estimation method as described above.

[0050] In a fourth aspect, the present invention also provides a storage medium, which is a computer storage medium. A target pose estimation method program is stored on the computer storage medium. When the target pose estimation method program is executed by a processor, the steps of the target pose estimation method as described above are implemented.

[0051] The target pose estimation method and system of the present invention have the following beneficial effects compared with the prior art:

[0052] (1) By fusing the multi-branch structural block of multi-scale convolution, a pose estimation model is constructed, and the image data of the target object to be identified is input into the pose estimation model to obtain the pose estimation value of the target object. The design of the multi-branch structural block can replace the convolution structure of the traditional detection network and construct the pose estimation network branch to obtain high-precision and high-efficiency pose estimation value inference;

[0053] (2) In each iteration of training, the BeiDou Satellite Positioning System (BDS) initial value is used to generate a set of sparse random pixels and add them to the training data. This not only increases the diversity and richness of the data, but also helps the model learn the characteristics and laws of the data more comprehensively, improves the training effect, and helps explore a wider solution space, so as to better find the optimal solution of the loss function. This can improve the efficiency and accuracy of the optimization algorithm, enable the model to converge to the optimal solution faster, and improve the accuracy of the target pose estimation value;

[0054] (3) Based on the real-time images captured by the visual camera and the implicit three-dimensional model of the neural radiation field, a positioning solution model with continuous learning and output is constructed based on the photometric loss function. Through iterative training of the back-propagation algorithm, the optimized implicit three-dimensional model of the neural radiation field is obtained, which plays an auxiliary correction role for the target posture and further improves the accuracy of the target posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 A schematic diagram of the structure of a device in a hardware operating environment involved in an embodiment of the present invention;

[0057] Figure 2 A flowchart of a target pose estimation method of the present invention;

[0058] Figure 3 A schematic diagram of a multi-branch structural block structure of a target pose estimation method of the present invention;

[0059] Figure 4 The present invention is a flowchart of a method for estimating target pose. DETAILED DESCRIPTION

[0060] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] like Figure 1As shown, the device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0062] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation of the device. In actual applications, the device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0063] like Figure 1 As shown, the memory 1005 as a medium may include an operating system, a network communication module, a user interface module, and a target pose estimation method program.

[0064] exist Figure 1 In the device shown, the network interface 1004 is mainly used to establish a communication connection between the device and a server that stores all data required in the target pose estimation method system; the user interface 1003 is mainly used to interact with the user for data; the processor 1001 and the memory 1005 in the target pose estimation method device of the present invention can be set in the target pose estimation method device, and the target pose estimation method device calls the target pose estimation method program stored in the memory 1005 through the processor 1001, and executes the target pose estimation method provided by the implementation of the present invention.

[0065] It should be noted that due to the current use of the Beidou Satellite Positioning System (BDS) for positioning, there will be problems with inaccuracy. In some areas or specific environments, such as in urban canyons, indoors or other satellite signal-blocked environments, the Beidou Satellite Positioning System (BDS) signal may be interfered with or blocked, which will affect the positioning accuracy. On the other hand, computer vision can provide the target's relative position information in an environment lacking a GPS signal by analyzing the images captured by the camera. Although computer vision is more accurate in a local range, it relies on the quality of the image and the extraction of feature points, and is easily affected by problems such as changes in illumination, occlusion, and a lack of feature points, resulting in deviations in the positioning position, thereby reducing the accuracy of target positioning.

[0066] like Figure 2-4 As shown, a target pose estimation method of the present invention comprises the following steps:

[0067] S1, obtaining image data of a target object to be identified;

[0068] The image data of the target object is obtained through a camera or other image acquisition device, and the data will be used as input for subsequent pose estimation and three-dimensional reconstruction.

[0069] S2, a multi-branch structure block that integrates multi-scale convolutions to build a pose estimation model;

[0070] Wherein, step S2 includes the following sub-steps:

[0071] S21, a multi-branch structure block integrating multi-scale convolution, sets a 4th-order tensor convolution kernel F, whose input channel number is C, output channel number is D, and kernel size is K×K, then the convolution kernel F satisfies:

[0072] F∈R D×H×W ;

[0073] Where H is the height of the input feature map, and W is the width of the input feature map;

[0074] It should be noted that the convolution kernel will extract information from the feature map of the C channel and output the feature map of the D channel. The kernel size K×K defines the size of the local area considered by the convolution kernel at each position.

[0075] S22, the multi-branch structure block receives a C-channel feature image I as input and outputs a D-channel feature image M, where

[0076] The feature image I satisfies: I∈R C×H×W , the feature image M satisfies:

[0077] It should be noted that this structure block will perform a series of transformations on the input feature map and finally output a feature map with different number of channels, H 0 is the height of the output feature map, W 0 is the width of the output feature map;

[0078] S23, add a bias term b∈R D , and define the bias term as REP(b), then the expression is:

[0079]

[0080] S24, convolution operation is performed on the C channel feature map I and the D channel feature map M, and a bias term is added to obtain the output feature map M expression:

[0081]

[0082] S25, construct a pose estimation model, which is expressed as:

[0083]

[0084] In the formula, C is the number of input channels, K is the convolution kernel size, and X(c,h,w)∈R K×K It is represented as the sliding window of the coordinate (h, w) in the feature map M for the c-th channel of the input feature map I, F j,c,u,v represents the cth input channel, jth output channel, the weight of the convolution kernel in the uth row and vth column, b j is the bias of the jth output channel.

[0085] In this embodiment, a multi-branch structural block integrating multi-scale convolution is designed, which adopts a multi-path parallel topology structure, adds multi-scale convolution to the branch design, 1×1 cascade K×K convolution, average pooling and branch superposition, constructs a multi-scale receptive field, improves the complexity of the reasoning path, and enriches the feature extraction space; and, the K×K convolution can be adjusted according to the specific scenario to extract features of different scales, thereby improving the performance of the model.

[0086] The multi-branch structural block can be equivalently converted into a single convolution for reasoning. For the backbone network of the existing basic detection algorithm, the conventional convolution layer can be replaced by the multi-branch structural block to build a more complex microstructure for training, and then convert it back to the original structure. Based on this idea, the receptive field of the algorithm model can be expanded without increasing the algorithm's time and space complexity, enriching the feature extraction space, thereby optimizing the algorithm's training and reasoning results.

[0087] In the target pose solution scenario, this solution uses the basic detection network algorithm architecture as the backbone, replaces the original K×K convolution used for reasoning with a multi-branch structure block, sets the number of internal channels to the number of input channels, and initializes the 1x1 convolution kernel to the unit matrix. Other convolution kernels are initialized in the conventional way. Each convolution or average pooling layer is followed by a normalization layer to achieve nonlinear transformation during training. In the new reasoning branch, the regression logic of the target pose and posture is designed, and a pose estimation model is constructed.

[0088] S3, inputting the image data of the target object to be identified into the pose estimation model to obtain the pose estimation value of the target object;

[0089] It should be noted that before step S3, it also includes iterative training of the pose estimation model to obtain the trained pose estimation model, wherein the training includes the following sub-steps:

[0090] Obtain a training data set, which includes a two-dimensional bounding box used to mark the position of the target object in the image, the true value of the target label, and the corresponding pose information, and the pose information includes quaternion and depth information;

[0091] The pose estimation model at least includes an original reasoning branch and a new reasoning branch, the original reasoning branch is used to predict the two-dimensional bounding box of the target object, and the new reasoning branch is used to predict the pose information of the target object;

[0092] The training data set is input into the pose estimation model. The pose estimation model outputs the prediction result of the 2D detection box and the pose estimation value. The loss of the original reasoning branch is calculated. The grid area and prior anchor box corresponding to the true value of each target label are found in the new reasoning branch. The loss between the predicted value and the true value of the target pose information is calculated to obtain the loss of the new reasoning branch.

[0093] Based on the original reasoning branch loss and the new reasoning branch loss, back propagation optimization is performed, the gradient of the model parameters is calculated through the back propagation algorithm, and the model parameters are updated using the optimizer to minimize the total loss and complete the training of the pose estimation model.

[0094] It should be noted that during training, the loss of the original reasoning branch is calculated, and the grid area and prior anchor box corresponding to the true value of each target label are found in the new reasoning branch, and then the loss of the quaternion of the target pose information and the target depth information, a total of 5 components, relative to the true value are calculated, and then the model is back-propagated and optimized.

[0095] Among them, the image data of the target object to be identified is input into the pose estimation model, and the prediction result of the two-dimensional bounding box of the target object is output through the original reasoning branch to obtain the grid position of each target on the feature map and the corresponding prior anchor box. The corresponding quaternion and depth information are output based on the new reasoning branch. According to the output of the corresponding quaternion and depth information, the pose estimation value of the target object is obtained.

[0096] It should be noted that during reasoning, the grid position and prior anchor frame encoding of each detected target are solved based on the two-dimensional detection frame output by the original reasoning branch, and the corresponding quaternion and depth information are output based on the new reasoning branch. The position and posture information of the target object in three-dimensional space can be obtained based on the output of the corresponding quaternion and depth information.

[0097] In addition, when the pose estimation model is back-propagated and optimized, it also includes obtaining the initial positioning value of the target object based on the Beidou satellite positioning system BDS, and rendering a set of sparse random pixels according to the initial positioning value of the target.

[0098] The positioning data of the target object is obtained through the BDS receiver, and the BDS positioning data is converted into coordinate points in three-dimensional space as the initial positioning value of the target object. According to the initial positioning value of the target, a set of sparse pixels is randomly selected in the space around the target object. The number and distribution of these pixels can be adjusted as needed to balance the amount of calculation and accuracy. In each iteration of training, the BDS initial value is used to generate a set of sparse random pixels and added to the training data. Among them, firstly, by rendering sparse random pixels, the diversity and richness of the data can be increased, which helps the model to learn the characteristics and laws of the data more comprehensively. This can improve the generalization ability of the model, so that it can better cope with changes in different scenarios and data; secondly, the introduction of random sparse pixels can help the model better learn the distribution and characteristics of the data, thereby improving the training effect. Sparse random pixels can enable the model to adapt to different input situations more flexibly, which helps to improve the performance and robustness of the model. Thirdly, in the iterative optimization process, rendering a set of sparse random pixels may help explore a wider solution space, so as to better find the optimal solution of the loss function. This can improve the efficiency and accuracy of the optimization algorithm and make the model converge to the optimal solution more quickly.

[0099] S4, using a hybrid voxel grid method to process the target pose estimation value, obtain the volume density and color information of the voxel, and input the volume density and color information of the voxel into the neural radiation field model to obtain a rendering key frame;

[0100] Step S4 includes the following sub-steps:

[0101] S41, determining the resolution and size of the voxel grid according to the size and reconstruction accuracy of the target object, and creating an initialization voxel grid;

[0102] S42, acquiring multi-view images of the target object, and for each view image, mapping image pixels to a three-dimensional space according to the target pose estimation value, and traversing a voxel grid along rays emitted from all pixels;

[0103] S43, in the process of ray traversal, according to the color information of adjacent pixels, an interpolation method is used to obtain the color information of the voxel, and the depth value of the voxel is fused in combination with the depth information of the multi-view image of the target object, and the volume density of the voxel is obtained based on the depth information;

[0104] The interpolation method can be implemented by linear interpolation or bilinear interpolation.

[0105] S44, constructing a training neural radiation field model, inputting the volume density and color information of the voxels into the neural radiation field model, and obtaining a rendering key frame.

[0106] Among them, a training neural radiation field model is constructed. First, a NeRF model is initialized. The NeRF model is usually composed of a multi-layer perceptron network, which is used to learn the mapping from points in three-dimensional space to their corresponding colors and volume densities. The volume density and color information in the mixed voxel grid is converted into a format adapted to the NeRF model, and the position, color and volume density of each point are extracted. The extracted voxel data is preprocessed as necessary, such as normalization, filtering and data cleaning, to improve the training effect. The preprocessed voxel data and the corresponding camera pose are input as input to the NeRF model. The NeRF model starts from the camera and samples along the ray path in three-dimensional space by ray casting. The MLP network is used to predict the color and volume density of each sampling point, and the loss between the rendering result and the real image is calculated, which includes color loss and volume density loss. The parameters of the NeRF model are updated through the back propagation algorithm to minimize the loss function, and the key frame perspective to be rendered is determined. For each key frame perspective, the trained NeRF model is used for rendering to obtain the rendered key frame.

[0107] S5, obtaining the real-time observation true value of the target object, and calculating the loss between the rendered key frame and the real-time observation true value of the target object according to the photometric loss function to obtain a loss value;

[0108] The photometric loss function calculation expression in step S5 is:

[0109]

[0110] In the formula, To render a keyframe, C[r] is the real-time observed true value of the target object, r represents a ray passing through the image plane, and R represents the set of rays passing through the image plane.

[0111] S6, iteratively training through the back propagation algorithm to update the parameters of the neural radiation field model to obtain the optimal volume density result;

[0112] S7, based on the optimal body density result, the corresponding posture is solved.

[0113] It should be noted that after the optimization process, the parameters of the implicit expression model NeRF have been adjusted to minimize the difference between the rendered image and the real image. These optimized parameters not only reflect the structure and texture information of the scene, but also imply information related to the pose. Using the optimized model parameters, we can render synthetic images from any perspective. These rendering results not only have high-quality visual performance, but also are highly consistent with the real scene in structure and texture. They can serve as an important reference for pose estimation. Feature points are extracted from the image rendered by the optimized implicit expression model and matched with the feature points in the real image. The corresponding pose of the target is calculated using the matched feature points.

[0114] In this scheme, the BDS positioning result is used as the initial value, and the real-time image captured by the visual camera and the implicit three-dimensional model of the neural radiation field are used as the positioning and navigation source. Based on the photometric loss function, a positioning solution model with continuous learning and output is constructed. Through iterative training of the back propagation algorithm, the optimized implicit three-dimensional model of the neural radiation field is obtained, which plays a role in auxiliary correction and further improves the accuracy of the target position.

[0115] The present invention also provides a target pose estimation system, the system comprising:

[0116] An acquisition module, used to acquire image data of a target object to be identified;

[0117] Build a model module to fuse the multi-branch structure blocks of multi-scale convolution and build a pose estimation model;

[0118] A prediction module, used to input the image data of the target object to be identified into the pose estimation model to obtain the pose estimation value of the target object;

[0119] A mapping module is used to process the target pose estimation value by adopting a hybrid voxel grid method to obtain the volume density and color information of the voxels, and input the volume density and color information of the voxels into the neural radiation field model to obtain a rendering key frame;

[0120] A calculation module is used to obtain the real-time observation true value of the target object, and calculate the loss between the rendered key frame and the real-time observation true value of the target object according to the photometric loss function to obtain the loss value;

[0121] The optimization training module is used to iteratively train the neural radiation field model through the back-propagation algorithm to obtain the optimal volume density result;

[0122] The solution module is used to output the corresponding pose based on the optimized trained neural radiation field model.

[0123] The present invention also provides a storage medium, which is a computer storage medium. The computer storage medium stores a target pose estimation method program. When the target pose estimation method program is executed by a processor, the steps of the above-mentioned target pose estimation method are implemented.

[0124] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A target pose estimation method, characterized in that: The following steps are involved: S1, obtaining image data of a target object to be identified; S2, a multi-branch structure block that integrates multi-scale convolutions to build a pose estimation model; S3, inputting the image data of the target object to be identified into the pose estimation model to obtain the pose estimation value of the target object; The image data of the target object to be identified is input into the pose estimation model, and the prediction result of the two-dimensional bounding box of the target object is output through the original reasoning branch to obtain the grid position of each target on the feature map and the corresponding prior anchor box, and the corresponding quaternion and depth information are output based on the new reasoning branch, and the pose estimation value of the target object is obtained according to the output corresponding quaternion and depth information; When the pose estimation model is back-propagated and optimized, it also includes obtaining the initial positioning value of the target object based on the Beidou satellite positioning system BDS, and rendering a group of sparse random pixels according to the initial positioning value of the target; In each iteration of training, the BDS initial value is used to generate a set of sparse random pixels and added to the training data; S4, using a hybrid voxel grid method to process the target pose estimation value, obtain the volume density and color information of the voxel, and input the volume density and color information of the voxel into the neural radiation field model to obtain a rendering key frame; S5, obtaining the real-time observation true value of the target object, and calculating the loss between the rendered key frame and the real-time observation true value of the target object according to the photometric loss function to obtain a loss value; S6, iteratively training through the back propagation algorithm to update the parameters of the neural radiation field model to obtain the optimal volume density result; S7, based on the optimal body density result, the corresponding posture is solved.

2. The target pose estimation method according to claim 1, wherein: In step S2, a multi-branch structure block of multi-scale convolution is integrated to construct a pose estimation model, which includes the following sub-steps: S21, a multi-branch structure block integrating multi-scale convolution, sets a 4th-order tensor convolution kernel F, whose input channel number is C, output channel number is D, and kernel size is K×K, then the convolution kernel F satisfies: F∈R D×H×W ; S22, the multi-branch structure block receives a C-channel feature image I as input and outputs a D-channel feature image M, where The feature image I satisfies: I∈R C×H×W , the feature image M satisfies: S23, add a bias term b∈R D , and define the bias term as REP(b), then the expression is: S24, convolution operation is performed on the C channel feature map I and the D channel feature map M, and a bias term is added to obtain the output feature map M expression: S25, construct a pose estimation model, which is expressed as: In the formula, C is the number of input channels, K is the convolution kernel size, and X(c,h,w)∈R K×K It is represented as the sliding window of the coordinate (h, w) in the feature map M for the c-th channel of the input feature map I, F j,c,u,v represents the cth input channel, jth output channel, the weight of the convolution kernel in the uth row and vth column, b j is the bias of the jth output channel.

3. The target pose estimation method according to claim 2, wherein: Before step S3, it also includes iterative training of the pose estimation model to obtain a trained pose estimation model, wherein the training includes the following sub-steps: Obtain a training data set, which includes a two-dimensional bounding box used to mark the position of the target object in the image, the true value of the target label, and the corresponding pose information, and the pose information includes quaternion and depth information; The pose estimation model at least includes an original reasoning branch and a new reasoning branch, the original reasoning branch is used to predict the two-dimensional bounding box of the target object, and the new reasoning branch is used to predict the pose information of the target object; The training data set is input into the pose estimation model. The pose estimation model outputs the prediction result of the 2D detection box and the pose estimation value. The loss of the original reasoning branch is calculated. The grid area and prior anchor box corresponding to the true value of each target label are found in the new reasoning branch. The loss between the predicted value and the true value of the target pose information is calculated to obtain the loss of the new reasoning branch. Based on the original reasoning branch loss and the new reasoning branch loss, back-propagation optimization is performed to complete the training of the pose estimation model.

4. The target pose estimation method according to claim 3, wherein: In step S4, the target pose estimation value is processed by using a hybrid voxel grid method to obtain the volume density and color information of the voxels, and the volume density and color information of the voxels are input into the neural radiation field model to obtain a rendering key frame, which includes the following sub-steps: S41, determining the resolution and size of the voxel grid according to the size and reconstruction accuracy of the target object, and creating an initialization voxel grid; S42, acquiring multi-view images of the target object, and for each view image, mapping image pixels to a three-dimensional space according to the target pose estimation value, and traversing a voxel grid along rays emitted from all pixels; S43, obtaining color information of voxels using an interpolation method according to color information of adjacent pixels, fusing depth values ​​of the voxels in combination with multi-view images of the target object, and obtaining volume density of the voxels based on the depth information; S44, constructing a training neural radiation field model, inputting the volume density and color information of the voxels into the neural radiation field model, and obtaining a rendering key frame.

5. The target pose estimation method according to claim 4, wherein: The calculation expression of the photometric loss function in step S5 is: In the formula, To render a keyframe, C(r) is the real-time observed true value of the target object, r represents a ray passing through the image plane, and R represents the set of rays passing through the image plane.

6. A target pose estimation system, characterized in that: The system includes: An acquisition module, used to acquire image data of a target object to be identified; Build a model module to fuse the multi-branch structure blocks of multi-scale convolution and build a pose estimation model; A prediction module, used to input the image data of the target object to be identified into the pose estimation model to obtain the pose estimation value of the target object; The image data of the target object to be identified is input into the pose estimation model, and the prediction result of the two-dimensional bounding box of the target object is output through the original reasoning branch to obtain the grid position of each target on the feature map and the corresponding prior anchor box, and the corresponding quaternion and depth information are output based on the new reasoning branch, and the pose estimation value of the target object is obtained according to the output corresponding quaternion and depth information; When the pose estimation model is back-propagated and optimized, it also includes obtaining the initial positioning value of the target object based on the Beidou satellite positioning system BDS, and rendering a group of sparse random pixels according to the initial positioning value of the target; In each iteration of training, the BDS initial value is used to generate a set of sparse random pixels and added to the training data; A mapping module is used to process the target pose estimation value by adopting a hybrid voxel grid method to obtain the volume density and color information of the voxels, and input the volume density and color information of the voxels into the neural radiation field model to obtain a rendering key frame; A calculation module is used to obtain the real-time observation true value of the target object, and calculate the loss between the rendered key frame and the real-time observation true value of the target object according to the photometric loss function to obtain the loss value; The optimization training module is used to iteratively train the neural radiation field model through the back-propagation algorithm to obtain the optimal volume density result; The solution module is used to output the corresponding pose based on the optimized trained neural radiation field model.

7. A terminal device, characterized in that: The terminal device includes: a memory, a processor, and a target pose estimation method program stored in the memory and executable on the processor, wherein the target pose estimation method program is configured to implement the steps of the target pose estimation method as claimed in any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium is a computer storage medium, on which a target pose estimation method program is stored. When the target pose estimation method program is executed by a processor, the steps of the target pose estimation method as claimed in any one of claims 1 to 5 are implemented.

Images