Statistically guaranteed non-cooperative target pose measurement method and apparatus
By using deep neural network prediction and conformal sampling techniques, the pose uncertainty caused by occlusion and illumination changes is solved, enabling accurate pose estimation in the measurement of non-cooperative target spacecraft and outputting a pose set with statistical guarantees.
Patent Information
- Application Number
- CN202410710505.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-03
- Publication Date
- 2026-02-13
- Estimated Expiration
- 2044-06-03
AI Technical Summary
Existing methods cannot effectively represent pose uncertainties caused by problems such as occlusion, illumination changes, and image quality degradation when measuring the pose of a target spacecraft.
A trained deep neural network is used to predict semantic key points in non-cooperative target images. A conformal set of semantic key points is generated through conformal prediction and random sampling. The pose is estimated by combining an N-point perspective model, and the average pose with statistical guarantee is obtained through iterative calculation.
Accurate non-cooperative target pose measurement is achieved under conditions of occlusion, illumination changes, and image quality degradation. The output pose set has statistical guarantees, and the probability of covering the true pose value reaches 1-ε.
Smart Images
Figure CN118570299B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image processing, in particular to a non-cooperative target pose measurement method and device with statistical guarantee. BACKGROUND
[0002] With the rapid development of technology, for example, formation flight, failed satellite and debris removal tasks require measuring the position and attitude of the target spacecraft relative to the service spacecraft. The existing method predicts the position of the semantic key point defined on the target spacecraft in the image, and then solves the N-point perspective problem to obtain the relative position and attitude.
[0003] However, the single pose output by the existing method cannot express the pose uncertainty caused by problems such as occlusion, illumination change and image quality degradation in actual tasks. SUMMARY
[0004] Therefore, it is necessary to provide a non-cooperative target pose measurement method and device with statistical guarantee that can obtain accurate pose in view of the above technical problems.
[0005] A non-cooperative target pose measurement method with statistical guarantee, the method comprises:
[0006] Obtaining a non-cooperative target image to be measured for pose;
[0007] Using a trained deep neural network to predict semantic key points in the non-cooperative target image, obtaining a plurality of semantic key points, and the index classification item of the three-dimensional coordinates of each semantic key point in the non-cooperative target three-dimensional coordinate system and the pixel coordinates on the non-cooperative target image;
[0008] According to the preset non-conformal weight, the conformal prediction is carried out for each semantic key point, and a semantic key point conformal set is generated;
[0009] Randomly sampling in the semantic key point conformal set according to a preset number, and obtaining a group of sample point pixel coordinates on the non-cooperative target image;
[0010] According to the group of sample point pixel coordinates, based on the point correspondence relationship between the sample points and their corresponding three-dimensional coordinates in the non-cooperative target coordinate system, and the N-point perspective model, the estimated pose of the non-cooperative target is estimated, and it is judged whether the estimated pose meets the preset standard. If it meets, the estimated pose is retained in the target pose set, one iteration calculation is completed, another group of sample point pixel coordinates is obtained by randomly sampling in the semantic key point conformal set, and the corresponding estimated pose is generated, until the iteration number meets the preset number, then the iteration calculation is stopped;
[0011] An average pose is calculated according to all the estimated poses in the target pose set, to obtain the target pose set of the non-cooperative target and the average pose, so as to realize the non-cooperative target pose measurement.
[0012] In one embodiment, when training the deep neural network:
[0013] A sample data set is obtained, which includes a plurality of sample target images and a true value label corresponding to each sample image;
[0014] The sample data set is divided into a training sample data set and a calibration sample data set;
[0015] The deep neural network is trained using the training sample data set, to obtain the trained neural network capable of predicting target semantic key points in sample target images;
[0016] The trained neural network is used to predict semantic key points in the sample target images in the calibration sample data set, to obtain a plurality of predicted semantic key points, and the calibration sample data set is calibrated by the plurality of predicted semantic key points and the corresponding true value labels, to obtain a preset non-conformal score.
[0017] In one embodiment, the deep neural network includes a feature extraction network, an index classification item prediction head, and an image coordinate position prediction head;
[0018] The feature extraction network is used to extract a feature map of the non-cooperative target image;
[0019] The index classification item prediction head and the image coordinate position prediction head respectively predict the index classification item of the three-dimensional coordinates of the semantic key points in the non-cooperative target three-dimensional coordinate system and the pixel coordinates on the non-cooperative target image according to the feature map.
[0020] In one embodiment, the calibration of the calibration sample data set by the plurality of predicted semantic key points and the corresponding true value labels to obtain the preset non-conformal score includes:
[0021] For each predicted semantic key point in each of the sample target images in the calibration sample data set, a non-conformal degree value between the predicted pixel coordinates and the true value label is calculated based on a non-conformal function;
[0022] The non-conformal degree values of each of the predicted semantic key points in each of the sample target images are arranged in descending order to obtain a corresponding non-conformal score sequence;
[0023] According to the preset fault tolerance rate and the non-conformal score sequence, a non-conformal quantile corresponding to each of the predicted semantic key points is obtained.
[0024] In one embodiment, the generating the semantic key point conformal set according to the preset non-conformal quantile and the conformal prediction for each of the semantic key points comprises:
[0025] According to the form of the non-conformal score, a conformal prediction range corresponding to each of the semantic key points is generated.
[0026] According to the conformal prediction range generated for each of the semantic key points, the semantic key point conformal set is obtained.
[0027] In one embodiment, each of the sample point pixel coordinates in the group of sample point pixel coordinates is sampled in a different conformal prediction range of the semantic key points.
[0028] In one embodiment, the estimating the estimated pose of the non-cooperative target according to the group of sample point pixel coordinates, based on the point correspondence relationship between the sample points and the three-dimensional coordinates of the sample points in the non-cooperative target coordinate system, and the N-point perspective model comprises:
[0029] Converting each of the two-dimensional coordinates of the sample points into a three-dimensional coordinate of the sample point in the non-cooperative target three-dimensional coordinate system;
[0030] According to each of the three-dimensional coordinates, the N-point perspective problem is solved to obtain the estimated pose of the corresponding non-cooperative target.
[0031] In one embodiment, the judging whether the estimated pose meets the preset standard comprises:
[0032] Projecting the estimated pose into the non-cooperative target image to obtain a projected two-dimensional coordinate;
[0033] Determining whether the estimated pose meets the preset standard by the residual value between the projected two-dimensional coordinate and the pixel coordinate predicted.
[0034] In one embodiment, when the number of iterations meets the preset number of times, the target pose set is empty, then a second preset number of iterations is performed, and in each iteration process, the estimated pose obtained is directly retained in the target pose set.
[0035] The application also provides a non-cooperative target pose measurement device with statistical guarantee, the device comprising:
[0036] An image acquisition module for acquiring a non-cooperative target image to be measured for pose;
[0037] a semantic key point prediction module configured to predict semantic key points in the non-cooperative target image by using a trained deep neural network, to obtain a plurality of semantic key points, an index classification item of three-dimensional coordinates of each semantic key point in a three-dimensional coordinate system of the non-cooperative target, and pixel coordinates of the non-cooperative target image on the non-cooperative target image;
[0038] a semantic key point conformal set obtaining module configured to perform conformal prediction on each semantic key point according to a preset non-conformal fraction, to generate a semantic key point conformal set;
[0039] a random sampling module configured to randomly sample a preset number of semantic key points in the semantic key point conformal set, to obtain a group of sample point pixel coordinates on the non-cooperative target image;
[0040] a target pose iterative estimation module configured to estimate an estimated pose of the non-cooperative target based on a point correspondence relationship between the sample points and three-dimensional coordinates of the sample points in a non-cooperative target coordinate system and an N-point perspective model according to the group of sample point pixel coordinates, to determine whether the estimated pose meets a preset standard, to retain the estimated pose in a target pose set if the estimated pose meets the preset standard, to complete an iteration, to randomly sample another group of sample point pixel coordinates in the semantic key point conformal set, to generate a corresponding estimated pose, and to stop the iteration until a preset number of iterations is reached.
[0041] a non-cooperative target pose measurement result obtaining module configured to calculate an average pose based on all estimated poses in the target pose set, to obtain a target pose set and the average pose of the non-cooperative target, and to measure the pose of the non-cooperative target.
[0042] A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:
[0043] obtaining a non-cooperative target image to be measured for a pose;
[0044] predicting semantic key points in the non-cooperative target image by using a trained deep neural network, to obtain a plurality of semantic key points, an index classification item of three-dimensional coordinates of each semantic key point in a three-dimensional coordinate system of the non-cooperative target, and pixel coordinates of the non-cooperative target image on the non-cooperative target image;
[0045] performing conformal prediction on each semantic key point according to a preset non-conformal fraction, to generate a semantic key point conformal set;
[0046] randomly sampling a preset number of semantic key points in the semantic key point conformal set, to obtain a group of sample point pixel coordinates on the non-cooperative target image.
[0047] According to the one set of sampling point pixel coordinates, based on the point correspondence relationship of the sampling points and their corresponding three-dimensional coordinates under the non-cooperative target body coordinate system, and the N-point perspective model, an estimated pose of the non-cooperative target is estimated, and it is judged whether the estimated pose meets a preset standard. If it meets, the estimated pose is retained in the target pose set, one iteration calculation is completed, another set of sampling point pixel coordinates is obtained by random sampling in the semantic key point conformal set, and the corresponding estimated pose is generated, until the iteration number meets the preset number, then the iteration calculation is stopped.
[0048] According to all estimated poses in the target pose set, an average pose is calculated, and a target pose set and an average pose of the non-cooperative target are obtained, so as to realize non-cooperative target pose measurement.
[0049] A computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the following steps:
[0050] An image of a non-cooperative target to be measured is obtained.
[0051] A trained deep neural network is used to predict semantic key points in the non-cooperative target image, to obtain a plurality of semantic key points, and an index classification item of three-dimensional coordinates of each semantic key point under a non-cooperative target three-dimensional coordinate system and a pixel coordinate on the non-cooperative target image.
[0052] According to a preset non-conformal sub-number, a conformal prediction is performed for each semantic key point to generate a semantic key point conformal set.
[0053] A preset number of random samplings are performed in the semantic key point conformal set to obtain a set of sampling point pixel coordinates on the non-cooperative target image.
[0054] According to the one set of sampling point pixel coordinates, based on the point correspondence relationship of the sampling points and their corresponding three-dimensional coordinates under the non-cooperative target body coordinate system, and the N-point perspective model, an estimated pose of the non-cooperative target is estimated, and it is judged whether the estimated pose meets a preset standard. If it meets, the estimated pose is retained in the target pose set, one iteration calculation is completed, another set of sampling point pixel coordinates is obtained by random sampling in the semantic key point conformal set, and the corresponding estimated pose is generated, until the iteration number meets the preset number, then the iteration calculation is stopped.
[0055] According to all estimated poses in the target pose set, an average pose is calculated, and a target pose set and an average pose of the non-cooperative target are obtained, so as to realize non-cooperative target pose measurement.
[0056] The above statistically guaranteed non-cooperative target pose measurement method and device, by using the trained deep neural network to predict the non-cooperative target image, obtains the index classification items of the three-dimensional coordinates of the multiple semantic key points in the three-dimensional coordinate system of the non-cooperative target and the pixel coordinates on the non-cooperative target image, according to the preset non-conformal distribution, generates a semantic key point conformal set for each semantic key point through conformal prediction, obtains a group of sampling point pixel coordinates through random sampling in the semantic key point conformal set, obtains the estimated pose of the non-cooperative target according to the group of sampling point pixel coordinates, and judges the estimated pose that meets the preset standard to be reserved in the target pose set. After multiple iterations, the target pose set and the average pose of the non-cooperative target are obtained. The method can accurately measure the pose of the non-cooperative target in the case of visual blur caused by problems such as occlusion, illumination change and image quality degradation. BRIEF DESCRIPTION OF DRAWINGS
[0057] Figure 1 A flowchart of a statistically guaranteed non-cooperative target pose measurement method in an embodiment;
[0058] Figure 2 A flowchart of an embodiment based on iterative sampling output of a statistically guaranteed target pose set and average pose;
[0059] Figure 3 A structural block diagram of a statistically guaranteed non-cooperative target pose measurement device in an embodiment;
[0060] Figure 4 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION
[0061] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.
[0062] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. Figure 1 As shown in the embodiment, a statistically guaranteed non-cooperative target pose measurement method is provided, including the following steps:
[0063] Step S100, obtaining a non-cooperative target image to be measured.
[0064] Step S110, the semantic key points in the non-cooperative target image are predicted by using the trained deep neural network to obtain a plurality of semantic key points, an index classification item of three-dimensional coordinates of each semantic key point in a three-dimensional coordinate system of the non-cooperative target, and pixel coordinates of the non-cooperative target image.
[0065] Step S120, according to a preset non-conformal distribution, a conformal prediction is performed on each semantic key point to generate a conformal set of semantic key points.
[0066] Step S130, a preset number of random sampling is performed in the conformal set of semantic key points to obtain a group of sample point pixel coordinates on the non-cooperative target image.
[0067] Step S140, according to the group of sample point pixel coordinates, an estimated pose of the non-cooperative target is estimated based on a point correspondence relationship between the sample points and the three-dimensional coordinates of the sample points in the non-cooperative target coordinate system, and an N-point perspective model, and it is judged whether the estimated pose meets a preset standard, if it meets, the estimated pose is reserved to a target pose set, one iteration calculation is completed, another group of sample point pixel coordinates are obtained by random sampling in the conformal set of semantic key points, and a corresponding estimated pose is generated, until the number of iterations meets a preset number, then the iteration calculation is stopped.
[0068] Step S150, an average pose is calculated according to all estimated poses in the target pose set to obtain a target pose set and an average pose of the non-cooperative target, so as to realize the non-cooperative target pose measurement.
[0069] In step S100, the non-cooperative target image to be measured can be detected and acquired by a service spacecraft on a non-cooperative target spacecraft in formation flight, failed satellite and debris clearance and other tasks. The non-cooperative target image is a visible light image.
[0070] In step S110, the semantic key points in the non-cooperative target image are extracted (predicted) by using the trained deep neural network to obtain three-dimensional coordinates of each semantic key point in the non-cooperative target coordinate and pixel coordinates of the two-dimensional image on the non-cooperative target image.
[0071] In this embodiment, the three-dimensional coordinates of the semantic key points in the non-cooperative target coordinate are actually represented by the index classification item of the correspondence relationship between each semantic key point and the semantic key point in the non-cooperative target coordinate.
[0072] In the embodiment, when training the deep neural network: a sample data set is obtained, the sample data set including a plurality of sample target images and a true value label corresponding to each sample image, the sample data set is divided into a training sample data set and a calibration sample data set, the deep neural network is trained by using the training sample data set, a trained neural network capable of predicting target semantic key points in the sample target image is obtained, and then the trained neural network is used to predict semantic key points in the sample target image in the calibration sample data set, a plurality of predicted semantic key points are obtained, the calibration sample data set is calibrated by using the plurality of predicted semantic key points and the corresponding true value label, and a preset non-conformal score is obtained.
[0073] In the embodiment, the sample data set is divided into a training sample data set for training the deep neural network and a calibration sample data set for obtaining the non-conformal score
[0074] In the embodiment, the targets in the sample target images in the sample data set are optical images of the same type of target. The target is a type of non-cooperative target in actual application, for example, various types of spacecrafts, etc.
[0075] Specifically, the sample data set includes the true value label of each sample target image in addition to the plurality of sample target images. The true value label of each sample target image is actually a true value label set, which is composed of a total of P semantic key point elements. The i-th semantic key point element is composed of an index classification item describing the corresponding relationship between the i-th semantic key point element and the semantic key point on the non-cooperative target coordinate system, and an X and Y axis image coordinate position item describing the coordinates of the i-th semantic key point element on the X and Y axes.
[0076] In the embodiment, the deep neural network includes a feature extraction network, an index classification item prediction head, and an image coordinate position prediction head. The feature extraction network is used to extract a feature map of the non-cooperative target image. The index classification item prediction head and the image coordinate position prediction head respectively predict the index classification item of the three-dimensional coordinates of the semantic key point in the three-dimensional coordinate system of the non-cooperative target and the pixel coordinates on the image of the non-cooperative target according to the feature map.
[0077] Specifically, the feature extraction network can use any architecture such as Transformer, CNN, or VMamba to extract the feature map of the input image. The index classification item prediction head and the image coordinate position item prediction head accept the features output by the feature extraction network to respectively predict the index classification item of the semantic key point and the XY axis image coordinate position item.
[0078] Further, the index classification item can adopt forms including but not limited to one-hot encoding or Gaussian distribution, and the XY axis image coordinate position item can adopt forms including but not limited to one-dimensional heat map, two-dimensional heat map, image pixel coordinates and vector. The above forms can be converted into image pixel coordinate form, so in the method, only the image pixel coordinate form is taken as an example.
[0079] In the embodiment, when the deep neural network is trained, i.e., optimized, any optimization method such as a stochastic gradient descent method can be used. Any feasible classification loss including but not limited to cross-entropy loss and any feasible image position loss including but not limited to KL loss are used to supervise the training of the deep neural network model until it converges.
[0080] Specifically, when the training sample data set is used to train the deep neural network, the sample target images in the training sample data set are sequentially input into the deep neural network, a prediction value set matched one by one with the true value labels is obtained, a loss function is calculated according to the prediction value and the true value label, and the adjustable parameters in the deep neural network are updated and optimized according to the loss function until it converges, thereby obtaining the trained deep neural network.
[0081] In the embodiment, after obtaining the trained deep neural network, the calibration sample data set is used to obtain the non-conformity score. By inputting the sample target images in the calibration sample data set into the trained deep neural network, a plurality of predicted semantic key points are obtained, the calibration sample data set is calibrated by the plurality of predicted semantic key points and the corresponding true value labels, and the preset non-conformity score is obtained.
[0082] Specifically, for each predicted semantic key point in each sample target image in the calibration sample data set , a non-conformity degree value between the predicted pixel coordinates and the true value label is calculated based on the non-conformity function, the non-conformity degree values of the predicted semantic key points in each sample target image are arranged in descending order to obtain a corresponding non-conformity score sequence, and the non-conformity fractile of each predicted semantic key point is obtained by calculating the preset fault tolerance and the non-conformity score sequence.
[0083] Further, since the forms of the semantic key point prediction values are different, the form of the non-conformity function is not unique. In the method, the non-conformity function is represented as:
[0084] α i = φ (y i , f (x i )) = p i || y i - q i|| (1)
[0085] In formula (1), x represents an input image, i.e., a sample target image, f represents a deep neural network model; p i represents a probability that the prediction value corresponds to the i-th semantic key point, ||y i -q i represents a deep neural network model i-th semantic key point, image pixel position prediction value q i and the true value y i . φ represents a non-conformal function, which measures the degree of non-conformality between the image pixel position of the semantic key point predicted based on the deep neural network and the true value.
[0086] Further, the non-conformal score of the i-th semantic key point in the k-th image in the entire calibration data set is calculated as N cal , which represents the total number of samples in the calibration data set. Then, the non-conformal scores are arranged in descending order as where π(n) ∈ {1, 2,..., N cal} represents an index sequence, and P descendingly arranged non-conformal score sequences are output after conformal calibration, i.e., the non-conformal degrees of each semantic key point in the k sample target images are arranged in descending order to obtain the corresponding non-conformal score sequences.
[0087] Further, the conformal fractiles of each semantic key point are calculated according to the respective non-conformal score sequences to assist subsequent induction of conformal prediction sets of semantic key points, which will be described below.
[0088] In this embodiment, after the training of the deep neural network and the calculation of the conformal fractiles of each semantic key point, i.e., when processing the non-cooperative target image, the conformal prediction of each semantic key point is induced by using the conformal fractiles of each semantic key point after extracting the semantic key points in the non-cooperative target image by using the trained deep neural network, i.e., the process in step S120.
[0089] It should be noted that, in the following, represents a non-cooperative target image to be measured, which actually represents a test sample image to be tested by the method, but since the content is consistent in the test stage and the actual application stage, the non-cooperative target image is represented by Since the test sample image includes multiple images during testing, the subscript k represents the k-th test sample image.
[0090] In the embodiment, according to the preset non-conformal score, the conformal prediction is performed for each semantic key point to generate the semantic key point conformal set, which includes: generating the conformal prediction range corresponding to each semantic key point according to the form of the non-conformal score, and obtaining the semantic key point conformal set according to the conformal prediction range generated for each semantic key point.
[0091] In one embodiment, when the form of the non-conformal score is shown in formula (1), the semantic key point induction co-occurrence prediction set is represented as:
[0092]
[0093] In formula (2), X and Y represent the image position coordinate labels of all semantic key points. The induction conformal prediction set includes the label set of all semantic key points whose non-conformal scores are not greater than the quantile corresponding to the semantic key point. The non-conformal score of all semantic key points in the formula (1) is not greater than the quantile corresponding to the semantic key point . Substituting the definition of the non-conformal score in the method, the semantic key point induction conformal prediction set is represented as:
[0094]
[0095] As can be seen from formula (3), according to the preset non-conformal score, the conformal prediction is performed for each semantic key point to generate the semantic key point conformal set, which is actually to generate the circular semantic key point conformal prediction range with each semantic key point as the center and the non-conformal score corresponding to each semantic key point as the radius, and to obtain the semantic key point conformal set according to the circular semantic key point conformal prediction range generated for each semantic key point.
[0096] In one embodiment, the form of the non-conformal score can also adopt other forms, such as:
[0097]
[0098] In formula (4), the semantic key point adopts the representation form of the heat map, represents the inverse of the covariance matrix of the heat map, represents the first J most likely detection average positions of the heat map. Generating the semantic key point conformal set is actually to generate the elliptical semantic key point conformal prediction range with each semantic key point as the center and the covariance matrix corresponding to each semantic key point as the radius, and to obtain the semantic key point conformal set according to the elliptical semantic key point conformal prediction range generated for each semantic key point.
[0099] In step S130, sampling is performed within the semantic key point induction conformal prediction set range, and an XY axis image position coordinate label of the sampling point is output where P n(i) represents the coordinates of the non-cooperative target body coordinate system corresponding to the i-th sampling point, and subscript n(i) represents the index of random sampling from P semantic key points. The total number of n(i) is determined by the solving algorithm of the final selected N-point perspective problem, and at least three sampling points are required. Preferably, a set of sampling points includes four sampling points.
[0100] Further, each sampling point pixel coordinate in the set of sampling point pixel coordinates is sampled in a different circular semantic key point conformal prediction range.
[0101] In step S140, based on the set of sampling point pixel coordinates, the point correspondence relationship between the sampling points and their corresponding three-dimensional coordinates in the non-cooperative target body coordinate system, and the N-point perspective model, the estimated pose of the non-cooperative target is estimated, including: converting each sampling point two-dimensional coordinate to a sampling point three-dimensional coordinate in the non-cooperative target three-dimensional coordinate system, solving the N-point perspective problem according to each three-dimensional coordinate, and obtaining the corresponding estimated pose of the non-cooperative target.
[0102] Specifically, based on the correspondence relationship between the sampling points and their positions in the non-cooperative target body coordinate system, an N-point perspective model is constructed, and the position and attitude of the non-cooperative target in the camera coordinate system are obtained by solving the N-point perspective model, wherein the N-point perspective model is represented as:
[0103]
[0104] In formula (5), and respectively represent the best estimated value of the rotation matrix and the translation vector of the non-cooperative target in the camera coordinate system, and the estimated pose of the non-cooperative target is represented by the two. ρ(·) is a robust estimation function, e i is the reprojection residual, which is represented as:
[0105] e i = (λ i p i -K[R|t]P n(i) ) T (λ i p i -K[R|t]P n(i) ) (6)
[0106] In formula (6), K represents an internal parameter matrix of the camera, [R|t] represents the pose of the non-cooperative target, wherein R represents a rotation matrix of the non-cooperative target in the camera coordinate system, t represents a translation vector of the non-cooperative target in the camera coordinate system, and λ i represents the photographic depth of the i th sampling point.
[0107] After obtaining the estimated pose corresponding to the group sampling point pixel coordinate, it is necessary to judge whether the estimated pose meets the preset standard, that is, whether the estimated pose has a statistically guaranteed target pose set.
[0108] In the embodiment, judging whether the estimated pose meets the preset standard comprises: projecting the estimated pose into the non-cooperative target image to obtain a projected two-dimensional coordinate, and judging whether the estimated pose meets the preset standard through a residual value between the projected two-dimensional coordinate and the predicted pixel coordinate.
[0109] Specifically, the pose true value belongs to the target pose set S ε ={[R|t]∈SE(3)} only when the semantic key point image coordinate position true value belongs to the semantic key point inductive conformal prediction set.
[0110] Whether the pose estimated based on the sampling point and is in the set S ε is judged by the following formula:
[0111]
[0112] In formula (7), P represents the predicted i th sampling point in the body coordinate system, n(i) under the pose and is projected into the image pixel coordinate system under the following formula to obtain the projected two-dimensional coordinate:
[0113]
[0114] In formula (8), Π(·) represents a camera projection function, and after n times of sampling iteration, m The average rotation matrix is solved as follows:
[0115]
[0116] In formula (9), d represents a distance function between two rotation matrices, and the F norm is adopted, which is defined as follows:
[0117]
[0118] In the embodiment, when the number of iterations meets the preset number, the target pose set is empty, then the second preset number of iteration calculations are performed, in each iteration calculation process, the obtained estimated pose is directly reserved to the target pose set.
[0119] Specifically, after n iterations, there is still no predicted pose that meets the target pose set judgment condition with statistical guarantee. Then, n / 20 iterations are re-performed, each iteration samples each semantic key point, a total of P sampling points are output, and the pose set with statistical guarantee is calculated according to the same process as described above. And the average pose
[0120] Specifically, the average displacement is obtained by directly averaging n translation vectors:
[0121]
[0122] In fact, step S130 and step S140 are an iteration process, and the iteration process can be as shown in Figure 2 .
[0123] In step S150, the pose set with statistical guarantee and the average pose are output, wherein the pose set has a probability of 1-ε to cover the true value of the pose.
[0124] In the above statistical guarantee non-cooperative target pose measurement method, the positions of the semantic key points in the image coordinate system in the non-cooperative target are predicted by training a deep neural network, and through steps such as conformal calibration, inductive conformal prediction and random sampling, a non-cooperative target pose set with statistical guarantee is output, which can express visual ambiguity caused by problems such as occlusion, illumination change and image quality degradation in actual tasks, and is more suitable for expressing the uncertainty in the output pose. Users can customize the fault tolerance rate ε, and the output non-cooperative target pose set with statistical guarantee has a probability of 1-fault tolerance rate 1-ε to cover the true value of the pose.
[0125] It should be understood that, although each step in the flowchart of Figure 1 is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified in this article, the execution of these steps has no strict order limitation, and these steps can be executed in other orders. Moreover, Figure 1At least one of the steps in the method can comprise a plurality of sub-steps or stages, which are not necessarily performed at the same time, but can be performed at different times, and the order of the sub-steps or stages is not necessarily sequential, but can be performed in rotation or alternation with other steps or sub-steps or stages of other steps.
[0126] In one embodiment, as shown in Figure 3 A device for statistically guaranteed non-cooperative target pose measurement is provided, comprising: an image acquisition module 200, a semantic key point prediction module 210, a semantic key point conformal set obtaining module 220, a random sampling module 230, a target pose iterative estimation module 240, and a non-cooperative target pose measurement result obtaining module 250, wherein:
[0127] The image acquisition module 200 is configured to acquire an image of a non-cooperative target to be measured for pose.
[0128] The semantic key point prediction module 210 is configured to use a trained deep neural network to predict semantic key points in the non-cooperative target image, to obtain a plurality of semantic key points, and an index classification item of three-dimensional coordinates of each semantic key point in a non-cooperative target three-dimensional coordinate system and a pixel coordinate on the non-cooperative target image.
[0129] The semantic key point conformal set obtaining module 220 is configured to perform conformal prediction for each semantic key point according to a preset non-conformal index, to generate a semantic key point conformal set.
[0130] The random sampling module 230 is configured to randomly sample a preset number of points in the semantic key point conformal set, to obtain a group of sample point pixel coordinates on the non-cooperative target image.
[0131] The target pose iterative estimation module 240 is configured to estimate an estimated pose of the non-cooperative target based on the point correspondence relationship between the sample points and their corresponding three-dimensional coordinates in the non-cooperative target coordinate system, and an N-point perspective model according to the group of sample point pixel coordinates, to determine whether the estimated pose meets a preset standard, and if so, to retain the estimated pose in a target pose set, complete an iteration, and randomly sample another group of sample point pixel coordinates in the semantic key point conformal set, to generate a corresponding estimated pose, until the number of iterations meets a preset number, and then stop the iteration.
[0132] The non-cooperative target pose measurement result obtaining module 250 is configured to calculate an average pose based on all estimated poses in the target pose set, to obtain a target pose set and an average pose of the non-cooperative target, to realize non-cooperative target pose measurement.
[0133] The specific limitations of the non-cooperative target pose measurement device with statistical guarantee can refer to the limitations of the non-cooperative target pose measurement method with statistical guarantee described above, which will not be repeated here. Each module in the above non-cooperative target pose measurement device with statistical guarantee can be realized by software, hardware and their combination in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor calls to execute the operations corresponding to the above modules.
[0134] In one embodiment, a computer device is provided, which can be a terminal, and its internal structure diagram can be as shown in Figure 4 The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a non-cooperative target pose measurement method with statistical guarantee. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.
[0135] Those skilled in the art can understand that Figure 4 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.
[0136] In one embodiment, a computer device is provided, which includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the following steps:
[0137] Obtaining a non-cooperative target image to be measured for pose measurement;
[0138] Using the trained deep neural network to predict semantic key points in the non-cooperative target image to obtain a plurality of semantic key points, an index classification item of three-dimensional coordinates of each semantic key point in a non-cooperative target three-dimensional coordinate system, and pixel coordinates on the non-cooperative target image.
[0139] According to a preset non-conformal sub-bit number, conformal prediction is performed on each semantic key point to generate a semantic key point conformal set;
[0140] Random sampling is performed in the semantic key point conformal set according to a preset number to obtain a group of sample point pixel coordinates on the non-cooperative target image;
[0141] According to the group of sample point pixel coordinates, an estimated pose of the non-cooperative target is estimated based on a point correspondence relationship between the sample points and three-dimensional coordinates under a non-cooperative target body coordinate system and an N-point perspective model, and it is determined whether the estimated pose meets a preset standard. If yes, the estimated pose is retained in a target pose set, one iteration calculation is completed, another group of sample point pixel coordinates is obtained by random sampling in the semantic key point conformal set, and a corresponding estimated pose is generated. The iteration calculation is stopped until the number of iterations meets a preset number.
[0142] An average pose is calculated according to all estimated poses in the target pose set, and a target pose set and an average pose of the non-cooperative target are obtained to realize non-cooperative target pose measurement.
[0143] In one embodiment, a computer readable storage medium is provided, and a computer program is stored on the computer readable storage medium. The computer program is executed by a processor to implement the following steps:
[0144] An image of a non-cooperative target to be measured is obtained.
[0145] A trained deep neural network is used to predict semantic key points in the non-cooperative target image to obtain a plurality of semantic key points, an index classification item of three-dimensional coordinates of each semantic key point under a non-cooperative target three-dimensional coordinate system, and pixel coordinates on the non-cooperative target image.
[0146] According to a preset non-conformal sub-bit number, conformal prediction is performed on each semantic key point to generate a semantic key point conformal set;
[0147] Random sampling is performed in the semantic key point conformal set according to a preset number to obtain a group of sample point pixel coordinates on the non-cooperative target image;
[0148] According to the set of sampling point pixel coordinates, based on the point correspondence relationship of the sampling points and their corresponding three-dimensional coordinates in the non-cooperative target body coordinate system, and the N-point perspective model, an estimated pose of the non-cooperative target is estimated, and it is judged whether the estimated pose meets a preset standard. If it meets, the estimated pose is retained in the target pose set, one iteration calculation is completed, and another set of sampling point pixel coordinates is obtained by random sampling in the semantic key point conformal set, and a corresponding estimated pose is generated, until the number of iterations meets a preset number, then the iteration calculation is stopped.
[0149] According to all the estimated poses in the target pose set, an average pose is calculated to obtain the target pose set and the average pose of the non-cooperative target, so as to realize the non-cooperative target pose measurement.
[0150] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0151] The technical features of the above embodiments can be combined in any way. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.
[0152] The above embodiments only express several implementation ways of the present application, and the description is more specific and detailed, but it should not be understood as a limitation to the patent scope of the application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.
Claims
1. A statistically guaranteed non-cooperative target pose measurement method, characterized in that, The method comprises: acquiring an image of a non-cooperative target to be measured for pose; using a trained deep neural network to predict semantic key points in the image of the non-cooperative target, to obtain a plurality of semantic key points, and an index classification item of three-dimensional coordinates of each semantic key point in a three-dimensional coordinate system of the non-cooperative target and pixel coordinates on the image of the non-cooperative target; performing conformal prediction on each semantic key point according to a preset non-conformal quantile, to generate a conformal set of semantic key points; performing random sampling in the conformal set of semantic key points according to a preset number, to obtain a set of pixel coordinates of sampling points on the image of the non-cooperative target; estimating an estimated pose of the non-cooperative target based on a point correspondence relationship between the set of pixel coordinates of sampling points and three-dimensional coordinates of the non-cooperative target in a non-cooperative target coordinate system and an N-point perspective model, and determining whether the estimated pose meets a preset standard, if so, retaining the estimated pose to a target pose set, completing an iteration, and then performing random sampling in the conformal set of semantic key points to obtain another set of pixel coordinates of sampling points and generating a corresponding estimated pose, until the number of iterations meets a preset number, then stopping the iteration; calculating an average pose based on all estimated poses in the target pose set, to obtain a target pose set of the non-cooperative target and the average pose, to realize non-cooperative target pose measurement; when training the deep neural network: acquiring a sample data set, which comprises a plurality of sample target images and true value labels corresponding to each sample image; dividing the sample data set into a training sample data set and a calibration sample data set; training the deep neural network using the training sample data set, to obtain the trained neural network having the ability to predict target semantic key points in a sample target image; predicting semantic key points in the sample target images in the calibration sample data set using the trained neural network, to obtain a plurality of predicted semantic key points, and calibrating the calibration sample data set using the plurality of predicted semantic key points and corresponding true value labels, to obtain a preset non-conformal quantile; the calibration of the calibration sample data set using the plurality of predicted semantic key points and corresponding true value labels to obtain the preset non-conformal quantile comprises: calculating, based on a non-conformal function, a non-conformal degree value between the predicted pixel coordinates and the true value labels for each predicted semantic key point in each sample target image in the calibration sample data set; wherein the non-conformal function is represented as: wherein, denotes an input image, i.e. a sample target image, denotes a deep neural network model, denotes a probability that the prediction value corresponds to the semantically key point, denotes a semantically key point of the deep neural network model, denotes a two-norm between the prediction value and the ground truth for the image pixel position, denotes a non-conformal function; The entire calibration data set is calculated Zhang image , the non-conformal fraction of the corresponding semantic key point is ; The non-conformity degree values of each of the predicted semantic key points in each of the sample target images are arranged in descending order to obtain a corresponding non-conformity score sequence; wherein the non-conformity score sequence is represented as , ; calculating, according to a preset fault tolerance rate and a non-conformal quantile sequence, a non-conformal quantile corresponding to each predicted semantic key point; the conformal prediction of each semantic key point according to the preset non-conformal quantile to generate the conformal set of semantic key points comprises: generating a conformal prediction range corresponding to each semantic key point according to the form of the non-conformal quantile; A semantic key point conforming set is obtained according to the conformal prediction ranges of the semantic key points, wherein the semantic key point conforming set is represented as: wherein, denotes the XY axis image position coordinate label of all semantic keypoints, to denote the non-cooperative target image, denotes the quantile of the th semantic keypoint in.
2. The method of pose measurement of a non-cooperative target with statistical guarantees of claim 1, wherein, The deep neural network comprises a feature extraction network, an index classification item prediction head, and an image coordinate position prediction head; The feature extraction network is configured to extract a feature map of the non-cooperative target image. The index classification item prediction head and the image coordinate position prediction head are configured to respectively predict an index classification item of a three-dimensional coordinate of a semantic key point in a non-cooperative target three-dimensional coordinate system and a pixel coordinate on a non-cooperative target image according to the feature map.
3. The method of pose measurement of a non-cooperative target with statistical guarantees of claim 2, wherein, Each of the pixel coordinates of the set of sampling points is sampled in a different conformal prediction range of a semantic key point.
4. The method of pose measurement of a non-cooperative target with statistical guarantees of claim 3, wherein, The estimated pose of the non-cooperative target is obtained based on the point correspondence between the sampling points and their corresponding three-dimensional coordinates in the non-cooperative target three-dimensional coordinate system and the N-point perspective model according to the set of sampling point pixel coordinates, and comprises: Converting the two-dimensional coordinates of the sampling points into sampling point three-dimensional coordinates in the non-cooperative target three-dimensional coordinate system; Solving an N-point perspective problem according to the three-dimensional coordinates to obtain an estimated pose of the corresponding non-cooperative target.
5. The method of pose measurement of a non-cooperative target with statistical guarantees of claim 4, wherein, Determining whether the estimated pose meets a preset standard comprises: Projecting the estimated pose into the non-cooperative target image to obtain projected two-dimensional coordinates; Determining whether the estimated pose meets the preset standard by a residual value between the projected two-dimensional coordinates and the predicted pixel coordinates.
6. The method of pose measurement of non-cooperative targets with statistical guarantees of claim 1, wherein, When the number of iterations meets a preset number and the target pose set is empty, a second preset number of iterations is performed, and in each iteration process, the obtained estimated pose is directly retained in the target pose set.
7. An apparatus for statistically guaranteed non-cooperative target pose measurement, characterized in that A device for implementing the method for measuring a pose of a non-cooperative target with a statistical guarantee according to any one of claims 1 to 6, the device comprising: An image acquisition module configured to acquire a non-cooperative target image to be measured for a pose; A semantic key point prediction module configured to predict semantic key points in the non-cooperative target image by using a trained deep neural network to obtain a plurality of semantic key points and an index classification item of a three-dimensional coordinate of each of the semantic key points in a non-cooperative target three-dimensional coordinate system and a pixel coordinate on a non-cooperative target image; A semantic key point conforming set obtaining module configured to perform conformal prediction on each of the semantic key points according to a preset non-conforming index number to obtain a semantic key point conforming set; A random sampling module configured to randomly sample a preset number of sampling points in the semantic key point conforming set to obtain a set of sampling point pixel coordinates on the non-cooperative target image; and An estimated pose obtaining module configured to obtain an estimated pose of the non-cooperative target according to the set of sampling point pixel coordinates based on a point correspondence between the sampling points and their corresponding three-dimensional coordinates in the non-cooperative target three-dimensional coordinate system and an N-point perspective model. The target pose iterative estimation module is configured to estimate an estimated pose of the non-cooperative target based on the point correspondence relationship between the sampling points and the three-dimensional coordinates of the non-cooperative target body coordinate system corresponding to the sampling points and the N-point perspective model according to the set of sampling point pixel coordinates, and determine whether the estimated pose meets a preset standard. If yes, the estimated pose is retained in the target pose set, one iteration is completed, another set of sampling point pixel coordinates is randomly sampled in the semantic key point conformal set, and a corresponding estimated pose is generated. The iteration is stopped until the number of iterations meets a preset number. The non-cooperative target pose measurement result obtaining module is configured to calculate an average pose based on all the estimated poses in the target pose set, obtain the target pose set and the average pose of the non-cooperative target, and realize the non-cooperative target pose measurement.