Multi-view engine jet pipe swing angle measuring method based on non-cooperative measurement

Abstract

The invention relates to a multi-view engine jet pipe swing angle measuring method based on non-cooperative measurement. The invention relates to the technical field of measurement of the swing angle of an engine jet pipe, and aims to realize accurate measurement of the swing angle of the jet pipe, extract feature points of the jet pipe, and select a SuperPoint network to extract image features; after the feature points are extracted, matching of the feature points is completed by using a SuperGlue network; estimating a camera pose, wherein the camera pose needs to be estimated after the feature points of the spray pipe are extracted and matched; and three-dimensional feature point reconstruction: calculating a camera projection matrix corresponding to the added image through camera pose estimation, and reconstructing non-reconstructed feature points in the newly added image. The method does not depend on cooperation information provided by the spray pipe, has high measurement precision, and meets the requirement for non-cooperation measurement of the swing angle of the spray pipe.

Description

Technical Field

[0001] The invention relates to the technical field of measuring the swing angle of an engine nozzle, and is a multi-view engine nozzle swing angle measuring method based on non-cooperative measurement. Background Art

[0002] In order to achieve accurate measurement of nozzle swing angle, contact measurement method is one of the commonly used measurement methods. The common contact measurement method is to calculate the nozzle swing angle by integrating sensor measurement data. Domestic and foreign researchers have successively proposed nozzle swing angle measurement schemes based on infrared laser diodes and displacement detectors, nozzle swing angle measurement schemes based on hemispherical contact displacement sensors, and nozzle swing angle measurement schemes based on linear displacement sensors. And based on displacement meters, rocker bars and nozzle actuators, these studies have reduced the influence of nozzle complex structures on swing angle measurement results to varying degrees, but there are generally problems such as low measurement efficiency, poor consistency of repeated measurement accuracy, and complex sensor assembly. With the development of measurement technology, visual measurement technology has been widely used in the measurement process of various types of equipment due to its high measurement accuracy and fast measurement speed. Some scholars have conducted research on the visual measurement method of nozzle swing angle. The main application idea of ​​the visual measurement method is to infer the swing angle change by detecting the movement change of the characteristic marking points on the nozzle. This scheme has been verified to be feasible in many experiments, but the visual measurement method based on cooperative marking points all adopts the method of pasting cooperative marking points. It is necessary to paste cooperative target balls or manually designed marking points on the nozzle surface, and then calculate the nozzle swing angle by tracking and locating the displacement change of the cooperative marking points. Although this measurement method improves the measurement efficiency, there are problems such as complex installation and maintenance of marking points, which cannot meet the measurement requirements under different experimental conditions. Therefore, it is necessary to propose a new non-cooperative measurement scheme for nozzle swing angle measurement to further improve the measurement accuracy of the nozzle swing angle and meet the design and development requirements of high-power engines. Summary of the invention

[0003] In view of the shortcomings of the prior art, the present invention provides a multi-view engine nozzle swing angle measurement method based on non-cooperative measurement. The present invention provides the following technical solutions:

[0004] A method for measuring a multi-view engine nozzle swing angle based on non-cooperative measurement, the method comprising the following steps:

[0005] Step 1: Extract nozzle feature points and use SuperPoint network to extract image features;

[0006] Step 2: After feature point extraction, use the SuperGlue network to complete feature point matching;

[0007] Step 3: Camera pose estimation: after extracting and matching the nozzle feature points, the camera pose needs to be estimated;

[0008] Step 4: Reconstruct three-dimensional feature points. Calculate the camera projection matrix corresponding to the added image through camera pose estimation and reconstruct the unreconstructed feature points in the newly added image.

[0009] Preferably, the step 1 is specifically:

[0010] The SuperPoint feature extraction network can simultaneously extract feature points in an image and generate feature descriptors. The network model includes an encoder, a feature point decoder, and a descriptor decoder.

[0011] The SuperPoint network is trained using the labeled data set, and its loss function is composed of the feature point loss L p and descriptor loss L d It consists of two parts:

[0012] loss = L p (χ,Y)+L p (χ′,Y′)+λL d (D,D′,S)(1)

[0013] Among them, λ is the balance coefficient of feature point loss and descriptor loss; Y is the feature point label; χ′, D′, Y′ are the results obtained in the image after homography transformation; S represents the feature point matching set in the image;

[0014]

[0015]

[0016] Among them, l p ,l d For each pixel, the feature point loss and descriptor loss are hw ∈χ,y hw ∈Y,d hw ∈D,d h ″ w′ ∈D′,s hwh′w′ ∈S.

[0017] Preferably, the step 2 is specifically:

[0018] Step 2.1: The SuperGlue network obtains the matching relationship of feature points based on the feature points and descriptors in the two images, including the attention GNN and the optimal matching layer module. For the feature point positions output by the SuperPoint network and feature descriptors Will In the input key point encoder, the attention GNN uses a multi-layer perceptron to increase the dimension of the position information of the feature point and couples it with the descriptor to obtain the initial information of the feature point:

[0019]

[0020] Step 2.2: When setting is the feature point information of the feature point in the l-th layer network, and the enhanced feature point information is expressed as:

[0021]

[0022] in, Indicates serial operation, m ε→i Represents the aggregation information of feature points, ε∈{ε self ,ε cross}, ε self and ε cross The autocorrelation and cross-correlation attention mechanisms are used to aggregate information for feature points. When l is an odd number, ε = ε self ; When l is an even number, ε=ε cross ;

[0023] Step 2.3: The enhanced feature point information is transformed linearly to obtain the matching descriptor of the feature point:

[0024]

[0025] Among them, W,b are the weight and bias parameters in the linear transformation;

[0026] In the optimal matching layer, the allocation matrix P is constructed to satisfy:

[0027] P1 N ≤1 M

[0028] P T 1 M ≤1 N (6)

[0029] Match the descriptors for the feature points in images A and B respectively, and calculate the score matrix S based on the vector inner product:

[0030]

[0031] By maximizing The optimal allocation matrix P is obtained, and the Sinkhorn algorithm is used in SuperGlue to solve the optimal transportation problem.

[0032] Preferably, the step 3 comprises:

[0033] Step 3.1: Assume that the coordinates of n feature points in the world coordinate system and the camera coordinate system are P i =(X i ,Y i ,Z i ) T and Q i =(X′ i ,Y′ i ,Z′ i ) T , from the motion properties of rigid bodies, we can know that:

[0034] Q i =RP i +t (8)

[0035] in, and t=(t x t y t z ) T Represents the transformation relationship between the camera coordinate system and the world coordinate system. The camera pose estimation problem is to solve R and t;

[0036] Camera optical center O c , feature point Q i and the projection point p on the normalized image plane i Collinearity, according to the collinearity relationship, we can get:

[0037]

[0038]

[0039]

[0040] Step 3.2: Establish the collinearity error objective function of feature point image space:

[0041]

[0042] In the formula, are the image coordinates affected by the noise.

[0043] The collinear relationship is expressed as O c Q i In O c p i The projection vector in the direction is equal to O c Q i :

[0044] RP i +t=V i (RP i +t) (11)

[0045] in, is the ideal line-of-sight projection matrix;

[0046] Step 3.3: Affected by factors such as image noise, O c , With Q i Not collinear, O c Q i exist The projection vector O in the direction c Q i⊥ With O c Q i There is a deviation between them, which is the feature point projection vector error;

[0047] The feature point projection vector error is expressed as:

[0048]

[0049] in, is the actual line of sight projection matrix.

[0050] Step 3.4: Establish the feature point projection vector error objective function:

[0051]

[0052] In order to reduce the parameters to be solved, the CGR parameterized rotation matrix is ​​used:

[0053]

[0054] Among them, s1, s2, s3 are three unknown rotation parameters.

[0055] Introducing Kronecker product:

[0056]

[0057] in

[0058]

[0059] Vec(R)=[r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 ] T

[0060] In formula (13), all feature point information is included in In the equation, when R is given, t can be expressed as:

[0061]

[0062] in

[0063]

[0064] Substitute equation (15) and equation (16) into equation (13):

[0065]

[0066] Substituting the rotation parameterized R into equation (17), we can obtain a polynomial containing three unknown rotation parameters. In order to obtain the solution of the rotation parameters, the partial derivative of the objective function with respect to s1, s2, and s3 is calculated:

[0067]

[0068] Thus, three polynomial equations consisting of parameter variables s1, s2, and s3 are obtained. Substituting the obtained rotation parameters into equations (14) and (16) to obtain the camera pose.

[0069] Preferably, the step 4 is specifically as follows: after extracting and matching feature points of the nozzle area in the captured image, the trajectories of n feature points in m images are obtained. Based on the pinhole camera model, the feature points in each image can be expressed as:

[0070] λ ij u ij =H i X j (19)

[0071] Among them, H i is the projection matrix of the i-th camera; X j is the world coordinate of the feature point; u ij is the image coordinate of the jth feature point on the i-th camera; ij is a scalar;

[0072] The three-dimensional coordinates of the feature points can be obtained according to the triangulation method. Assuming that the camera intrinsic parameter matrix K remains unchanged, the world coordinate system coincides with the camera coordinate system in the first frame image, then the camera projection matrices corresponding to the two images can be expressed as:

[0073] H1=K[R1t1]=K[I0]

[0074] H2=K[R2t2]=K[Rt](20)

[0075] Among them, R and t are the external parameters of the camera corresponding to the second frame image.

[0076] Convert the fundamental matrix to the essential matrix:

[0077] E=K T FK(21)

[0078] Perform singular value decomposition on the essential matrix:

[0079] E=UDV T (twenty two)

[0080] In the singular value decomposition, for any E, two rotation matrices and translation vectors can be decomposed:

[0081]

[0082]

[0083] in, Represents the rotation matrix obtained by rotating 90° around the Z axis.

[0084] The reconstructed feature points are located in front of the camera to eliminate the ambiguous values, so as to obtain the correct R and t; given the camera projection matrix, X can be calculated based on the image coordinates of the feature points in two or more views. j The coordinates of .

[0085] Rewrite equation (19) as:

[0086] [u ij ] × H i X j =AX j =0(24)

[0087] X j The solution is the matrix A T The eigenvector corresponding to the smallest eigenvalue of A.

[0088] Preferably, each time an image is added, the correspondence between the two-dimensional feature points and the reconstructed three-dimensional feature points is obtained based on the matching relationship of the feature points in the image, and the camera projection matrix corresponding to the added image is calculated using the camera pose estimation algorithm and the unreconstructed feature points in the newly added image are reconstructed.

[0089] Preferably, when the nozzle swings within a range in the horizontal and vertical directions, the maximum measurement errors of the nozzle swing angle are 0.12° and 0.13° respectively; in dynamic experiments, most of the swing angle measurement errors are controlled within an error range of ±0.15.

[0090] A multi-view engine nozzle swing angle measurement system based on non-cooperative measurement, the system comprising:

[0091] A feature extraction module, wherein the feature extraction module extracts nozzle feature points and uses a SuperPoint network to extract image features;

[0092] A matching module, which uses a SuperGlue network to complete the matching of feature points after feature point extraction;

[0093] A pose estimation module, which performs camera pose estimation. After extracting and matching nozzle feature points, the camera pose needs to be estimated;

[0094] A three-dimensional reconstruction module is used to reconstruct three-dimensional feature points, calculate the camera projection matrix corresponding to the added image through camera pose estimation, and reconstruct the unreconstructed feature points in the newly added image.

[0095] A computer-readable storage medium stores a computer program, which is executed by a processor to implement a multi-view engine nozzle swing angle measurement method based on non-cooperative measurement.

[0096] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, a multi-view engine nozzle swing angle measurement method based on non-cooperative measurement is implemented.

[0097] The present invention has the following beneficial effects:

[0098] Compared with the prior art, the present invention has the following advantages:

[0099] In the static experiment of the present invention, when the nozzle swings in the horizontal direction and the vertical direction within the range, the maximum measurement errors of the nozzle swing angle are 0.12° and 0.13° respectively; in the dynamic experiment, most of the swing angle measurement errors are controlled within the error range of ±0.15. The method does not rely on the cooperative information provided by the nozzle, has high measurement accuracy, and meets the needs of non-cooperative measurement of the nozzle swing angle. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0101] Figure 1 The display is a schematic diagram of the feature point triangulation method;

[0102] Figure 2 Displayed as a feature point extraction effect diagram;

[0103] Figure 3 Displayed as a feature point matching effect diagram. DETAILED DESCRIPTION

[0104] The present invention is described in detail below in conjunction with specific embodiments. Specific embodiment one:

[0106] according to Figures 1 to 3 As shown, the specific optimization technical solution adopted by the present invention to solve the above technical problems is: the present invention relates to a multi-view engine nozzle swing angle measurement method based on non-cooperative measurement.

[0107] A method for measuring a multi-view engine nozzle swing angle based on non-cooperative measurement, the method comprising the following steps:

[0108] Step 1: Extract nozzle feature points and use SuperPoint network to extract image features;

[0109] Step 2: After feature point extraction, use the SuperGlue network to complete feature point matching;

[0110] Step 3: Camera pose estimation: after extracting and matching the nozzle feature points, the camera pose needs to be estimated;

[0111] Step 4: Reconstruct three-dimensional feature points. Calculate the camera projection matrix corresponding to the added image through camera pose estimation and reconstruct the unreconstructed feature points in the newly added image. Specific embodiment 2:

[0113] The difference between the second embodiment of the present application and the first embodiment is that:

[0114] The step 1 is specifically as follows:

[0115] The SuperPoint feature extraction network can simultaneously extract feature points in an image and generate feature descriptors. The network model includes an encoder, a feature point decoder, and a descriptor decoder.

[0116] The SuperPoint network is trained using the labeled data set, and its loss function is composed of the feature point loss L p and descriptor loss L d It consists of two parts:

[0117] loss = L p (χ,Y)+L p (χ′,Y′)+λL d (D,D′,S) (1)

[0118] Among them, λ is the balance coefficient of feature point loss and descriptor loss; Y is the feature point label; χ′, D′, Y′ are the results obtained in the image after homography transformation; S represents the feature point matching set in the image;

[0119]

[0120]

[0121] Among them, l p ,l d For each pixel, the feature point loss and descriptor loss are hw ∈χ,y hw ∈Y,d hw ∈D,d′ h′w′ ∈D′,s hwh′w′ ∈S. Specific embodiment three:

[0123] The difference between the third embodiment of the present application and the second embodiment is that:

[0124] The step 2 is specifically as follows:

[0125] Step 2.1: The SuperGlue network obtains the matching relationship of feature points based on the feature points and descriptors in the two images, including the attention GNN and the optimal matching layer module. For the feature point positions output by the SuperPoint network and feature descriptors Will In the input key point encoder, the attention GNN uses a multi-layer perceptron to increase the dimension of the position information of the feature point and couples it with the descriptor to obtain the initial information of the feature point:

[0126]

[0127] Step 2.2: When setting is the feature point information of the feature point in the l-th layer network, and the enhanced feature point information is expressed as:

[0128]

[0129] in, Indicates serial operation, m ε→i Represents the aggregation information of feature points, ε∈{ε self ,ε cross}, ε self and ε cross The autocorrelation and cross-correlation attention mechanisms are used to aggregate information for feature points. When l is an odd number, ε = ε self ; When l is an even number, ε=ε cross ;

[0130] Step 2.3: The enhanced feature point information is transformed linearly to obtain the matching descriptor of the feature point:

[0131]

[0132] Among them, W,b are the weight and bias parameters in the linear transformation;

[0133] In the optimal matching layer, the allocation matrix P is constructed to satisfy:

[0134] P1 N ≤1 M

[0135] P T 1 M ≤1 N (6)

[0136] f i A , Match the descriptors for the feature points in images A and B respectively, and calculate the score matrix S based on the vector inner product:

[0137]

[0138] By maximizing The optimal allocation matrix P is obtained, and the Sinkhorn algorithm is used in SuperGlue to solve the optimal transportation problem. Specific embodiment four:

[0140] The difference between the fourth embodiment of the present application and the third embodiment is that:

[0141] The step 3 comprises:

[0142] Step 3.1: Assume that the coordinates of n feature points in the world coordinate system and the camera coordinate system are P i =(X i ,Y i ,Z i ) T and Q i =(X′ i ,Y′ i ,Z′ i ) T , from the motion properties of rigid bodies, we can know that:

[0143] Q i =RP i +t (8)

[0144] in, and t=(t x t yt z ) T Represents the transformation relationship between the camera coordinate system and the world coordinate system. The camera pose estimation problem is to solve R and t;

[0145] Camera optical center O c , feature point Q i and the projection point p on the normalized image plane i Collinearity, according to the collinearity relationship, we can get:

[0146]

[0147]

[0148]

[0149] Step 3.2: Establish the collinearity error objective function of feature point image space:

[0150]

[0151] In the formula, are the image coordinates affected by the noise.

[0152] The collinear relationship is expressed as O c Q i In O c p i The projection vector in the direction is equal to O c Q i :

[0153] RP i +t=V i (RP i +t)(11)

[0154] in, is the ideal line-of-sight projection matrix;

[0155] Step 3.3: Affected by factors such as image noise, O c , With Q i Not collinear, O c Q i exist The projection vector O in the direction c Q i⊥ With O c Q i There is a deviation between them, which is the feature point projection vector error;

[0156] The feature point projection vector error is expressed as:

[0157]

[0158] in, is the actual line of sight projection matrix.

[0159] Step 3.4: Establish the feature point projection vector error objective function:

[0160]

[0161] In order to reduce the parameters to be solved, the CGR parameterized rotation matrix is ​​used:

[0162]

[0163] Among them, s1, s2, s3 are three unknown rotation parameters.

[0164] Introducing Kronecker product:

[0165]

[0166] in

[0167]

[0168] Vec(R)=[r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 ] T

[0169] In formula (13), all feature point information is included in In the equation, when R is given, t can be expressed as:

[0170]

[0171] in

[0172]

[0173] Substitute equation (15) and equation (16) into equation (13):

[0174]

[0175] Substituting the rotation parameterized R into equation (17), we can obtain a polynomial containing three unknown rotation parameters. In order to obtain the solution of the rotation parameters, the partial derivative of the objective function with respect to s1, s2, and s3 is calculated:

[0176]

[0177] Thus, three polynomial equations consisting of parameter variables s1, s2, and s3 are obtained. Substituting the obtained rotation parameters into equations (14) and (16) to obtain the camera's position and posture:

[0178] The difference between the fifth embodiment of the present invention and the fourth embodiment is that:

[0179] The step 4 is specifically as follows:

[0180] After extracting and matching the feature points of the nozzle area in the captured image, the trajectories of n feature points in m images are obtained. Based on the pinhole camera model, the feature points in each image can be expressed as:

[0181] λ ij u ij =H i X j (19)

[0182] Among them, H i is the projection matrix of the i-th camera; X j is the world coordinate of the feature point; u ij is the image coordinate of the jth feature point on the i-th camera; ij is a scalar;

[0183] The three-dimensional coordinates of the feature points can be obtained according to the triangulation method. Assuming that the camera intrinsic parameter matrix K remains unchanged, the world coordinate system coincides with the camera coordinate system in the first frame image, then the camera projection matrices corresponding to the two images can be expressed as:

[0184] H1=K[R1t1]=K[I0]

[0185] H2=K[R2t2]=K[Rt](20)

[0186] Among them, R and t are the external parameters of the camera corresponding to the second frame image.

[0187] Convert the fundamental matrix to the essential matrix:

[0188] E=K T FK(21)

[0189] Perform singular value decomposition on the essential matrix:

[0190] E=UDV T (twenty two)

[0191] In the singular value decomposition, for any E, two rotation matrices and translation vectors can be decomposed:

[0192]

[0193]

[0194] in, Represents the rotation matrix obtained by rotating 90° around the Z axis.

[0195] The reconstructed feature points are located in front of the camera to eliminate the ambiguous values, so as to obtain the correct R and t; given the camera projection matrix, X can be calculated based on the image coordinates of the feature points in two or more views. j The coordinates of .

[0196] Rewrite equation (19) as:

[0197] [u ij ] × H i X j =AX j =0 (24)

[0198] X j The solution is the matrix A T The eigenvector corresponding to the smallest eigenvalue of A. Specific embodiment six:

[0200] The difference between the sixth embodiment of the present invention and the fifth embodiment is that:

[0201] Each time an image is added, the correspondence between the two-dimensional feature points and the reconstructed three-dimensional feature points is obtained based on the matching relationship of the feature points in the image. The camera projection matrix corresponding to the added image is calculated using the camera pose estimation algorithm, and the unreconstructed feature points in the newly added image are reconstructed. Specific embodiment seven:

[0203] The difference between the seventh embodiment of the present invention and the sixth embodiment is that:

[0204] When the nozzle swings within the horizontal and vertical ranges, the maximum measurement errors of the nozzle swing angle are 0.12° and 0.13° respectively; in dynamic experiments, most of the swing angle measurement errors are controlled within the error range of ±0.15. Specific embodiment eight:

[0206] The difference between the eighth embodiment of the present invention and the seventh embodiment is that:

[0207] The present invention provides a multi-view engine nozzle swing angle measurement system based on non-cooperative measurement, the system comprising:

[0208] A feature extraction module, wherein the feature extraction module extracts nozzle feature points and uses a SuperPoint network to extract image features;

[0209] A matching module, which uses a SuperGlue network to complete the matching of feature points after feature point extraction;

[0210] A pose estimation module, which performs camera pose estimation. After extracting and matching nozzle feature points, the camera pose needs to be estimated;

[0211] A three-dimensional reconstruction module is used to reconstruct three-dimensional feature points, calculate the camera projection matrix corresponding to the added image through camera pose estimation, and reconstruct the unreconstructed feature points in the newly added image. Specific embodiment nine:

[0213] The difference between the ninth embodiment of the present invention and the eighth embodiment is that:

[0214] The present invention provides a computer-readable storage medium on which a computer program is stored. The program is executed by a processor to implement a multi-view engine nozzle swing angle measurement method based on non-cooperative measurement.

[0215] The method is specifically as follows:

[0216] Step 1: Extraction of nozzle feature points. Since the shape of the nozzle is similar to a truncated cone, the features of the nozzle surface edge, side edge and their intersections vary significantly, so the SuperPoint network is used to extract image features.

[0217] The SuperPoint feature extraction network can simultaneously extract feature points in an image and generate feature descriptors. The network model is mainly composed of three parts: an encoder, a feature point decoder, and a descriptor decoder. The encoder uses a structure similar to VGG, which mainly reduces the dimension of the input image and obtains image features.

[0218] The SuperPoint network uses a self-supervised training method. First, a simple virtual graph with known corner point positions is used to train the encoder and feature point decoder. The trained encoder and feature point decoder are used to annotate the feature points in the real image and the image after random homography transformation. All feature points are transformed to the original image to complete the labeling of image feature points. The SuperPoint network is trained using the annotated data set, and its loss function is composed of the feature point loss L p and descriptor loss L d It consists of two parts:

[0219] loss = L p (χ,Y)+L p (χ′,Y′)+λL d (D,D′,S) (1)

[0220] Among them, λ is the balance coefficient between feature point loss and descriptor loss; Y is the feature point label; χ′, D′, Y′ are the results obtained in the image after homography transformation; S represents the feature point matching set in the image.

[0221]

[0222]

[0223] In the formula, l p ,l d For each pixel, the feature point loss and descriptor loss are hw ∈χ,y hw ∈Y,d hw ∈D,d′ h′w′ ∈D′,s hwh′w′ ∈S.

[0224] Step 2: Feature point matching. After feature point extraction, the SuperGlue network is used to complete feature point matching. The SuperGlue network obtains the matching relationship of feature points based on the feature points and descriptors in the two images. It is mainly composed of two modules: attention GNN and optimal matching layer. For the feature point position output by the SuperPoint network and feature descriptors Will In the input key point encoder, the attention GNN uses a multi-layer perceptron (MLP) to upgrade the location information of the feature points and couple it with the descriptor to obtain the initial information of the feature points:

[0225]

[0226] Assumptions is the feature point information of the feature point in the l-th layer network, then the enhanced feature point information can be expressed as:

[0227]

[0228] in, Indicates serial operation, m ε→i Represents the aggregation information of feature points, ε∈{ε self ,ε cross}, ε self and ε cross The autocorrelation and cross-correlation attention mechanisms are used to aggregate information for feature points. When l is an odd number, ε = ε self ; When l is an even number, ε=ε cross .

[0229] The enhanced feature point information is transformed linearly to obtain the matching descriptor of the feature point:

[0230]

[0231] Among them, W,b are the weight and bias parameters in the linear transformation.

[0232] In the optimal matching layer, the allocation matrix P is constructed to satisfy:

[0233] P1 N ≤1 M

[0234] P T 1 M ≤1 N (6)

[0235] f i A , Match the descriptors for the feature points in images A and B respectively, and calculate the score matrix S based on the vector inner product:

[0236]

[0237] By maximizing The optimal allocation matrix P is obtained, and the Sinkhorn algorithm is used in SuperGlue to solve the optimal transportation problem.

[0238] Step 3: Camera pose estimation. In order to accurately measure the nozzle swing angle, the camera pose needs to be estimated after the nozzle feature points are extracted and matched. Suppose the coordinates of n feature points in the world coordinate system and the camera coordinate system are P and i =(X i ,Y i ,Z i ) T and Q i =(X i ′,Y i ′,Z i ′) T . From the motion properties of a rigid body, we know that:

[0239] Q i =RP i +t(8)

[0240] In the formula, and t=(t x t y t z ) T It represents the transformation relationship between the camera coordinate system and the world coordinate system. The camera pose estimation problem is to solve R and t.

[0241] In an ideal situation, the camera optical center O c , feature point Qi and the projection point p on the normalized image plane i Collinearity, according to the collinearity relationship, we can get:

[0242]

[0243]

[0244]

[0245] Therefore, the collinearity error objective function of the feature point image space can be established:

[0246]

[0247] In the formula, are the image coordinates affected by the noise.

[0248] The feature point image space collinearity error objective function established in formula (5) is universal and does not fully utilize the information in the pose estimation problem. The collinearity relationship can be expressed as c Q i In O c p i The projection vector in the direction is equal to O c Q i :

[0249] RP i +t=V i (RP i +t)(11)

[0250] in, is the ideal line-of-sight projection matrix.

[0251] In actual situations, affected by factors such as image noise, O c , With Q i Not collinear, O c Q i exist The projection vector O in the direction c Q i⊥ With O c Q i There is a deviation between them, which is the feature point projection vector error.

[0252] The feature point projection vector error can be expressed as:

[0253]

[0254] in, is the actual line of sight projection matrix.

[0255] Finally, the feature point projection vector error objective function is established:

[0256]

[0257] In order to reduce the parameters to be solved, the Cayley-Gibbs-Rodriguez (CGR) parameterized rotation matrix is ​​used:

[0258]

[0259] Where s1, s2, s3 are three unknown rotation parameters.

[0260] Introducing Kronecker product:

[0261]

[0262] in

[0263]

[0264] Vec(R)=[r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 ] T

[0265] In formula (13), all feature point information is included in In the equation, when R is given, t can be expressed as:

[0266]

[0267] in

[0268]

[0269] Substitute equation (15) and equation (16) into equation (13):

[0270]

[0271] Substituting the rotation parameterized R into equation (17), we can get a polynomial containing three unknown rotation parameters. In order to obtain the solution of the rotation parameters, the partial derivative of the objective function with respect to s1, s2, and s3 is calculated:

[0272]

[0273] Thus, three polynomial equations consisting of parameter variables s1, s2, and s3 are obtained. Substituting the obtained rotation parameters into equations (14) and (16) to obtain the camera pose.

[0274] Step 4: 3D feature point reconstruction. After extracting and matching the feature points of the nozzle area in the captured image, the trajectories of n feature points in m images are obtained. Based on the pinhole camera model, the feature points in each image can be expressed as:

[0275] λ ij u ij =H i X j (19)

[0276] In the formula, H i is the projection matrix of the i-th camera; X j is the world coordinate of the feature point; u ij is the image coordinate of the jth feature point on the i-th camera; ij is a scalar.

[0277] The three-dimensional coordinates of the feature points can be obtained according to the triangulation method. Assuming that the camera intrinsic parameter matrix K remains unchanged, the world coordinate system coincides with the camera coordinate system in the first frame image, then the camera projection matrices corresponding to the two images can be expressed as:

[0278] H1=K[R1 t1]=K[I 0]

[0279] H2=K[R2 t2]=K[R t] (20)

[0280] Among them, R and t are the external parameters of the camera corresponding to the second frame image.

[0281] Convert the fundamental matrix to the essential matrix:

[0282] E=K T FK (21)

[0283] Perform singular value decomposition on the essential matrix:

[0284] E=UDV T (twenty two)

[0285] In the singular value decomposition, for any E, two rotation matrices and translation vectors can be decomposed:

[0286]

[0287]

[0288] in, Represents the rotation matrix obtained by rotating 90° around the Z axis.

[0289] The reconstructed feature points are located in front of the camera to eliminate the ambiguous values ​​and obtain the correct R and t. Given the camera projection matrix, X can be calculated based on the image coordinates of the feature points in two or more views. j The coordinates of .

[0290] Rewrite equation (19) as:

[0291] [u ij ] × H i X j =AX j =0(24)

[0292] X j The solution is the matrix A T The eigenvector corresponding to the minimum eigenvalue of A. Each time an image is added, the correspondence between the two-dimensional feature points and the reconstructed three-dimensional feature points is obtained based on the matching relationship of the feature points in the image. The camera projection matrix corresponding to the added image is calculated using the camera pose estimation algorithm and the unreconstructed feature points in the newly added image are reconstructed. Specific embodiment ten:

[0294] The difference between the tenth embodiment of the present invention and the ninth embodiment is that:

[0295] The present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, a multi-view engine nozzle swing angle measurement method based on non-cooperative measurement is implemented. Specific embodiment eleven:

[0297] The difference between the eleventh embodiment of the present invention and the tenth embodiment is that:

[0298] Step 1: Extraction of nozzle feature points. Since the shape of the nozzle is similar to a truncated cone, the features of the nozzle surface edge, side edge and their intersections vary significantly, so the SuperPoint network is used to extract image features.

[0299] The SuperPoint feature extraction network can simultaneously extract feature points in an image and generate feature descriptors. The network model is mainly composed of three parts: an encoder, a feature point decoder, and a descriptor decoder. The encoder uses a structure similar to VGG, which mainly reduces the dimension of the input image and obtains image features.

[0300] The SuperPoint network uses a self-supervised training method. First, a simple virtual graph with known corner point positions is used to train the encoder and feature point decoder. The trained encoder and feature point decoder are used to annotate the feature points in the real image and the image after random homography transformation. All feature points are transformed to the original image to complete the labeling of image feature points. The SuperPoint network is trained using the annotated data set, and its loss function is composed of the feature point loss L p and descriptor loss L d It consists of two parts:

[0301] loss = L p (χ,Y)+L p (χ′,Y′)+λL d (D,D′,S)(1)

[0302] Among them, λ is the balance coefficient between feature point loss and descriptor loss; Y is the feature point label; χ′, D′, Y′ are the results obtained in the image after homography transformation; S represents the feature point matching set in the image.

[0303]

[0304]

[0305] In the formula, l p ,l d For each pixel, the feature point loss and descriptor loss are hw ∈χ,y hw ∈Y,d hw ∈D,d′ h′w′ ∈D′,s hwh′w′ ∈S.

[0306] Step 2: Feature point matching. After feature point extraction, the SuperGlue network is used to complete feature point matching. The SuperGlue network obtains the matching relationship of feature points based on the feature points and descriptors in the two images. It is mainly composed of two modules: attention GNN and optimal matching layer. For the feature point position output by the SuperPoint network and feature descriptors Will In the input key point encoder, the attention GNN uses a multi-layer perceptron (MLP) to upgrade the location information of the feature points and couple it with the descriptor to obtain the initial information of the feature points:

[0307]

[0308] Assumptions is the feature point information of the feature point in the l-th layer network, then the enhanced feature point information can be expressed as:

[0309]

[0310] in, Indicates serial operation, m ε→i Represents the aggregation information of feature points, ε∈{ε self ,ε cross}, ε self and ε cross The autocorrelation and cross-correlation attention mechanisms are used to aggregate information for feature points. When l is an odd number, ε = ε self ; When l is an even number, ε=ε cross .

[0311] The enhanced feature point information is transformed linearly to obtain the matching descriptor of the feature point:

[0312]

[0313] Among them, W,b are the weight and bias parameters in the linear transformation.

[0314] In the optimal matching layer, the allocation matrix P is constructed to satisfy:

[0315] P1 N ≤1 M

[0316] P T 1 M ≤1 N (6)

[0317] f i A , Match the descriptors for the feature points in images A and B respectively, and calculate the score matrix S based on the vector inner product:

[0318]

[0319] By maximizing The optimal allocation matrix P is obtained, and the Sinkhorn algorithm is used in SuperGlue to solve the optimal transportation problem.

[0320] Step 3: Camera pose estimation. In order to accurately measure the nozzle swing angle, the camera pose needs to be estimated after the nozzle feature points are extracted and matched. Suppose the coordinates of n feature points in the world coordinate system and the camera coordinate system are P and i =(X i ,Y i ,Z i ) Tand Q i =(X′ i ,Y′ i ,Z′ i ) T . From the motion properties of a rigid body, we know that:

[0321] Q i =RP i +t (8)

[0322] In the formula, and t=(t x t y t z ) T It represents the transformation relationship between the camera coordinate system and the world coordinate system. The camera pose estimation problem is to solve R and t.

[0323] In an ideal situation, the camera optical center O c , feature point Q i and the projection point p on the normalized image plane i Collinearity, according to the collinearity relationship, we can get:

[0324]

[0325]

[0326]

[0327] Therefore, the collinearity error objective function of the feature point image space can be established:

[0328]

[0329] In the formula, are the image coordinates affected by the noise.

[0330] The feature point image space collinearity error objective function established in formula (5) is universal and does not fully utilize the information in the pose estimation problem. The collinearity relationship can be expressed as c Q i In O c p i The projection vector in the direction is equal to O c Q i :

[0331] RP i +t=V i (RP i +t)(11)

[0332] in, is the ideal line-of-sight projection matrix.

[0333] In actual situations, affected by factors such as image noise, O c , With Q i Not collinear, O c Q i exist The projection vector O in the direction c Q i⊥ With O c Q i There is a deviation between them, which is the feature point projection vector error.

[0334] The feature point projection vector error can be expressed as:

[0335]

[0336] in, is the actual line of sight projection matrix.

[0337] Finally, the feature point projection vector error objective function is established:

[0338]

[0339] In order to reduce the parameters to be solved, the Cayley-Gibbs-Rodriguez (CGR) parameterized rotation matrix is ​​used:

[0340]

[0341] Where s1, s2, s3 are three unknown rotation parameters.

[0342] Introducing Kronecker product:

[0343]

[0344] in

[0345]

[0346] Vec(R)=[r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 ] T

[0347] In formula (13), all feature point information is included in In the equation, when R is given, t can be expressed as:

[0348]

[0349] in

[0350]

[0351] Substitute equation (15) and equation (16) into equation (13):

[0352]

[0353] Substituting the rotation parameterized R into equation (17), we can get a polynomial containing three unknown rotation parameters. In order to obtain the solution of the rotation parameters, the partial derivative of the objective function with respect to s1, s2, and s3 is calculated:

[0354]

[0355] Thus, three polynomial equations consisting of parameter variables s1, s2, and s3 are obtained. Substituting the obtained rotation parameters into equations (14) and (16) to obtain the camera pose.

[0356] Step 4: 3D feature point reconstruction. After extracting and matching the feature points of the nozzle area in the captured image, the trajectories of n feature points in m images are obtained. Based on the pinhole camera model, the feature points in each image can be expressed as:

[0357] λ ij u ij =H i X j (19)

[0358] In the formula, H i is the projection matrix of the i-th camera; X j is the world coordinate of the feature point; u ij is the image coordinate of the jth feature point on the i-th camera; ij is a scalar.

[0359] The three-dimensional coordinates of the feature points can be obtained according to the triangulation method. Assuming that the camera intrinsic parameter matrix K remains unchanged, the world coordinate system coincides with the camera coordinate system in the first frame image, then the camera projection matrices corresponding to the two images can be expressed as:

[0360] H1=K[R1t1]=K[I0]

[0361] H2=K[R2t2]=K[Rt](20)

[0362] Among them, R and t are the external parameters of the camera corresponding to the second frame image.

[0363] Convert the fundamental matrix to the essential matrix:

[0364] E=K T FK(21)

[0365] Perform singular value decomposition on the essential matrix:

[0366] E=UDV T (twenty two)

[0367] In the singular value decomposition, for any E, two rotation matrices and translation vectors can be decomposed:

[0368]

[0369]

[0370] in, Represents the rotation matrix obtained by rotating 90° around the Z axis.

[0371] The reconstructed feature points are located in front of the camera to eliminate the ambiguous values ​​and obtain the correct R and t. Given the camera projection matrix, X can be calculated based on the image coordinates of the feature points in two or more views. j The coordinates of .

[0372] Rewrite equation (19) as:

[0373] [u ij ] × H i X j =AX j =0(24)

[0374] X j The solution is the matrix A T The eigenvector corresponding to the minimum eigenvalue of A. Each time an image is added, the correspondence between the two-dimensional feature points and the reconstructed three-dimensional feature points is obtained based on the matching relationship of the feature points in the image. The camera projection matrix corresponding to the added image is calculated using the camera pose estimation algorithm and the unreconstructed feature points in the newly added image are reconstructed.

[0375] Experiments show that in static experiments. When the nozzle swings within the range in the horizontal and vertical directions, the maximum measurement errors of the nozzle swing angle are 0.12° and 0.13° respectively; in dynamic experiments, most of the swing angle measurement errors are controlled within the error range of ±0.15. This method does not rely on the cooperative information provided by the nozzle, has high measurement accuracy, and meets the needs of non-cooperative measurement of the nozzle swing angle. The above is only a preferred implementation of a multi-view engine nozzle swing angle measurement method based on non-cooperative measurement. The protection scope of a multi-view engine nozzle swing angle measurement method based on non-cooperative measurement is not limited to the above embodiments. All technical solutions under this idea belong to the protection scope of the present invention. It should be pointed out that for those skilled in the art, a number of improvements and changes without departing from the principles of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A multi-view engine nozzle swing angle measurement method based on non-cooperative measurement, characterized in that: The method comprises the following steps: Step 1: Extract nozzle feature points and use SuperPoint network to extract image features; Step 2: After feature point extraction, use the SuperGlue network to complete feature point matching; Step 3: Camera pose estimation: after extracting and matching the nozzle feature points, the camera pose needs to be estimated; Step 4: Reconstruct three-dimensional feature points. Calculate the camera projection matrix corresponding to the added image through camera pose estimation and reconstruct the unreconstructed feature points in the newly added image.

2. The method according to claim 1, characterized in that: The step 1 is specifically as follows: The SuperPoint feature extraction network can simultaneously extract feature points in an image and generate feature descriptors. The network model includes an encoder, a feature point decoder, and a descriptor decoder. The SuperPoint network is trained using the labeled data set, and its loss function is composed of the feature point loss L p and descriptor loss L d It consists of two parts: loss=L p (x,Y)+L p (x′,Y′)+λL d (D,D′,S) (1) Among them, l p ,l d For each pixel, the feature point loss and descriptor loss are hw ∈χ,y hw ∈Y,d hw ∈D,d h ″ w′ ∈D′,s hwh′w′ ∈S.

3. The method according to claim 2, characterized in that: The step 2 is specifically as follows: Step 2.1: The SuperGlue network obtains the matching relationship of feature points based on the feature points and descriptors in the two images, including the attention GNN and the optimal matching layer module. For the feature point positions output by the SuperPoint network and feature descriptors Will In the input key point encoder, the attention GNN uses a multi-layer perceptron to increase the dimension of the position information of the feature point and couples it with the descriptor to obtain the initial information of the feature point: Step 2.2: When setting is the feature point information of the feature point in the l-th layer network, and the enhanced feature point information is expressed as: Step 2.3: The enhanced feature point information is transformed linearly to obtain the matching descriptor of the feature point: Among them, W,b are the weight and bias parameters in the linear transformation; In the optimal matching layer, the allocation matrix P is constructed to satisfy: P1 N ≤1 M P T 1 M ≤1 N (6) f i A , Match the descriptors for the feature points in images A and B respectively, and calculate the score matrix S based on the vector inner product: By maximizing The optimal allocation matrix P is obtained, and the Sinkhorn algorithm is used in SuperGlue to solve the optimal transportation problem.

4. The method according to claim 1, characterized in that: The step 3 comprises: Step 3.1: Assume that the coordinates of n feature points in the world coordinate system and the camera coordinate system are P i =(X i ,Y i ,Z i ) T and Q i =(X i ′,Y i ′,Z i ′) T , from the motion properties of rigid bodies, we can know that: Q i =RP i +t (8) in, and t=(t x t y t z ) T Represents the transformation relationship between the camera coordinate system and the world coordinate system. The camera pose estimation problem is to solve R and t; Camera optical center O c , feature point Q i and the projection point p on the normalized image plane i Collinearity, according to the collinearity relationship, we can get: Step 3.2: Establish the collinearity error objective function of feature point image space: In the formula, is the image coordinates disturbed by noise; The collinear relationship is expressed as O c Q i In O c p i The projection vector in the direction is equal to O c Q i : RP i +t=V i (RP i +t) (11) in, is the ideal line-of-sight projection matrix; Step 3.3: Affected by factors such as image noise, O c , With Q i Not collinear, O c Q i exist The projection vector O in the direction c Q i⊥ With O c Q i There is a deviation between them, which is the feature point projection vector error; The feature point projection vector error is expressed as: in, is the actual line of sight projection matrix; Step 3.4: Establish the feature point projection vector error objective function: In order to reduce the parameters to be solved, the CGR parameterized rotation matrix is ​​used: Among them, s1, s2, s3 are three unknown rotation parameters. Introducing Kronecker product: in Vec(R)=[r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 ] T In formula (13), all feature point information is included in In the equation, when R is given, t can be expressed as: in Substitute equation (15) and equation (16) into equation (13): Substituting the rotation parameterized R into equation (17), we can obtain a polynomial containing three unknown rotation parameters. In order to obtain the solution of the rotation parameters, the partial derivative of the objective function with respect to s1, s2, and s3 is calculated: Thus, three polynomial equations consisting of parameter variables s1, s2, and s3 are obtained. Substituting the obtained rotation parameters into equations (14) and (16) to obtain the camera pose.

5. The method according to claim 4, characterized in that: The step 4 is specifically as follows: After extracting and matching the feature points of the nozzle area in the captured image, the trajectories of n feature points in m images are obtained. Based on the pinhole camera model, the feature points in each image can be expressed as: l ij you ij =H i X j (19) The three-dimensional coordinates of the feature points can be obtained according to the triangulation method. Assuming that the camera intrinsic parameter matrix K remains unchanged, the world coordinate system coincides with the camera coordinate system in the first frame image, then the camera projection matrices corresponding to the two images can be expressed as: H1=K[R1 t1]=K[I 0] H2=K[R2 t2]=K[R t] (20) Among them, R and t are the external parameters of the camera corresponding to the second frame image; Convert the fundamental matrix to the essential matrix: E=K T FK (21) Perform singular value decomposition on the essential matrix: E=UDV T (22) In the singular value decomposition, for any E, two rotation matrices and translation vectors can be decomposed: in, Represents the rotation matrix obtained by rotating 90° around the Z axis; The reconstructed feature points are located in front of the camera to eliminate the ambiguous values, so as to obtain the correct R and t; given the camera projection matrix, X can be calculated based on the image coordinates of the feature points in two or more views. j The coordinates of Rewrite equation (19) as: [u ij ] × H i X j =AX j =0 (24) X j The solution is the matrix A T The eigenvector corresponding to the smallest eigenvalue of A.

6. The method according to claim 1, characterized in that: Each time an image is added, the correspondence between the two-dimensional feature points and the reconstructed three-dimensional feature points is obtained based on the matching relationship of the feature points in the image. The camera projection matrix corresponding to the added image is calculated using the camera pose estimation algorithm, and the unreconstructed feature points in the newly added image are reconstructed.

7. The method according to claim 5, characterized in that: When the nozzle swings within the horizontal and vertical ranges, the maximum measurement errors of the nozzle swing angle are 0.12° and 0.13° respectively; in dynamic experiments, most of the swing angle measurement errors are controlled within the error range of ±0.

15.

8. A multi-view engine nozzle swing angle measurement system based on non-cooperative measurement, characterized by: The system comprises: A feature extraction module, wherein the feature extraction module extracts nozzle feature points and uses a SuperPoint network to extract image features; A matching module, which uses a SuperGlue network to complete the matching of feature points after feature point extraction; A pose estimation module, which performs camera pose estimation. After extracting and matching nozzle feature points, the camera pose needs to be estimated; A three-dimensional reconstruction module is used to reconstruct three-dimensional feature points, calculate the camera projection matrix corresponding to the added image through camera pose estimation, and reconstruct the unreconstructed feature points in the newly added image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to claims 1-7.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method of claims 1-7 is implemented.

Images