Ship-borne equipment attitude measurement method based on multiple feature points and multiple views and related device

Abstract

The invention discloses a ship-borne equipment attitude measurement method based on multiple feature points and multiple views and a related device, and relates to the technical field of ship-borne equipment attitude monitoring, and the method comprises the steps: obtaining multiple view angle images of ship-borne equipment, and predicting the pixel coordinates and confidence of each feature point in each view angle image through a feature point recognition network model; determining the pixel coordinates of the effective feature points based on the confidence coefficient; according to the pixel coordinates of the effective feature points corresponding to each feature part and the camera parameters, determining the feature point three-dimensional coordinates of each feature part in a camera coordinate system by using a multi-view measurement method; and determining a mapping relation between the camera coordinate system and the ship-borne equipment body coordinate system by combining the feature point three-dimensional coordinates of each feature part in the ship-borne equipment body coordinate system, and determining the attitude of the ship-borne equipment in the camera coordinate system based on the mapping relation. According to the invention, the stability and the measurement precision of the attitude monitoring system in a complex environment are improved.

Description

Technical Field

[0001] The present application relates to the technical field of shipborne equipment attitude monitoring, and particularly to a shipborne equipment attitude measurement method and related device based on multi-feature points and multi-views. Background Art

[0002] During the use of shipborne equipment (such as unmanned aerial vehicles, naval guns, etc.), precise attitude monitoring is required to ensure operation safety. However, the traditional attitude monitoring methods have the following problems:

[0003] Inertial system: It cannot provide stable high-precision attitude data for a long time.

[0004] GPS system: It is vulnerable to external environment interference and cannot work stably in a high-interference environment.

[0005] Single-view vision method: The marked points are easily blocked, resulting in the inability to obtain complete attitude information.

[0006] Therefore, there is an urgent need for a stable, reliable and high-precision attitude monitoring method at present. Summary of the Invention

[0007] The purpose of the present application is to provide a shipborne equipment attitude measurement method and related device based on multi-feature points and multi-views, which can improve the reliability and accuracy of equipment attitude monitoring.

[0008] To achieve the above purpose, the present application provides the following solutions:

[0009] In the first aspect, the present application provides a shipborne equipment attitude measurement method based on multi-feature points and multi-views, including:

[0010] Obtain images of the shipborne equipment from multiple perspectives; each of the images contains a number of feature points; one feature point in each of the images represents a point at a feature part of the shipborne equipment.

[0011] Input the images of each perspective into a trained feature point recognition network model, and predict the pixel coordinates and corresponding confidence levels of each feature point in the images of each perspective.

[0012] Determine the pixel coordinates of the feature points with a confidence level greater than a preset threshold in the images of each perspective as the pixel coordinates of the valid feature points.

[0013] According to the pixel coordinates of the valid feature points corresponding to each feature part and the camera parameters, use the multi-view measurement method to calculate the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the camera coordinate system.

[0014] Determine the mapping relationship between the camera coordinate system and the shipborne equipment body coordinate system according to the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the camera coordinate system and the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the shipborne equipment body coordinate system; the mapping relationship includes a rotation matrix and a translation matrix.

[0015] Determine the attitude of the shipborne equipment in the camera coordinate system according to the mapping relationship between the camera coordinate system and the shipborne equipment body coordinate system; the attitude includes pitch angle, roll angle and heading angle.

[0016] In a second aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the shipborne equipment attitude measurement method based on multi-feature points and multi-views described in the first aspect.

[0017] In a third aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the shipborne equipment attitude measurement method based on multi-feature points and multi-views described in the first aspect.

[0018] In a fourth aspect, the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the shipborne equipment attitude measurement method based on multi-feature points and multi-views described in the first aspect.

[0019] According to the specific embodiments provided by the present application, the present application has the following technical effects:

[0020] The present application provides a shipborne equipment attitude measurement method and related device based on multi-feature points and multi-views. The method includes: acquiring images of the shipborne equipment from multiple perspectives; inputting the images of each perspective into a trained feature point recognition network model to predict the pixel coordinates and corresponding confidence levels of each feature point in the images of each perspective; determining the pixel coordinates of the feature points with a confidence level greater than a preset threshold in the images of each perspective as the pixel coordinates of the valid feature points; calculating the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the camera coordinate system by using the multi-view measurement method according to the pixel coordinates of the valid feature points corresponding to each feature part and the camera parameters; determining the mapping relationship between the camera coordinate system and the shipborne equipment body coordinate system according to the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the camera coordinate system and the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the shipborne equipment body coordinate system; and determining the attitude of the shipborne equipment in the camera coordinate system according to the mapping relationship between the camera coordinate system and the shipborne equipment body coordinate system. By combining images from multiple perspectives to measure the attitude of shipborne equipment, the present application improves the measurement accuracy, overcomes the occlusion problem of traditional single-view methods, and does not rely on GPS and inertial systems, so it is not affected by the external environment, improving the stability and measurement accuracy of the attitude monitoring system in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0022] Figure 1 It is an application environment diagram of a shipborne equipment attitude measurement method based on multi-feature points and multi-views in an embodiment of the present application;

[0023] Figure 2 It is a flowchart of a shipborne equipment attitude measurement method based on multi-feature points and multi-views provided in an embodiment of the present application;

[0024] Figure 3 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the protection scope of the present application.

[0026] To make the above objects, features, and advantages of the present application more obvious and understandable, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0027] The shipborne equipment attitude measurement method based on multi-feature points and multi-views provided by the embodiments of the present application can be applied to an application environment as Figure 1 shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send images of multiple perspectives of the shipborne equipment (abbreviated as equipment) to the server 104. After receiving the images of multiple perspectives of the shipborne equipment, the server 104 inputs the images of each perspective into the trained feature point recognition network model to predict the pixel coordinates and corresponding confidence levels of each feature point in the images of each perspective; the pixel coordinates of the feature points with a confidence level greater than the preset threshold in the images of each perspective are determined as the pixel coordinates of the valid feature points; according to the pixel coordinates of the valid feature points corresponding to each feature part and the camera parameters, using the multi-view measurement method, the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the camera coordinate system are calculated; according to the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the camera coordinate system and the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the shipborne equipment body coordinate system (also called the equipment body coordinate system, the body coordinate system), the mapping relationship between the camera coordinate system and the shipborne equipment body coordinate system is determined; according to the mapping relationship between the camera coordinate system and the shipborne equipment body coordinate system, the attitude of the shipborne equipment in the camera coordinate system is determined. The server 104 can feedback the obtained attitude of the shipborne equipment in the camera coordinate system to the terminal 102. In addition, in some embodiments, the shipborne equipment attitude measurement method based on multi-feature points and multi-views can also be implemented by the server 104 or the terminal 102 alone. For example, the terminal 102 can directly process the images of multiple perspectives of the shipborne equipment, or the server 104 can obtain the images of multiple perspectives of the shipborne equipment from the data storage system and process the images of multiple perspectives of the shipborne equipment.

[0028] Among them, the terminal 102 can be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers, and can also be a cloud server.

[0029] In an exemplary embodiment, as Figure 2 shown, a shipborne equipment attitude measurement method based on multi-feature points and multi-views is provided. This method is executed by a computer device, and specifically can be executed alone by a computer device such as a terminal or a server, or jointly executed by a terminal and a server. In the embodiments of the present application, taking this method applied to Figure 1 the server 104 therein as an example for illustration, it includes the following steps 201 to step 206. Among them:

[0030] Step 201, obtain images of the shipborne equipment from multiple perspectives; each of the images contains a number of feature points; one feature point in each of the images represents a point at a feature part of the shipborne equipment; the number of feature points are points at different feature parts of the shipborne equipment.

[0031] Step 202, input the images of each perspective into a trained feature point recognition network model (abbreviated as the model), and predict the pixel coordinates and corresponding confidence levels of each feature point in the images of each perspective.

[0032] Step 203, determine the pixel coordinates of the feature points with a confidence level greater than a preset threshold in the images of each perspective as the pixel coordinates of the valid feature points.

[0033] Step 204, according to the pixel coordinates of the valid feature points corresponding to each feature part and the camera (i.e., the camera) parameters, use the multi-view measurement method to calculate the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the camera coordinate system.

[0034] Step 205, according to the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the camera coordinate system and the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the shipborne equipment body coordinate system, determine the mapping relationship between the camera coordinate system and the shipborne equipment body coordinate system; the mapping relationship includes a rotation matrix and a translation matrix.

[0035] Step 206, according to the mapping relationship between the camera coordinate system and the shipborne equipment body coordinate system, determine the attitude of the shipborne equipment in the camera coordinate system; the attitude includes a pitch angle, a roll angle, and a yaw angle.

[0036] Due to factors such as the image shooting angle, the shapes of shipborne equipment on different images are not the same, making it difficult to select stable line features and regional features. In the structural shape of the equipment, there are many representative feature points, and the shape description based on them is an effective simplification of the equipment shape.

[0037] By taking shipborne equipment from multiple perspectives, image data at different angles is obtained. The image data needs to cover different angles of the equipment, including perspectives such as top view, side view, front view, rear view, etc., to ensure that all feature points can be captured. In some embodiments, the specific requirements for multi-perspective data collection are as follows:

[0038] 1) At least include images from 3 different perspectives.

[0039] 2) In each perspective, the equipment needs to occupy the center of the image to avoid the influence of edge distortion on the detection effect.

[0040] 3) Visibility of feature point distribution:

[0041] Each image should clearly capture the feature points to avoid the loss of feature points due to occlusion or uneven illumination.

[0042] Examples of feature points: the nose, tail, and motor positions (front right, front left, rear right, rear left) of the drone.

[0043] For complex equipment, it should be ensured that all significant features (such as iconic protrusions, mounting holes, etc.) can be visualized.

[0044] 4) Environmental conditions:

[0045] Background complexity: Include complex backgrounds in the shipborne environment (such as the ocean, deck) to simulate actual usage conditions.

[0046] Lighting conditions: Cover various lighting conditions such as strong light, backlight, and shadow to enhance the robustness of the model.

[0047] 5) Requirements for acquisition equipment

[0048] Use a camera with a resolution higher than 1080p to ensure that the feature points are clear enough.

[0049] The camera needs to be calibrated to obtain the internal parameters (focal length, optical axis) and external parameters (pose).

[0050] In another exemplary embodiment of the present application, the feature point recognition network model is an improved MobileNet-v2 neural network; the improved MobileNet-v2 neural network is a MobileNet-v2 neural network with an SE module added; the SE module is used to enhance the global information modeling ability between channels of the MobileNet-v2 neural network.

[0051] Further, the determination process of the improved MobileNet-v2 neural network specifically includes:

[0052] An SE module is added between the first bottleneck layer of the MobileNet-v2 neural network and the previous network layer adjacent to the first bottleneck layer to obtain the first MobileNet-v2 neural network.

[0053] An SE module is added between the last bottleneck layer of the first MobileNet-v2 neural network and the next network layer adjacent to the last bottleneck layer to obtain the second MobileNet-v2 neural network.

[0054] An SE module is added between the global average pooling layer of the second MobileNet-v2 neural network and the previous network layer adjacent to the global average pooling layer to obtain the third MobileNet-v2 neural network;

[0055] A fully connected layer is added after the last network layer of the third MobileNet-v2 neural network to obtain the improved MobileNet-v2 neural network. See Table 1.

[0056] Table 1 Structure of the improved MobileNet-v2 neural network

[0057] Input Operator r c n s 224×224×3 Conv3×3 - 32 1 2 112×112×32 SE 1 32 1 - 112×112×32 Bottleneck 1 16 1 1 56×56×24 Bottleneck 6 32 2 2 28×28×32 Bottleneck 6 64 3 2 14×14×96 Bottleneck 6 160 3 2 7×7×160 Bottleneck 6 320 3 1 7×7×320 SE 1 320 1 - 7×7×320 Conv1×1 - 1280 1 1 7×7×1280 SE 1 1280 1 - 7×7×1280 AvgPool7×7 - - 1 - 1×1×1280 Conv1×1 - 1280 - - 1×1×1280 Linear - 12 - -

[0058] Among them, Input represents the input of each layer; Operator represents the operation of each layer; Conv represents standard convolution; SE represents the Squeeze-and-Excitation module (i.e., the SE module); Bottleneck represents the bottleneck layer; r represents the expansion multiple; c represents the number of output channels; n represents the number of module repetitions; s represents the stride; Avg Pool represents the average filtering convolution operation, i.e., average pooling; Linear represents the linear layer.

[0059] In another exemplary embodiment of the present application, step 202 specifically includes:

[0060] Step 202.1, normalize the multi-view images to a unified size (224×224) and perform data augmentation processing to obtain the normalized image data as the input.

[0061] Step 202.2, input the obtained normalized image data into the improved MobileNet-v2 neural network for processing. This network is based on MobileNet-v2, and inserts SE (Squeeze-and-Excitation) modules before and after its "Bottleneck" layer to perform global average pooling, weight generation, and feature recalibration operations on the features in the channel dimension, realizing global modeling of the features in the channel dimension and improving the network performance by learning global information. The SE module performs the following operations:

[0062] Extract the global information z of the channels from the input feature tensor U through global average pooling (squeeze):

[0063]

[0064] where H and W are the height and width of the feature map respectively.

[0065] Secondly is the excitation operation. Through linear and non-linear transformations, the global compressed features of 1×1×C generate weights of 1×1×C. This step is mainly to obtain the mutual relationship between C channels. The specific formula is as follows:

[0066] s = σ(W 2 δ(W 1 z));

[0067] where z is the global information on C channels obtained through the compression (Squeeze) operation, that is, for a feature tensor U of scale H×W×C, global average pooling compression is performed on the channel dimension C, and the two-dimensional feature is changed into a group of values containing global information; W 1 is the weight generated by the fully connected layer for z, used to explicitly model the relationship between channels. Its dimension is C×C / r, where r is the reduction coefficient used to reduce the number of channels. δ() is the non-linear activation function ReLU, used to increase non-linear relationships. W 2 is the parameter weight generated by another fully connected layer, with a dimension of C×C / r. σ() is the Sigmoid function, used to normalize the weights between channels. The dimension of z is 1×1×C, the dimension of W 1 z is 1×1×C / r, and the dimension of δ(W 1 z) does not change and is still 1×1×C / r, and the dimension of W 2 δ(W 1 z) is still 1×1×C.

[0068] The excitation operation is through the weight coefficients W 1 and W 2And the conversion process of the non-linear activation function ReLU can better model the non-linear relationship between channels, while reducing the number of parameters in the intermediate calculation process, and finally obtaining the weights on the channel features through normalization by the Sigmoid function.

[0069] Then perform feature recalibration to output the enhanced feature tensor.

[0070] The improved MobileNet-v2 network performs depth convolution operations on the input features through linear bottleneck layers and corresponding expansion factors (usually 6), captures more spatial information, reduces redundant features, and directly predicts the pixel coordinates P of the equipment feature points pred and the corresponding confidence level C.

[0071] In some embodiments, step 203 specifically includes: performing confidence filtering on the prediction results, and only retaining the prediction results with confidence level C > 0.9 (i.e., the pixel coordinates of valid feature points). If some feature points do not pass the confidence filtering, they are supplemented according to the prediction results from other perspectives.

[0072] In another exemplary embodiment of the present application, the training process of the feature point recognition network model is as follows:

[0073] Obtain the image set of the shipborne equipment; the image set includes a number of shipborne equipment images from different perspectives and corresponding label data;

[0074] Train the feature point recognition network model according to the image set and the loss function to obtain the trained feature point recognition network model; wherein,

[0075] The loss function is:

[0076]

[0077] wherein, represents the loss function; N represents the number of feature points in the image; represents the prediction data; represents the label data.

[0078] That is to say, during the training process, the regression method is used to calculate the difference between the predicted point P pred and the real point P gt The mean square error is used as the loss function to continuously optimize the feature point recognition network model. After the training is completed, the trained feature point recognition network model can be used to output the optimized feature point prediction results.

[0079] In another exemplary embodiment of the present application, step 204 specifically includes:

[0080] Step 204.1: According to the camera internal parameters, convert the pixel coordinates of the effective feature points corresponding to each feature part in the images of each perspective to obtain the coordinates of the points corresponding to each feature part in each perspective on the normalized plane.

[0081] Specifically, according to the camera internal parameter matrix and the relationship between pixel coordinates and camera internal parameters, convert the pixel coordinates of the effective feature points to points on the normalized plane;

[0082] Among them, the relationship between pixel coordinates and camera internal parameters is:

[0083]

[0084] x i =(x i , y i , 1) is the point coordinate on the normalized plane; (u i , v i ) is the pixel coordinate of the feature point; K is the internal parameter matrix of the camera.

[0085] Step 204.2: According to the coordinates of the points corresponding to each feature part in each perspective on the normalized plane and the camera external parameters, use the multi-perspective triangulation method to determine the values of the width direction and height direction of the three-dimensional coordinates of the feature points of each feature part in the camera coordinate system.

[0086] That is, use the multi-perspective triangulation method, combined with the camera external parameters (pose matrix R, t), to calculate the values of the width direction and height direction of the three-dimensional coordinates of the feature points in the camera coordinate system.

[0087] When using the multi-perspective triangulation algorithm, at least 3 perspective images are required.

[0088] Step 204.3: Determine whether the single-perspective image contains depth information to obtain the first judgment result.

[0089] Step 204.4: When the first judgment result is yes, then according to the depth information corresponding to each feature part in any single-perspective image, determine the value of the depth direction of the three-dimensional coordinates of the feature points of each feature part in the camera coordinate system.

[0090] Step 204.5: When the first judgment result is no, then according to the pixel coordinates of the effective feature points corresponding to the feature part in the images of any two different perspectives and the disparity formula, determine the value of the depth direction of the three-dimensional coordinates of the feature points of the feature part in the camera coordinate system.

[0091] That is to say, if the depth information cannot be directly determined by a single perspective, by matching the corresponding points in other perspectives and using the disparity formula to restore the depth, the disparity formula is as follows:

[0092]

[0093] Among them, f is the camera focal length; b is the baseline length (the distance between the two-view cameras); d is the parallax of the feature point.

[0094] Finally, the three-dimensional coordinates of the feature points are output.

[0095] The obtained three-dimensional coordinate set (i.e., the three-dimensional coordinates of the feature points of each feature part in the camera coordinate system) is expressed as:

[0096] P camera = {(X i , Y i , Z i ) | i = 1, 2,..., N}.

[0097] The three-dimensional coordinates of each feature point are used for subsequent attitude calculation.

[0098] In the attitude calculation step, based on the equipment body coordinate system, the attitude of the equipment is calculated by means of the mapping relationship between the known three-dimensional coordinates of the feature points of the equipment system and the camera coordinate system. The specific steps for establishing the three-dimensional coordinates of the feature points of the equipment system are as follows:

[0099] The equipment body coordinate system is defined as a reference coordinate system fixed on the equipment, and its origin and coordinate axes are determined by the physical characteristics of the equipment to ensure consistency and uniqueness with the equipment structure. The following is the establishment process:

[0100] 1) Select the coordinate system origin

[0101] The origin is usually selected as the geometric center or the functional center point of the equipment. For example: for an unmanned aerial vehicle, the origin can be set as the center of gravity of the fuselage. For a naval gun, the origin can be set at the intersection of the gun barrel and the rotating base.

[0102] 2) Define the coordinate axis directions

[0103] X-axis: Along the forward direction (or functional direction) of the equipment. For example, the nose direction of an unmanned aerial vehicle is defined as the positive X-axis.

[0104] Y-axis: Perpendicular to the X-axis, along the lateral direction of the equipment. For example, the direction from the nose to the right motor of an unmanned aerial vehicle is defined as the positive Y-axis.

[0105] Z-axis: Determined by the right-hand rule from the X-axis and the Y-axis. For an unmanned aerial vehicle, the positive Z-axis points vertically upward from the equipment.

[0106] 3) Determine the three-dimensional coordinates of the feature points

[0107] According to the geometric dimensions of the equipment, calibrate the three-dimensional coordinates of the feature points of each feature part in the body coordinate system:

[0108] Taking a drone as an example:

[0109] Right front motor: (-12, 9, 0) mm

[0110] Left front motor: (12, 9, 0) mm

[0111] Left rear motor: (12, -9, 0) mm

[0112] Right rear motor: (-12, -9, 0) mm

[0113] Nose: (0, 12, 0) mm

[0114] Tail: (0, -7, 0) mm

[0115] For each set of data, 6 pairs of feature points can be obtained. Let the feature points in the body coordinate system be P = {p 1 , p 2 , …, p n}, and the feature points in the camera coordinate system be Q = {q 1 , q 2 , …, q n}, where the feature points are all three-dimensional coordinates and n = 6. Ideally, the attitude of the equipment relative to the camera can be solved by solving the rotation matrix R and translation matrix T of the feature points in the two coordinate systems, that is, there is: q i = Rp i + T.

[0116] In another exemplary embodiment of the present application, the process of determining the mapping relationship between the camera coordinate system and the body coordinate system of the shipborne equipment specifically includes:

[0117] According to the three-dimensional coordinates of the feature points of each characteristic part of the shipborne equipment in the camera coordinate system and the three-dimensional coordinates of the feature points of each characteristic part of the shipborne equipment in the body coordinate system of the shipborne equipment, establish an objective function; the objective function f(R, T) is:

[0118]

[0119] where n represents the number of each characteristic part of the shipborne equipment; p i represents the three-dimensional coordinates of the feature points of the i-th characteristic part of the shipborne equipment in the body coordinate system of the shipborne equipment; q i represents the three-dimensional coordinates of the feature points of the i-th characteristic part of the shipborne equipment in the camera coordinate system; R is the rotation matrix; T is the translation matrix.

[0120] Taking the minimization of the objective function as the goal, use the singular value decomposition method to determine the rotation matrix R.

[0121] Determine the translation matrix \(T\) according to the determined rotation matrix \(R\) and the relational formula between the rotation matrix \(R\) and the translation matrix \(T\).

[0122] In this embodiment, the principle of solving the rotation matrix \(R\) and the translation matrix \(T\) is as follows:

[0123] 1) Due to the existence of various errors and interferences, \(q\) i \(= Rp\) i \(+ T\) does not always hold. Therefore, the least squares method is used to establish the objective function \(f(R, T)\) to obtain the optimal \(R\) and \(T\) that minimize the feature point conversion error. The expression of the objective function is as follows:

[0124]

[0125] Aim to minimize the above objective function.

[0126] 2) The objective function \(f(R, T)\) reaches its minimum value when the partial derivatives with respect to \(R\) and \(T\) are zero.

[0127] Taking the partial derivative of \(f(R, T)\) with respect to \(T\) gives:

[0128]

[0129] Substitute this expression into the objective function expression and simplify to get:

[0130]

[0131] Where,

[0132] 3) Since \(R\) is a rotation matrix, there is \(R\) T \(R = I\), and and are both scalars, and their values remain unchanged before and after transposition. Then there is:

[0133]

[0134] 4) Let the matrix \(A = [a\) 1 \(a\) 2 … \(a\) n , the matrix \(B = [b\) 1 \(b\) 2 … \(b\) n . Denote the trace of a matrix by \(tr(·)\), and the trace of a matrix satisfies the commutation law \(tr(XY)=tr(YX)\). Then there is:

[0135]

[0136] 5) Perform singular value decomposition on \(AB\) T to get Then the formula:

[0137]

[0138] It can be written as:

[0139]

[0140] 6) Among them, U, V, and R are all 3×3 orthogonal matrices, and ∑ is a 3×3 diagonal matrix with the singular values of AB T as the diagonal elements, that is, ∑ = diag(σ 1 , σ 2 , σ 3 ). Let Z = V T RU, then Z is also a 3×3 orthogonal matrix, and the formula:

[0141]

[0142] It can be written as:

[0143]

[0144] 7) The orthogonal matrix satisfies that the sum of the squares of each column element is 1. For the orthogonal matrix Z, then Therefore, it can be known that z ij ≤ 1. Then there is:

[0145]

[0146] When z ii is 1, tr(∑Z) reaches the maximum value (at this time, the objective function is the smallest), that is, the orthogonal matrix Z is the identity matrix, and at this time, the objective function has a solution, then there is:

[0147] Z = V T RU = I.

[0148] Thus, there is: R = VU T .

[0149] Method for calculating the translation vector and rotation matrix based on SVD decomposition:

[0150] Calculating the translation vector and rotation matrix based on the singular value decomposition (SVD) is usually used for least squares problems, such as in tasks like point cloud registration and rigid transformation estimation. Suppose there are two sets of point clouds: the original point cloud P = {p 1 , p 2 ,..., p n} and the target point cloud Q = {q 1 , q 2 ,..., q n}, and we need to transform the original point cloud to the target point cloud through a rigid transformation (translation and rotation). This problem can be solved by SVD.

[0151] Centralized data:

[0152] 1) Calculate the centroids (means) of the two point sets:

[0153]

[0154] 2) Centralize the data by subtracting the centroid of each point set from its data:

[0155]

[0156] The purpose of this is to eliminate the translational component and focus on solving for rotation.

[0157] Calculate the covariance matrix:

[0158] Calculate the covariance matrix H between the two centralized point sets:

[0159] H = P' T Q';

[0160] where P' T is the transpose of P', and Q' is the target point set.

[0161] Singular Value Decomposition (SVD):

[0162] Perform singular value decomposition on the covariance matrix H:

[0163] H = UΣV T ;

[0164] where: U and V are orthogonal matrices; Σ is the singular value matrix (diagonal matrix).

[0165] Calculate the rotation matrix:

[0166] The rotation matrix R can be calculated by the following formula:

[0167] R = VU T ;

[0168] This rotation matrix R rotates the original point set to the direction of the target point set.

[0169] Calculate the translation vector:

[0170] The translation vector t is calculated by the difference between the centroids of the two point sets:

[0171]

[0172] Based on the above analysis, the rotation matrix between the vehicle coordinate system and the camera coordinate system can be solved by the Singular Value Decomposition method (SVD), and then the attitude of the equipment can be obtained.

[0173] Determining the attitude of shipborne equipment in the camera coordinate system according to the rotation matrix and translation matrix in step 206 belongs to conventional technical means with a mature theoretical basis, high efficiency and wide application, so it will not be elaborated here.

[0174] The present application also provides an application scenario, which applies the above-mentioned method for measuring the attitude of shipborne equipment based on multi-feature points and multi-views. Specifically: The method for measuring the attitude of shipborne equipment based on multi-feature points and multi-views provided in this embodiment can be applied to the scenario of measuring the attitude of shipborne equipment. The scenario of measuring the attitude of shipborne equipment includes the link of collecting multi-view images of shipborne equipment and the link of calculating the attitude of shipborne equipment; the multi-view images of shipborne equipment enter the link of calculating the attitude of shipborne equipment from the link of collecting multi-view images of shipborne equipment. The method for measuring the attitude of shipborne equipment based on multi-feature points and multi-views provided in this embodiment belongs to the link of calculating the attitude of shipborne equipment. Specifically for the images of multiple perspectives of shipborne equipment, the images of each perspective are input into the trained feature point recognition network model, and the pixel coordinates and corresponding confidence levels of each feature point in the images of each perspective are predicted; the pixel coordinates of the feature points with a confidence level greater than the preset threshold in the images of each perspective are determined as the pixel coordinates of valid feature points; according to the pixel coordinates of the valid feature points corresponding to each feature part and the camera parameters, using the multi-view measurement method, the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the camera coordinate system are calculated; according to the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the camera coordinate system and the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the body coordinate system of the shipborne equipment, the mapping relationship between the camera coordinate system and the body coordinate system of the shipborne equipment is determined; according to the mapping relationship between the camera coordinate system and the body coordinate system of the shipborne equipment, the attitude of the shipborne equipment in the camera coordinate system is determined.

[0175] A method for measuring the attitude of shipborne equipment based on multi-feature points and multi-views proposed in the present application is used for high-precision dynamic attitude monitoring of shipborne equipment such as unmanned aerial vehicles and naval guns. By using the improved MobileNet-v2 network to extract the feature points of the equipment, the spatial attitude information of the equipment is calculated by using the multi-view measurement and singular value decomposition method (SVD). The goal is to overcome the occlusion problem of the single-view method and improve the stability and measurement accuracy of the system in complex environments.

[0176] Therefore, the method for measuring the attitude of shipborne equipment based on multi-feature points and multi-views proposed in the present application has the following advantages:

[0177] 1) High precision: In a complex environment, the root mean square value of the attitude error is less than 1°, and the positioning accuracy reaches below 0.2°.

[0178] 2) Strong anti-interference ability: It does not rely on GPS and inertial systems, so it is not affected by external environmental interference.

[0179] 3) Multi-view redundant measurement: By taking pictures from multiple perspectives, the problem of feature point occlusion that may be caused by a single perspective is reduced.

[0180] 4) Strong adaptability: The system maintains stable measurement results under changes in light and background.

[0181] 5) Low-cost implementation: The vision system equipment is simple and suitable for the lightweight requirements in shipboard applications.

[0182] In an exemplary embodiment, a computer device is provided. The computer device can be a server or a terminal, and its internal structure diagram can be as Figure 3 shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for images of multiple perspectives of shipboard equipment. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it implements a method for measuring the attitude of shipboard equipment based on multiple feature points and multiple views.

[0183] Those skilled in the art can understand that Figure 3 the structure shown in

[0184] is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0185] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0185] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0186] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0187] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the various embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tapes, floppy disks, flash memories, optical memories, high-density embedded non-volatile memories, resistive random access memories (ReRAMs), magnetoresistive random access memories (MRAMs), ferroelectric random access memories (FRAMs), phase change memories (PCMs), graphene memories, etc. Volatile memories can include random access memory (RAM) or external cache memories, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0188] The databases involved in the various embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the various embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., and are not limited thereto.

[0189] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0190] In this text, specific examples are used to illustrate the principles and implementation manners of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manners and application scopes. To sum up, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A method for measuring the attitude of shipborne equipment based on multiple feature points and multiple views, characterized in that: The shipborne equipment posture measurement method based on multiple feature points and multiple views includes: Acquire images of shipborne equipment from multiple perspectives; each of the images contains a number of feature points; a feature point in each of the images represents a point at a feature part of the shipborne equipment; Input the image of each perspective into the trained feature point recognition network model to predict the pixel coordinates and corresponding confidence of each feature point in the image of each perspective; The pixel coordinates of the feature points in the image of each viewing angle whose confidence is greater than a preset threshold are determined as the pixel coordinates of the valid feature points; According to the pixel coordinates and camera parameters of the effective feature points corresponding to each feature part, the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the camera coordinate system are calculated by using a multi-view measurement method; Determine a mapping relationship between the camera coordinate system and the shipborne equipment body coordinate system according to the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the camera coordinate system and the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the shipborne equipment body coordinate system; the mapping relationship includes a rotation matrix and a translation matrix; According to the mapping relationship between the camera coordinate system and the coordinate system of the shipborne equipment body, the posture of the shipborne equipment in the camera coordinate system is determined; the posture includes a pitch angle, a roll angle and a heading angle.

2. The method for measuring the attitude of shipborne equipment based on multiple feature points and multiple views according to claim 1 is characterized in that: The feature point recognition network model is an improved MobileNet-v2 neural network; the improved MobileNet-v2 neural network is a MobileNet-v2 neural network with an SE module added; The SE module is used to enhance the global information modeling capability between channels of the MobileNet-v2 neural network.

3. The method for measuring the attitude of shipborne equipment based on multiple feature points and multiple views according to claim 2 is characterized in that: The determination process of the improved MobileNet-v2 neural network specifically includes: Adding an SE module between a first bottleneck layer of a MobileNet-v2 neural network and a previous network layer adjacent to the first bottleneck layer to obtain a first MobileNet-v2 neural network; Adding an SE module between the last bottleneck layer of the first MobileNet-v2 neural network and a subsequent network layer adjacent to the last bottleneck layer to obtain a second MobileNet-v2 neural network; Adding an SE module between the global average pooling layer of the second MobileNet-v2 neural network and a previous network layer adjacent to the global average pooling layer to obtain a third MobileNet-v2 neural network; A fully connected layer is added after the last network layer of the third MobileNet-v2 neural network to obtain an improved MobileNet-v2 neural network.

4. The method for measuring the attitude of shipborne equipment based on multiple feature points and multiple views according to claim 1, characterized in that: Determining a mapping relationship between the camera coordinate system and the shipborne equipment body coordinate system according to the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the camera coordinate system and the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the shipborne equipment body coordinate system specifically includes: According to the three-dimensional coordinates of the feature points of each characteristic part of the shipborne equipment in the camera coordinate system and the three-dimensional coordinates of the feature points of each characteristic part of the shipborne equipment in the shipborne equipment body coordinate system, an objective function is established; the objective function f(R, T) is: Wherein, n represents the number of characteristic parts of the shipborne equipment; p i represents the three-dimensional coordinates of the feature point of the ith feature part of the shipborne equipment in the coordinate system of the shipborne equipment body; q i represents the three-dimensional coordinates of the feature point of the i-th feature part of the shipborne equipment in the camera coordinate system; R is the rotation matrix; T is the translation matrix; With the goal of minimizing the objective function, the rotation matrix R is determined using the singular value decomposition method; According to the determined rotation matrix R and the relationship formula between the rotation matrix R and the translation matrix T, the translation matrix T is determined.

5. The method for measuring the attitude of shipborne equipment based on multiple feature points and multiple views according to claim 1, characterized in that: According to the pixel coordinates and camera parameters of the effective feature points corresponding to each feature part in the images of each viewing angle, the three-dimensional coordinates of the feature points of each feature part of the shipborne equipment in the camera coordinate system are calculated by using the multi-view measurement method, specifically including: According to the camera internal parameters, the pixel coordinates of the effective feature points corresponding to each feature part in the image of each viewing angle are converted to obtain the coordinates of the points corresponding to each viewing angle of each feature part on the normalized plane; According to the coordinates of the points corresponding to each viewing angle of each feature part on the normalized plane and the camera external parameters, the values ​​of the width direction and the height direction of the three-dimensional coordinates of the feature point of each feature part in the camera coordinate system are determined by using the multi-view triangulation method; Determine whether the image from a single perspective contains depth information, and obtain a first determination result; When the first judgment result is yes, the value of the depth direction of the three-dimensional coordinates of the feature point of each feature part in the camera coordinate system is determined according to the depth information corresponding to each feature part in the image of any single perspective; When the first judgment result is no, the depth direction value of the three-dimensional coordinates of the feature point of the feature part in the camera coordinate system is determined according to the pixel coordinates and parallax formula of the valid feature points corresponding to the feature part in any two images of different viewing angles.

6. The method for measuring the attitude of shipborne equipment based on multiple feature points and multiple views according to claim 1, characterized in that: The training process of the feature point recognition network model is as follows: Acquire an image set of the shipborne equipment; the image set includes images of the shipborne equipment at several different viewing angles and corresponding label data; According to the image set and the loss function, the feature point recognition network model is trained to obtain a trained feature point recognition network model; wherein, The loss function is: in, represents the loss function; N represents the number of feature points in the image; Represents forecast data; Represents label data.

7. The method for measuring the attitude of shipborne equipment based on multiple feature points and multiple views according to claim 1, characterized in that: The three-dimensional coordinates of the characteristic points of each characteristic part of the shipborne equipment in the coordinate system of the shipborne equipment body are determined according to the geometric dimensions of the shipborne equipment.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for measuring the attitude of shipborne equipment based on multiple feature points and multiple views as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for measuring the attitude of shipborne equipment based on multiple feature points and multiple views described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for measuring the attitude of shipborne equipment based on multiple feature points and multiple views described in any one of claims 1 to 7 is implemented.

Images