Method for generating relative positions of flight support nodes based on 3D reconstruction

Abstract

The present invention provides a method for generating the relative positions of flight support nodes based on 3D reconstruction, which belongs to the technical field of flight support node monitoring. The method includes identifying the 2D key point coordinates of an aircraft; selecting the key point positions of the aircraft's 3D model; using a first objective function to match and optimize the 3D model and 2D image of the aircraft based on the identified 2D key point coordinates and the selected 3D key point positions; adjusting and projecting the aircraft model using a particle swarm optimization algorithm, outputting the overlap and visualization results of the aircraft model and the detection results; optimizing the 3D positions of vehicles at flight support nodes, and completing the generation of the relative positions of flight support nodes. The present invention aims to achieve relative position identification and precise positioning of flight support nodes based on 3D reconstruction technology, thereby overcoming the limitations of relying entirely on image detection methods in complex environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of flight support node monitoring, and in particular relates to a method for generating relative positions of flight support nodes based on 3D reconstruction. Background Art

[0002] In airport operations, flight support nodes are crucial for ensuring flight punctuality, ground service coordination, and overall operational efficiency. With the development of intelligent technology, video recognition-based systems are widely used to monitor flight support nodes. These systems use cameras to capture the dynamics of aircraft and ground service vehicles in real time, automatically identifying key nodes for recording and analysis. However, in real-world operating environments, particularly in rain, fog, low light, or obstructions, the performance of video recognition systems is often significantly affected, leading to inaccuracies in the identification and estimation of flight support nodes.

[0003] In rainy and foggy weather, video footage is easily affected by factors such as water droplets and mist, resulting in blurred images and loss of key details, making it difficult for recognition algorithms to accurately extract the outlines and features of target objects. Furthermore, low-light conditions at night or on cloudy days further exacerbate this problem, making it difficult for the recognition system to effectively distinguish between different vehicles. Furthermore, airports are busy operating environments, with ground service vehicles, baggage trolleys, refueling trucks, and other equipment frequently moving around aircraft. These obstructions can also prevent cameras from fully capturing each object, affecting accurate judgment of flight nodes. Summary of the Invention

[0004] In response to the above-mentioned deficiencies in the prior art, the present invention provides a method for generating the relative positions of flight support nodes based on 3D reconstruction. The present invention aims to realize the relative position identification and precise positioning of flight support nodes based on 3D reconstruction technology, overcome the limitations of relying entirely on image detection methods in complex environments, improve the detection accuracy in complex environments, and enhance the efficiency of airport support services. At the same time, based on the optimization algorithm, high-precision alignment of aircraft key points is achieved, providing a more accurate benchmark for vehicle position estimation.

[0005] In order to achieve the above objectives, the present invention adopts a technical solution: a method for generating relative positions of flight support nodes based on 3D reconstruction, comprising the following steps:

[0006] S1, identify the 2D key point coordinates of the aircraft;

[0007] S2. Select key points of the 3D model of the aircraft;

[0008] S3. Based on the selected 3D key point positions, the first objective function is used to perform matching optimization on the 3D model and the 2D image of the aircraft to obtain a matching optimization result;

[0009] S4. Based on the 2D key point coordinates and matching optimization results of the aircraft, the aircraft model is adjusted and projected using the particle swarm optimization algorithm;

[0010] S5. Using the adjustment results and projection results of the aircraft model as reference benchmarks, the 3D position of the vehicle at the flight support node is optimized and identified, and the relative position of the flight support node is generated.

[0011] Furthermore, the S1 is specifically:

[0012] S101, marking the key point positions of the aircraft and generating data labels in the Yolov7 model annotation format;

[0013] S102. Use the trained Yolov7 model to identify the data labels, obtain the coordinates of each part of the aircraft, and return a list of information containing each detection box;

[0014] S103 , using the key coordinate derivation code to extract the coordinates of each 2D key point of the aircraft from the information list of each detection frame.

[0015] Furthermore, the S2 is specifically:

[0016] Load the 3D model of the aircraft and import it into the Blender software environment, then select the key points of the 3D model of the aircraft.

[0017] Furthermore, the S3 is specifically as follows:

[0018] S301, based on the selected 3D key point positions, expand the 3D model points into a homogeneous coordinate form, and apply a transformation matrix composed of a rotation matrix, a translation matrix, and a scaling matrix for transformation processing;

[0019] S302, applying a perspective projection matrix to the transformed 3D model points to project them from the three-dimensional space to a two-dimensional plane;

[0020] S303, converting the projected point from the normalized device coordinates to the screen coordinates of the image to map from the 3D model to the 2D image;

[0021] S304: For each key point position of the 3D model, calculate the weighted Euclidean distance between its projection coordinates and the corresponding 2D image key point using the first objective function, and quantify the registration error using the sum of the weighted Euclidean distances to obtain a matching optimization result.

[0022] Furthermore, the calculation process of the first objective function is as follows:

[0023] A1. Expand the key points of the 3D model into homogeneous coordinate form and apply a transformation matrix composed of rotation, translation rotation and scaling matrices to transform the key points of the 3D model;

[0024] A2. Apply a perspective projection matrix to the transformed 3D model key points to project them from 3D space onto a 2D plane, where the perspective projection parameters include field of view angle, aspect ratio, near plane distance, and far plane distance;

[0025] A3. Convert the projected key points from normalized device coordinates to the screen coordinates of the image, completing the mapping from the 3D model to the 2D image.

[0026] A4. For each 3D model key point, calculate the weighted Euclidean distance between its projected coordinates and the corresponding 2D image key point, where the first objective function is to take the sum of the weighted errors of all key points as the optimization target value. The optimization of the first objective function is to minimize the first objective function by finding the optimal parameter combination in the search space of rotation, translation, and scaling parameters.

[0027] Furthermore, the S4 is specifically:

[0028] S401, taking the 2D key point coordinates of the aircraft as the target 2D point;

[0029] S402: construct a 3D model point array based on the 3D key point position, and select the 3D model point array corresponding to the identifier in the 2D key point coordinate as the 3D model point array;

[0030] S403, setting a parameter range for a particle swarm optimization algorithm, wherein, in the particle swarm optimization algorithm, the 3D model points of the aircraft, the 2D points of the target aircraft, and the current 2D image size are passed into a second objective function to minimize the matching optimization result in S3 as the optimization goal;

[0031] S404, in each iteration, updating the particle velocity and position, and optimizing the particle position and velocity to complete the optimization of the particle swarm objective function;

[0032] S405, performing 3D model transformation and projection processing based on the optimization result of the particle swarm objective function;

[0033] S406: Based on the transformation and projection processing results, the 2D projection coordinates of the 3D model vertices are transmitted back to the detection file as the results, and the overlap and visualization results of the aircraft model and the detection results are output, completing the adjustment and projection processing of the aircraft model.

[0034] Furthermore, the calculation process of the second objective function is as follows:

[0035] B1. Obtain optimization parameters;

[0036] B2. Based on the optimized parameters, the transformation matrix is ​​obtained by constructing the rotation matrix, translation matrix, and scaling matrix;

[0037] B3. Construct a perspective projection matrix by setting perspective projection parameters, perform homogeneous coordinate expansion on each 3D model point, apply the transformation matrix and perspective projection matrix, and project the point into 2D space through perspective division.

[0038] B4. Convert the projection result from standard device coordinates to screen coordinates;

[0039] B5. Based on the transformation results, calculate the weighted Euclidean distance between each projected point and the corresponding target 2D point;

[0040] B6. Based on the weighted Euclidean distance, calculate a second objective function, where the second objective function is a weighted sum of all point errors.

[0041] Furthermore, the step S405 is specifically as follows:

[0042] C1. Based on the optimization results of the particle swarm objective function, the optimal parameters are used to construct the transformation matrix and perspective projection matrix. The transformation matrix includes the rotation matrix, translation matrix and scaling matrix.

[0043] C2. Load the 3D model file of the aircraft and obtain the model vertex data;

[0044] C3. For each vertex, expand it into homogeneous coordinate form, apply the transformation matrix and perspective projection matrix in sequence, perform perspective division, and convert the coordinates into screen coordinates to obtain the 2D projection coordinates of the 3D model vertex, completing the transformation and projection processing of the 3D model.

[0045] Furthermore, the S5 is specifically as follows:

[0046] S501, obtaining 2D image coordinate data and height data of the support vehicle, and annotating the data;

[0047] S502, measuring the error between the 3D projection point and the 2D key point using weighted Euclidean distance, and simulating the height difference between the sphere of the support vehicle and the support vehicle in the image, wherein the height difference data is used as one of the optimization targets to obtain the vertices of the 3D model;

[0048] S503: Minimize the error to match the 2D image of the support vehicle with the 3D model;

[0049] S504, setting and optimizing the particle swarm parameters of the relative positions of the flight support nodes;

[0050] S505. Based on the optimal parameters, a rotation moment, a translation moment, and a scaling matrix are constructed to generate a transformation matrix for the support vehicle. In the particle swarm position optimization algorithm, the 3D model points of the support vehicle, the 2D points of the target vehicle, and the 2D image size of the current support vehicle are input into the third objective function. The 3D model vertices are obtained by minimizing the difference between the third objective function and the height of the support vehicle sphere in the image and the support vehicle as the optimization target.

[0051] S506. Expand the vertices of the support vehicle 3D model into homogeneous coordinate form, apply the transformation matrix and the projection matrix in sequence, and convert the calculated results from normalized device coordinates to screen coordinates of the image through perspective division to obtain the 2D projection coordinates of the vertices of the support vehicle spherical model;

[0052] S507, using the adjustment result and the projection result of the aircraft model as a reference benchmark;

[0053] S508. Based on the reference benchmark, the 2D projection coordinates of the vertices of the support vehicle spherical model are plotted on the input image, and the matching results of the projection of the support vehicle spherical 3D model and the key points of the 2D image are displayed to complete the generation of the relative positions of the flight support nodes.

[0054] The beneficial effects of the present invention are:

[0055] (1) This invention aims to achieve relative position recognition and precise positioning of flight support nodes based on 3D reconstruction technology, overcoming the limitations of image-based detection methods in complex environments, such as rainy and foggy weather, insufficient light, or obstructions that can lead to recognition errors. By combining 2D image detection results with 3D model matching optimization, the accuracy and efficiency of support node detection can be improved, thereby providing more reliable technical support for airport ground services and ensuring the stability and efficiency of flight operations.

[0056] (2) This invention can improve detection accuracy in complex environments: In rain, fog, low light, or occlusion, traditional video recognition methods have difficulty accurately capturing the location of aircraft and surrounding support nodes. This invention improves the recognition capability of support nodes in complex environments by integrating and optimizing 3D models with 2D detection results.

[0057] (3) The present invention overcomes the limitations of relying solely on images: relying solely on 2D image detection cannot accurately estimate the three-dimensional relative position and spatial relationship of objects. By accurately reconstructing and optimizing the matching of 3D models, the present invention can achieve reliable estimation of the relative position of objects, thus compensating for the shortcomings of image detection methods.

[0058] (4) The present invention improves matching accuracy based on an optimization algorithm: The present invention minimizes the error between the 3D model projection and the 2D detection points through the particle swarm optimization (PSO) algorithm, thereby achieving high-precision registration of aircraft key points and providing a more accurate benchmark for ensuring vehicle position estimation.

[0059] (5) The present invention improves the efficiency of airport support services: Through automated identification and location matching, the present invention reduces the need for manual intervention, improves the identification efficiency of flight support nodes, and provides technical support for the intelligent and precise operation of airports. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 Flow chart of the method of the present invention.

[0061] Figure 2 A schematic diagram of the key point coordinate data for aircraft recognition based on YOLOV7.

[0062] Figure 3 Schematic diagram of the key point positions for selecting the aircraft 3D model.

[0063] Figure 4 Schematic diagram of aircraft model adjustment and projection for particle swarm optimization algorithm (PSO).

[0064] Figure 5 Schematic diagram for optimizing identification of 3D position of node vehicles.

[0065] Figure 6 Schematic diagram of the simulated sphere. DETAILED DESCRIPTION

[0066] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0067] Example

[0068] like Figure 1 As shown, the present invention provides a method for generating relative positions of flight support nodes based on 3D reconstruction, and its implementation method is as follows:

[0069] S1. Identify the 2D key point coordinates of the aircraft. The implementation method is as follows:

[0070] S101, marking the key point positions of the aircraft and generating data labels in the Yolov7 model annotation format;

[0071] S102. Use the trained Yolov7 model to identify the data labels, obtain the coordinates of each part of the aircraft, and return a list of information containing each detection box;

[0072] S103 , using the key coordinate derivation code to extract the coordinates of each 2D key point of the aircraft from the information list of each detection frame.

[0073] In this embodiment, Figure 2 As shown in the figure, the specific process of identifying the coordinates of the aircraft key points based on the Yolov7 model is as follows:

[0074] 1. Data annotation:

[0075] During the data annotation process, the Yolov7 model aims to detect the aircraft's position in the input image and identify key aircraft components. These components include the nose, tail, center of tail, upper tail, and wings. Key components are manually annotated on each image, generating dataset labels in the Yolov7 model's annotation format for subsequent model learning and recognition.

[0076] 2. Identification:

[0077] During the Yolov7 model recognition phase, the trained Yolov7 model is used to predict the image and obtain the coordinates of each part of the aircraft. The Yolov7 model returns a list of information about each detection box, each of which contains the following fields:

[0078] Class ID: A unique identifier for each class that distinguishes different aircraft parts (e.g., Class 0 represents the main body of the aircraft).

[0079] X, Y: X and Y coordinates of the upper left corner of the detection box.

[0080] W, H: width and height of the detection box.

[0081] CLS: confidence of the detection box.

[0082] The output of the Yolov7 model is post-processed to filter out the coordinates of key parts, and the center position of the detection box is further calculated. It is also converted into a representation using center point, width, and height. Subsequently, the largest detection box that meets the requirements is selected based on area as the key part.

[0083] 3. Derivation of key coordinates:

[0084] The key coordinate derivation code is responsible for extracting and calculating the precise position of each key part from the detection frame data. The code structure is as follows:

[0085] Based on the category and area information of the detection frame, the coordinates of key parts that meet the following conditions are screened out from multiple detection frames: aircraft head, aircraft tail, tail center, top of the tail, and wings.

[0086] Aircraft Head: Select the instance with the largest area in the category Aircraft Head.

[0087] Aircraft Tail: Select the category Aircraft Tail and it should be within the scope of the aircraft body.

[0088] Above the tail: Select the instance with the largest area in the category Above the tail.

[0089] Wing: Select the Wing category, which must be the largest width and within the aircraft body.

[0090] Aircraft body: Select the instance with the largest area and the category of aircraft.

[0091] Correct the tail tip coordinates and correct the detected tail tip coordinates to make the position of the tail top point more accurate.

[0092] If the detection lacks parts such as the tail or the part above the tail, it will be supplemented based on the existing information to ensure that all key parts can be identified.

[0093] Based on the detected relative positions of the aircraft and its wings, nose, and tail, the left and right positions of the wings are determined and the corresponding coordinates are adjusted.

[0094] S2. Select the key points of the 3D model of the aircraft, specifically:

[0095] Load the 3D model of the aircraft and import it into the Blender software environment, then select the key points of the 3D model of the aircraft.

[0096] In this embodiment, Figure 3 As shown, the key positions of the aircraft based on the Blender software are selected as follows:

[0097] Use Blender to import a precise 3D model of the target aircraft into the Blender environment. Ensure the 3D model is loaded completely, without errors or missing parts, to ensure accurate key location selection. Enter model editing mode and begin selecting key locations within the aircraft's 3D model through manual interaction. Locate the distinctive features of the nose, left and right wings, the center of the tail, and the upper portion of the tail. These locations are selected based on the aircraft's overall structure and appearance. Its unique shape and design characteristics can be accurately identified through its contours and details.

[0098] S3. Based on the selected 3D key point positions, the first objective function is used to perform matching optimization on the 3D model and 2D image of the aircraft to obtain a matching optimization result. The implementation method is as follows:

[0099] S301, based on the selected 3D key point positions, expand the 3D model points into a homogeneous coordinate form, and apply a transformation matrix composed of a rotation matrix, a translation matrix, and a scaling matrix for transformation processing;

[0100] S302, applying a perspective projection matrix to the transformed 3D model points to project them from the three-dimensional space to a two-dimensional plane;

[0101] S303, converting the projected point from the normalized device coordinates to the screen coordinates of the image to map from the 3D model to the 2D image;

[0102] S304: For each key point position of the 3D model, a weighted Euclidean distance between its projected coordinates and the corresponding key point of the 2D image is calculated using a first objective function, and the sum of the weighted Euclidean distances is used to quantify the registration error to obtain a matching optimization result. The calculation process of the first objective function is as follows:

[0103] A1. Expand the key points of the 3D model into homogeneous coordinate form and apply a transformation matrix composed of rotation, translation rotation and scaling matrices to transform the key points of the 3D model;

[0104] A2. Apply a perspective projection matrix to the transformed 3D model key points to project them from 3D space onto a 2D plane, where the perspective projection parameters include field of view angle, aspect ratio, near plane distance, and far plane distance;

[0105] A3. Convert the projected key points from normalized device coordinates to the screen coordinates of the image, completing the mapping from the 3D model to the 2D image.

[0106] A4. For each 3D model key point, calculate the weighted Euclidean distance between its projected coordinates and the corresponding 2D image key point, where the first objective function is to take the sum of the weighted errors of all key points as the optimization target value. The optimization of the first objective function is to minimize the first objective function by finding the optimal parameter combination in the search space of rotation, translation, and scaling parameters.

[0107] In this embodiment, different rotation, translation, and scaling parameters will correspond to different matching optimization results after application. The matching optimization results are actually the best rotation, translation, and scaling parameters found for S4 in the particle swarm optimization algorithm. These optimized parameters will serve as the basis for application in subsequent steps.

[0108] In this embodiment, Figure 4As shown in the figure, the specific process of matching and optimizing the 3D model of the aircraft with the 2D image is as follows:

[0109] 1. Data input:

[0110] The input data consists of two parts: 1. The coordinates of 2D image keypoints of key aircraft parts, identified and extracted from the image using the Yolov7 model. These coordinates cover locations such as the nose, tail, center of the tail, top of the tail, and wings. These coordinates provide information about the aircraft's position and layout in the 2D image. 2. The coordinates of 3D model keypoints of key locations manually annotated on the 3D aircraft model imported into Blender software. These points correspond to aircraft parts in the 2D image, providing geometric information about the aircraft in 3D space.

[0111] 2. Maximize matching accuracy:

[0112] The optimization goal of the present invention is to maximize matching accuracy. This means adjusting the 3D model's spatial transformation parameters, such as rotation, translation, and scaling, so that the 3D model, after perspective projection, is as consistent as possible with the coordinates of key points in the 2D image. To this end, an objective function is defined. This first objective function quantifies the registration error using the sum of the weighted Euclidean distances between each projection point and the image key points. By minimizing this registration error, the particle swarm optimization algorithm (PSO) is able to find the optimal spatial transformation parameters, resulting in a high degree of consistency between the model and the aircraft's position and attitude in the image.

[0113] The optimization parameters include rotation angles (around the X, Y, and Z axes), which are used to adjust the orientation of the 3D model to match the angle in the 2D image; translation parameters (along the X, Y, and Z axes), which are used to adjust the position of the model so that its center is at the target position in the image; and scaling factors, which are used to adjust the size of the model to match the proportions of the object in the image.

[0114] 3. First objective function:

[0115] The calculation process of the first objective function is relatively complex. First, the 3D model key points are expanded into homogeneous coordinate form, and a transformation matrix composed of rotation, translation, and scaling matrices is applied. Then, the perspective projection matrix is ​​applied to the transformed 3D model key points to project them from three-dimensional space to a two-dimensional plane. The perspective projection parameters include the field of view angle, aspect ratio, near plane distance, and far plane distance to ensure that the 3D model projection conforms to the image perspective. Next, the projected key points are converted from normalized device coordinates (NDC) to the screen coordinates of the image to complete the mapping from the 3D model to the 2D image. Finally, for each 3D model key point, the weighted Euclidean distance between its projection coordinates and the corresponding 2D image key point is calculated. The weight is set according to the importance of the point (such as the nose, wing, etc.). The first objective function uses the sum of the weighted errors of all key points as the optimization target value.

[0116] To optimize the first objective function, a particle swarm optimization (PSO) algorithm is employed. This algorithm minimizes the first objective function by searching for the optimal parameter combination within a search space of rotation, translation, and scaling parameters. Each particle represents a possible parameter combination, and the particle iteratively updates its position within the search space, gradually approaching the position with the minimum error. When the PSO algorithm meets certain convergence conditions or reaches the maximum number of iterations, it outputs the optimal parameter combination. This parameter combination can then be used to accurately project the model onto the image, achieving matching between model keypoints and image keypoints.

[0117] S4. Based on the 2D key point coordinates of the aircraft and the matching optimization results, the aircraft model is adjusted and projected using the particle swarm optimization algorithm. The implementation method is as follows:

[0118] S401, taking the 2D key point coordinates of the aircraft as the target 2D point;

[0119] S402: construct a 3D model point array based on the 3D key point position, and select the 3D model point array corresponding to the identifier in the 2D key point coordinate as the 3D model point array;

[0120] S403, setting a parameter range for a particle swarm optimization algorithm, wherein, in the particle swarm optimization algorithm, the 3D model points of the aircraft, the 2D points of the target aircraft, and the current 2D image size are passed into a second objective function to minimize the matching optimization result in S3 as the optimization goal;

[0121] The calculation process of the second objective function is as follows:

[0122] B1. Obtain optimization parameters;

[0123] B2. Based on the optimized parameters, the transformation matrix is ​​obtained by constructing the rotation matrix, translation matrix, and scaling matrix;

[0124] B3. Construct a perspective projection matrix by setting perspective projection parameters, perform homogeneous coordinate expansion on each 3D model point, apply the transformation matrix and perspective projection matrix, and project the point into 2D space through perspective division.

[0125] B4. Convert the projection result from standard device coordinates to screen coordinates;

[0126] B5. Based on the transformation results, calculate the weighted Euclidean distance between each projected point and the corresponding target 2D point;

[0127] B6. Calculate a second objective function based on the weighted Euclidean distance, where the second objective function is a weighted sum of all point errors;

[0128] S404, in each iteration, updating the particle velocity and position, and optimizing the particle position and velocity to complete the optimization of the particle swarm objective function;

[0129] S405: Based on the optimization result of the particle swarm objective function, 3D model transformation and projection processing are performed, which are specifically as follows:

[0130] C1. Based on the optimization results of the particle swarm objective function, the optimal parameters are used to construct the transformation matrix and perspective projection matrix. The transformation matrix includes the rotation matrix, translation matrix and scaling matrix.

[0131] C2. Load the 3D model file of the aircraft and obtain the model vertex data;

[0132] C3. For each vertex, expand it into homogeneous coordinate form, apply the transformation matrix and perspective projection matrix in sequence, perform perspective division, and convert the coordinates into screen coordinates to obtain the 2D projection coordinates of the 3D model vertex, completing the transformation and projection processing of the 3D model;

[0133] S406: Based on the transformation and projection processing results, the 2D projection coordinates of the 3D model vertices are transmitted back to the detection file as the results, and the overlap and visualization results of the aircraft model and the detection results are output, completing the adjustment and projection processing of the aircraft model.

[0134] In this embodiment, the adjustment and projection results of the aircraft model in S4 provide S5 with the aircraft's precise position and attitude information as a reference benchmark for clarifying the relative position of the support vehicle. The matrix construction and coordinate transformation methods used also provide an example for processing the support vehicle's 3D model to generate the support vehicle's 2D projection coordinates and complete the generation of the relative position of the flight support node.

[0135] In this embodiment, the aircraft model adjustment and projection process based on the particle swarm optimization algorithm is as follows:

[0136] 1. Data preparation:

[0137] Receives input image data and keypoint data detected by the Yolov7 model. These keypoint data represent the specific location information of the aircraft in the image and serve as target 2D points.

[0138] Receive the coordinate data of the corresponding key points obtained by loading the 3D model of the aircraft using Blender software and create a 3D model point array. Based on the specific identifiers in the input key point data (for example, the identifier used to distinguish the left and right wings), select the corresponding 3D model point array.

[0139] 2. Setting and optimizing the parameters of the particle swarm optimization algorithm:

[0140] Set the parameter range of the particle swarm optimization algorithm, the lower bound Set to , corresponding to the lower limit of rotation angle X, rotation angle Y, rotation angle Z, translation X axis, translation Y axis, translation Z axis and scaling factor. Set to , corresponding to the upper limits of the above parameters respectively.

[0141] In the particle swarm optimization algorithm optimization function, the aircraft's 3D model points, the target aircraft's 2D points, the current 2D image size, etc. are passed to the second objective function, with minimizing the key point error as the optimization goal. The calculation process of the second objective function is as follows:

[0142] (1) Receive optimization parameters: including rotation angle (X, Y, Z), translation vector ( , , ) and the scaling factor s.

[0143] (2) Construct the transformation matrix: Rotation matrix: The rotation matrices around the X, Y, and Z axes are R x ,R y ,R z ; Translation matrix: Translation matrix T represents translation in the X, Y, and Z directions; Scaling matrix: Scaling matrix S is used for uniform scaling; Combining these matrices forms the total transformation matrix M=T×R x ×R y ×R z ×S.

[0144] (3) Projection operation: Set the perspective projection parameters, including the field of view angle, aspect ratio, and the distances to the near and far planes, to construct the perspective projection matrix P. Perform homogeneous coordinate expansion on each 3D model point, applying the transformation matrix and the projection matrix P × M. Project the point into 2D space through perspective division, and convert the result from normalized device coordinates (NDC) to screen coordinates.

[0145] (4) Error calculation: Calculate the weighted Euclidean distance between each projection point and the corresponding target 2D point :

[0146]

[0147] in, Represents the weight of the corresponding point, which is set according to the importance of different key points. Represents the x-axis coordinate of the projection point, Indicates the x-axis coordinate of the target 2D point, Represents the y-axis coordinate of the projection point, Indicates the y-axis coordinate of the target 2D point.

[0148] The total second objective function is the weighted sum of all point errors :

[0149]

[0150] Among them, i represents different key points, and n represents the four key points of the aircraft head, aircraft tail, above the tail, and wings.

[0151] (5) Particle swarm optimization algorithm iterative update: In each iteration, the particle velocity and position are updated, and the particle position and velocity are optimized using the formula:

[0152]

[0153] in, Indicates the The velocity of the j-th dimension of a particle in generation t+1, represents the inertia coefficient, and represent the individual and group acceleration constants, and Represents a random number in the interval [0, 1], which is randomly generated at each iteration. Indicates the The historical optimal position of the j-th dimension of a particle, Indicates the The position of the j-th dimension of a particle in generation t, Indicates the The global optimal position of a particle in the jth dimension.

[0154] 3. The process of model transformation and projection is as follows:

[0155] Using the optimal parameters, the rotation, translation, scaling, and perspective projection matrices are constructed using the same process as in the second objective function. The aircraft's 3D model file is loaded to obtain the 3D model vertex data. For each vertex, the data is expanded to homogeneous coordinates. The transformation matrix and perspective projection matrix are then applied in sequence, followed by perspective division and conversion to screen coordinates. This ultimately yields the 2D projection coordinates of the model vertex. These operations are identical to those performed on a single point in the objective function, but are applied to all model vertices.

[0156] In this embodiment, the result output and application process are as follows:

[0157] The 2D projection coordinates of the model's vertices are returned to the detection file as the result, allowing the 2D keypoints to be printed on the image. The detection file can use an image rendering library (for example, OpenCV) to draw the 2D keypoints on the input image, overlaying and visualizing the aircraft model with the detection results. This allows for an intuitive display of the aircraft model's position and pose in the image, as well as their correspondence with the detected keypoints.

[0158] S5. Using the aircraft model adjustment results and projection results as reference benchmarks, optimize the identification of the 3D position of the vehicle at the flight support node and complete the generation of the relative position of the flight support node. The implementation method is as follows:

[0159] S501, obtaining 2D image coordinate data and height data of the support vehicle, and annotating the data;

[0160] S502, measuring the error between the 3D projection point and the 2D key point using weighted Euclidean distance, and simulating the height difference between the sphere of the support vehicle and the support vehicle in the image, wherein the height difference data is used as one of the optimization targets to obtain the vertices of the 3D model;

[0161] S503: Minimize the error to match the 2D image of the support vehicle with the 3D model;

[0162] S504, setting and optimizing the particle swarm parameters of the relative positions of the flight support nodes;

[0163] S505. Based on the optimal parameters, a rotation moment, a translation moment, and a scaling matrix are constructed to generate a transformation matrix for the support vehicle. In the particle swarm position optimization algorithm, the 3D model points of the support vehicle, the 2D points of the target vehicle, and the 2D image size of the current support vehicle are input into the third objective function. The 3D model vertices are obtained by minimizing the difference between the third objective function and the height of the support vehicle sphere in the image and the support vehicle as the optimization target.

[0164] S506. Expand the vertices of the support vehicle 3D model into homogeneous coordinate form, apply the transformation matrix and the projection matrix in sequence, and convert the calculated results from normalized device coordinates to screen coordinates of the image through perspective division to obtain the 2D projection coordinates of the vertices of the support vehicle spherical model;

[0165] S507, using the adjustment result and the projection result of the aircraft model as a reference benchmark;

[0166] S508. Based on the reference benchmark, the 2D projection coordinates of the vertices of the support vehicle spherical model are plotted on the input image, and the matching results of the projection of the support vehicle spherical 3D model and the key points of the 2D image are displayed to complete the generation of the relative positions of the flight support nodes.

[0167] In this embodiment, Figure 5 As shown in the figure, the process of optimizing and identifying the 3D position of the guarantee node vehicle is as follows:

[0168] 1. Data input and annotation:

[0169] Data input: Input the 2D image coordinate data and height data of the support vehicle in the image. The coordinates and height data are generated by the Yolov7 model detection. The coordinates are the center coordinates of the support vehicle position box.

[0170] Data annotation: During the data annotation process, ensure that the positions of the security vehicles are marked on the images at different locations for use in subsequent optimization steps.

[0171] 2. Problem definition:

[0172] After perspective projection of the support vehicle's 3D model, its keypoint coordinates on the 2D plane are aligned as closely as possible with those in the image. Weighted Euclidean distance is used to measure the error between the 3D projected points and the 2D keypoints, as well as the height difference between the sphere simulating the support vehicle and the support vehicle in the image. By minimizing this error, the 2D image and 3D model are matched.

[0173] like Figure 6 As shown, Figure 6 The left image in the middle shows the simulated sphere with a 2D background image, while the right image shows the simulated sphere without the background image. The parameters to be optimized are the center coordinates (x, y, z) of the simulated sphere. Among multiple spheres of the same size, within the algorithm's constraints, only sphere 1's center coordinates, after projecting, coincide with the 2D coordinates of the luggage cart's center point, and its diameter is consistent with the height of the luggage cart. Spheres 2 and 3 are projected at different locations. Their projected 2D coordinates and diameters vary, with spheres closer to the viewing angle having larger diameters and those farther away having smaller diameters.

[0174] 3. The third objective function

[0175] The calculation logic of the third objective function is to construct a rotation matrix for the transformation matrix, and construct a rotation matrix R around the X, Y, and Z axes x ,R y , R z , the translation matrix T defines the translation along the X, Y, and Z axes, and the scaling matrix S is used to adjust the overall scale of the model. The resulting transformation matrix M = T * R x * R y * R z * S represents the position and pose of the vehicle model in 3D space. Set the perspective projection parameters, including viewing angle, aspect ratio, and near and far plane distances. Apply the perspective projection matrix P to transform the 3D model points onto a 2D plane.

[0176] The weighted Euclidean distance between the center point of the model on the 2D plane and the corresponding 2D key point in the image and the difference between the height of the 2D plane model and the height of the support vehicle are calculated. The third objective function is the sum of the weighted errors of the two, which represents the matching degree of the model position.

[0177] 4. Particle Swarm Optimization (PSO) parameter setting and optimization

[0178] Particle Swarm Optimization Parameter Range: Lower Bound and upper bound The definition of is based on the size of the aircraft model, with a lower bound of (-50, 0.0, -52.0) and an upper bound of (50.0, 37.0, 47.0).

[0179] In particle swarm optimization, the particle swarm is initialized, and each particle represents a parameter combination (rotation angle, translation amount, and scaling factor). The parameter combination that minimizes the third objective function is found by continuously updating the position and velocity of the particles. Using the inertia weight ω, individual acceleration constant and the group acceleration constant To adjust the particle update process to ensure optimization convergence.

[0180] 5. Ensure that the vehicle sphere 3D model transformation and projection process is as follows:

[0181] Based on the optimal parameters of the particle swarm, the rotation, translation, and scaling matrices are constructed to generate the transformation matrix M of the support vehicle. The vertices of the support vehicle's 3D model are expanded into homogeneous coordinate form, and the transformation matrix and projection matrix are applied in sequence. After perspective division, the result is converted from normalized device coordinates (NDC) to image screen coordinates to obtain the 2D projection coordinates of the vertices of the support vehicle's spherical model.

[0182] 6. The result output and application process are as follows:

[0183] Draw the 2D projection points of the support vehicle onto the input image to visually demonstrate the matching effect between the projection of the support vehicle's spherical 3D model and the key points of the 2D image. Use a drawing library (such as OpenCV) to visualize the key points on the image to verify the accuracy of the optimization results.

[0184] After completing these steps, the 3D position of the support vehicle will be optimized and matched through the 2D information in the image, and the 3D position of the support vehicle can be effectively output, achieving the ultimate goal of position recognition to ensure the 3D relative position recognition of the vehicle luggage cart.

Claims

1. A method for generating relative positions of flight support nodes based on 3D reconstruction, characterized in that: The following steps are involved: S1, identify the 2D key point coordinates of the aircraft; S2. Select key points of the 3D model of the aircraft; S3. Based on the selected 3D key point positions, the first objective function is used to perform matching optimization on the 3D model and the 2D image of the aircraft to obtain a matching optimization result; S4. Based on the 2D key point coordinates and matching optimization results of the aircraft, the aircraft model is adjusted and projected using the particle swarm optimization algorithm; S5. Using the aircraft model adjustment results and projection results as reference benchmarks, the 3D positions of the vehicles at the flight support nodes are optimized and identified, and the relative positions of the flight support nodes are generated. The S5 is specifically: S501, obtaining 2D image coordinate data and height data of the support vehicle, and annotating the data; S502, measuring the error between the 3D projection point and the 2D key point using weighted Euclidean distance, and simulating the height difference between the sphere of the support vehicle and the support vehicle in the image, wherein the height difference data is used as one of the optimization targets to obtain the vertices of the 3D model; S503: Minimize the error to match the 2D image of the support vehicle with the 3D model; S504, setting and optimizing the particle swarm parameters of the relative positions of the flight support nodes; S505. Based on the optimal parameters, a rotation moment, a translation moment, and a scaling matrix are constructed to generate a transformation matrix for the support vehicle. In the particle swarm position optimization algorithm, the 3D model points of the support vehicle, the 2D points of the target vehicle, and the 2D image size of the current support vehicle are input into the third objective function. The 3D model vertices are obtained by minimizing the difference between the third objective function and the height of the support vehicle sphere in the image and the support vehicle as the optimization target. S506. Expand the vertices of the support vehicle 3D model into homogeneous coordinate form, apply the transformation matrix and the projection matrix in sequence, and convert the calculated results from normalized device coordinates to screen coordinates of the image through perspective division to obtain the 2D projection coordinates of the vertices of the support vehicle spherical model; S507, using the adjustment result and the projection result of the aircraft model as a reference benchmark; S508. Based on the reference benchmark, the 2D projection coordinates of the vertices of the support vehicle spherical model are plotted on the input image, and the matching results of the projection of the support vehicle spherical 3D model and the key points of the 2D image are displayed to complete the generation of the relative positions of the flight support nodes.

2. The method for generating relative positions of flight support nodes based on 3D reconstruction according to claim 1, characterized in that: The S1 is specifically: S101, marking the key point positions of the aircraft and generating data labels in the Yolov7 model annotation format; S102. Use the trained Yolov7 model to identify the data labels, obtain the coordinates of each part of the aircraft, and return a list of information containing each detection box; S103 , using the key coordinate derivation code to extract the coordinates of each 2D key point of the aircraft from the information list of each detection frame.

3. The method for generating relative positions of flight support nodes based on 3D reconstruction according to claim 1, characterized in that: The S2 is specifically: Load the 3D model of the aircraft and import it into the Blender software environment, then select the key points of the 3D model of the aircraft.

4. The method for generating relative positions of flight support nodes based on 3D reconstruction according to claim 1, characterized in that: The S3 is as follows: S301, based on the selected 3D key point positions, expand the 3D model points into a homogeneous coordinate form, and apply a transformation matrix composed of a rotation matrix, a translation matrix, and a scaling matrix for transformation processing; S302, applying a perspective projection matrix to the transformed 3D model points to project them from the three-dimensional space to a two-dimensional plane; S303, converting the projected point from the normalized device coordinates to the screen coordinates of the image to map from the 3D model to the 2D image; S304: For each key point position of the 3D model, calculate the weighted Euclidean distance between its projection coordinates and the corresponding 2D image key point using the first objective function, and quantify the registration error using the sum of the weighted Euclidean distances to obtain a matching optimization result.

5. The method for generating relative positions of flight support nodes based on 3D reconstruction according to claim 4, characterized in that: The calculation process of the first objective function is as follows: A1. Expand the key points of the 3D model into homogeneous coordinate form and apply a transformation matrix composed of rotation, translation rotation and scaling matrices to transform the key points of the 3D model; A2. Apply a perspective projection matrix to the transformed 3D model key points to project them from 3D space onto a 2D plane, where the perspective projection parameters include field of view angle, aspect ratio, near plane distance, and far plane distance; A3. Convert the projected key points from normalized device coordinates to the screen coordinates of the image, completing the mapping from the 3D model to the 2D image. A4. For each 3D model key point, calculate the weighted Euclidean distance between its projected coordinates and the corresponding 2D image key point, where the first objective function is to take the sum of the weighted errors of all key points as the optimization target value. The optimization of the first objective function is to minimize the first objective function by finding the optimal parameter combination in the search space of rotation, translation, and scaling parameters.

6. The method for generating relative positions of flight support nodes based on 3D reconstruction according to claim 1, characterized in that: The S4 is specifically: S401, taking the 2D key point coordinates of the aircraft as the target 2D point; S402: construct a 3D model point array based on the 3D key point position, and select the 3D model point array corresponding to the identifier in the 2D key point coordinate as the 3D model point array; S403, setting a parameter range for a particle swarm optimization algorithm, wherein, in the particle swarm optimization algorithm, the 3D model points of the aircraft, the 2D points of the target aircraft, and the current 2D image size are passed into a second objective function to minimize the matching optimization result in S3 as the optimization goal; S404, in each iteration, updating the particle velocity and position, and optimizing the particle position and velocity to complete the optimization of the particle swarm objective function; S405, performing 3D model transformation and projection processing based on the optimization result of the particle swarm objective function; S406: Based on the transformation and projection processing results, the 2D projection coordinates of the 3D model vertices are transmitted back to the detection file as the results, and the overlap and visualization results of the aircraft model and the detection results are output, completing the adjustment and projection processing of the aircraft model.

7. The method for generating relative positions of flight support nodes based on 3D reconstruction according to claim 6, characterized in that: The calculation process of the second objective function is as follows: B1. Obtain optimization parameters; B2. Based on the optimized parameters, the transformation matrix is ​​obtained by constructing the rotation matrix, translation matrix, and scaling matrix; B3. Construct a perspective projection matrix by setting perspective projection parameters, perform homogeneous coordinate expansion on each 3D model point, apply the transformation matrix and perspective projection matrix, and project the point into 2D space through perspective division. B4. Convert the projection result from standard device coordinates to screen coordinates; B5. Based on the transformation results, calculate the weighted Euclidean distance between each projected point and the corresponding target 2D point; B6. Based on the weighted Euclidean distance, calculate a second objective function, where the second objective function is a weighted sum of all point errors.

8. The method for generating relative positions of flight support nodes based on 3D reconstruction according to claim 6, characterized in that: The S405 is specifically as follows: C1. Based on the optimization results of the particle swarm objective function, the optimal parameters are used to construct the transformation matrix and perspective projection matrix. The transformation matrix includes the rotation matrix, translation matrix and scaling matrix. C2. Load the 3D model file of the aircraft and obtain the model vertex data; C3. For each vertex, expand it into homogeneous coordinate form, apply the transformation matrix and perspective projection matrix in sequence, perform perspective division, and convert the coordinates into screen coordinates to obtain the 2D projection coordinates of the 3D model vertex, completing the transformation and projection processing of the 3D model.

Images