A method for intelligent analysis of airport video images and automatic detection of events

Abstract

The present application relates to the field of computer vision and intelligent monitoring technology, and discloses a kind of apron video image intelligent analysis and event automatic detection method, including pose solution to apron real-time video, generate perspective projection matrix, three-dimensional work space constraint is projected to two-dimensional image, constructs perspective remapping corridor and work stop point.Succeedingly, the remaining approach distance of work vehicle in corridor relative to stop point and instantaneous image speed vector are calculated.Based on the remaining approach distance, the direction deviation threshold of speed vector is dynamically adjusted, and the consistency of vehicle driving direction is checked.Finally, in response to the compliance signal, the confidence of stop point alignment, kinematic static state and position in place state are integrated using finite state machine model, and the work event is output.The present application effectively overcomes the image perspective distortion interference by introducing three-dimensional physical constraint and dynamic tolerance mechanism, and realizes the accurate and normative detection of apron work process.

Description

Technical Field

[0001] This invention relates to the field of computer vision and intelligent monitoring technology, specifically to a method for intelligent analysis of apron video images and automatic event detection. Background Technology

[0002] Existing apron monitoring technologies primarily rely on the analysis of two-dimensional images captured by monocular cameras, typically using pixel coordinates on the image plane for target detection and region segmentation. However, due to the limitations of camera imaging principles, objective perspective distortion and depth compression exist in video footage, causing the same physical area to exhibit significant geometric deformation under different camera positions or shooting angles. Relying solely on fixed two-dimensional pixel areas makes it difficult to accurately define the actual physical operation boundaries, resulting in a lack of geometric consistency in the system across different scenarios and an inability to truly reflect operational constraints in three-dimensional space.

[0003] In analyzing the motion of ground support vehicles, existing technologies often use fixed parameter thresholds to determine the consistency of the vehicle's trajectory or direction. During approach to an aircraft, the required trajectory accuracy varies with distance. If the system uses a uniform fixed threshold, it often fails to balance the tolerance for pilot fine-tuning or noise at long distances with the strict control of path direction during close-range docking, leading to a contradiction between the continuity of target tracking and the accuracy of standardized judgments.

[0004] Furthermore, current event detection logic typically relies on a single trigger condition, such as simply determining whether a target has entered an area or is stationary. This approach lacks a comprehensive consideration of vehicle spatial attitude and operational timing, and cannot effectively quantify the relative angular relationship between the vehicle and the aircraft. In the complex apron operating environment, relying solely on position information or simple motion states makes it difficult to distinguish between normal passing vehicles and operational vehicles actually performing services, and is easily susceptible to interference from non-operational targets, leading to false alarms. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an intelligent analysis and automatic event detection method for apron video images. This method solves the problem that existing technologies rely solely on two-dimensional image pixel features for apron moving target detection. Due to the lack of three-dimensional spatial depth information and aircraft attitude constraints, it is difficult to accurately determine the approach trajectory and parking status of operating vehicles when dealing with scenes with severe perspective distortion or large changes in depth of field.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution: a method for intelligent analysis and automatic event detection of apron video images, comprising first performing pose calculation on the real-time video image sequence of the apron, establishing a mapping relationship between the two-dimensional image pixel coordinate system and the aircraft world coordinate system, and generating a perspective projection matrix; Subsequently, the perspective projection matrix is ​​used to project the three-dimensional operation space constraints defined in the aircraft world coordinate system onto the two-dimensional image pixel coordinate system to generate perspective remapping corridors and operation docking pixels. Based on this, the ground contact reference point of the work vehicle located in the perspective remapping corridor is extracted, and the remaining approach distance of the reference point relative to the work parking pixel point and the instantaneous image velocity vector of the work vehicle are calculated. The velocity vector direction deviation threshold is dynamically adjusted based on the remaining approach distance, and the threshold is used to verify the direction consistency of the instantaneous image velocity vector to generate a compliant approach signal. Finally, in response to the compliant approach signal, the finite state machine model is used to comprehensively determine the vanishing point alignment confidence, kinematic static state, and positional status of the work vehicle, and the work event is output.

[0007] Preferably, the pose calculation process includes: extracting the heat map response of key aircraft points from a real-time video image sequence using a deep learning network, and obtaining two-dimensional observation coordinates by decoding through differential expectation operation; matching the two-dimensional observation coordinates with the three-dimensional physical coordinates in a preset three-dimensional topology model to construct a 2D-3D point pair set; solving the camera's rotation matrix and translation vector by minimizing the reprojection error, and constructing the perspective projection matrix by combining the camera's intrinsic parameters.

[0008] Preferably, the generation process of the perspective remapping corridor includes: obtaining the three-dimensional physical boundary vertices that define the standard operating area and converting them into homogeneous coordinates; using the perspective projection matrix to transform the homogeneous coordinates, verify the depth validity, and perform dehomogenization processing to obtain two-dimensional projection vertices; and connecting the two-dimensional projection vertices according to the topological order to cover the relevant areas of the real-time video image sequence.

[0009] Preferably, the process of dynamically adjusting the velocity vector direction deviation threshold adopts a nonlinear convergence strategy: setting a minimum hard tolerance boundary and a maximum relaxation tolerance boundary, and constructing a distance mapping function; through the distance mapping function, as the remaining approach distance decreases, controlling the velocity vector direction deviation threshold to nonlinearly converge from the maximum relaxation tolerance boundary to the minimum hard tolerance boundary; in specific calculations, an exponential decay model can be used to weight the difference between the two boundary values.

[0010] Preferably, the calculation process of the vanishing point alignment confidence includes: projecting the point at infinity of the longitudinal axis of the aircraft fuselage in the world coordinate system onto the image plane using the perspective projection matrix to obtain the theoretical reference vanishing point; extracting edge line segments in the image area where the operating vehicle is located and fitting the observed vehicle vanishing point; and calculating the confidence of the degree of parallelism between the observed vehicle vanishing point and the theoretical reference vanishing point using the Gaussian kernel function based on the distance deviation between the observed vehicle vanishing point and the theoretical reference vanishing point.

[0011] Preferably, the process of verifying the direction consistency includes: using the perspective projection matrix to map the standard approach path tangent in three-dimensional space to the standard approach path tangent direction on the two-dimensional image plane; calculating the inverse cosine of the dot product of the instantaneous image velocity vector and the standard direction to obtain the direction angle deviation; and determining that the verification is passed when the direction angle deviation is less than or equal to the velocity vector direction deviation threshold at the current moment.

[0012] Preferably, the finite state machine model includes an approach state, a verification state, and a completion state. In the verification state, the system determines whether the following conditions are met simultaneously: the magnitude of the instantaneous image velocity vector is continuously lower than the micro-motion threshold within a preset time window, the remaining approach distance is less than the physical docking radius threshold, and the vanishing point alignment confidence is greater than the attitude qualification threshold. If all conditions are met, the system switches to the completion state and outputs a job start event.

[0013] Preferably, the method further includes: if it is detected that the work vehicle leaves the perspective remapping corridor along the opposite direction vector of the standard approach path, then the finite state machine model is reset and an operation end event is output.

[0014] This invention provides a method for intelligent analysis and automatic event detection of apron video images. It has the following beneficial effects: 1. This invention establishes a mapping relationship between the two-dimensional image pixel coordinate system and the aircraft world coordinate system, and uses a perspective projection matrix to generate a perspective remapping corridor, thereby achieving the delineation of the work area based on three-dimensional physical space constraints. This processing method can directly reproduce the real physical space limitations on the image plane, overcoming the perspective distortion and depth compression problems existing in monocular camera images, and ensuring geometric consistency in the determination of the work area under different camera positions or shooting angles.

[0015] 2. This invention implements convergence control from long-range relaxation tolerance to short-range hard tolerance by constructing a distance mapping function and dynamically adjusting the velocity vector direction deviation threshold based on the remaining approach distance. This mechanism conforms to the kinematic laws of actual ground vehicle approach processes, tolerates minor driving adjustments or sensor noise at long distances, and performs strict path direction consistency checks at close-range stopping stages, thus balancing the continuity of target tracking with the accuracy of operational procedure determination.

[0016] 3. This invention constructs a multi-condition constrained temporal logic by comprehensively determining the vanishing point alignment confidence, kinematic static state, and positional arrival state using a finite state machine model. By quantifying the relative attitude of the vehicle and aircraft using the geometric features of the vanishing point and combining this with the temporal flow mechanism of the state machine, it effectively distinguishes between compliant operating vehicles and interfering vehicles, avoiding false alarms caused by single-position triggers, and achieving automated detection of the entire process of apron operation events. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a method for apron video image analysis and automatic event detection according to an embodiment of the present invention; Figure 2 This is a system architecture principle block diagram of one embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the principle of standard operation trajectory perspective remapping and special vehicle motion feature extraction according to an embodiment of the present invention; Figure 4 This is a schematic diagram of dynamic tolerance gating logic based on a distance-precision correlation model according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the finite state machine decision logic for fusion vanishing point consistency constraints according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the dynamic tolerance gating region division according to an embodiment of the present invention; Figure 7 This is a time series comparison diagram of vanishing point alignment confidence in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] See attached document Figure 1 and attached Figure 2This invention provides a method for intelligent analysis and automatic event detection of apron video images. This method relies on a high-performance computing system, which mainly includes a video stream acquisition module, an aircraft spatial pose calculation module, a perspective operation corridor generation module, a dynamic tolerance gating module, and a timing state determination module. These modules are connected via a high-bandwidth data bus and utilize a shared, unified, high-precision system clock to ensure strict alignment of multi-source data in the time dimension. The instruction code for the method runs on an industrial-grade edge computing unit or a server with GPU acceleration capabilities. By executing program instructions stored in memory, it achieves the determination of operational compliance in complex dynamic scenarios on the apron.

[0020] The video stream acquisition module establishes a communication connection with the surveillance cameras installed on the apron high-mast lights or the top of the jet bridge. The surveillance cameras are configured in fixed-focus or zoom mode and undergo intrinsic parameter calibration to obtain the distortion coefficients. The video stream acquisition module continuously buffers and acquires real-time video image sequences containing the aircraft parking area, while accurately timestamping each frame. The real-time video image sequence is transmitted to the aircraft spatial pose calculation module as input data for subsequent steps.

[0021] In step S1, the aircraft spatial pose calculation module processes the input monitoring video frames. Its core task is to construct the rigid body transformation relationship between the two-dimensional image pixel coordinate system and the physical world coordinate system. The aircraft spatial pose calculation module first calls a pre-set aircraft 3D topology model library. This library, acting as a priori knowledge storage unit, stores rigid body structure data for different aircraft models and defines a world coordinate system with the aircraft's own structural features (e.g., the nose cone apex) as the origin. In this world coordinate system, each key structural node of the aircraft has definite three-dimensional spatial coordinates. To recover 3D information from the two-dimensional monitoring video frames, the aircraft spatial pose calculation module integrates a deep learning keypoint detection unit. This deep learning keypoint detection unit is trained to extract specific semantic features from the monitoring video frames. It takes as input an image feature tensor generated from the preprocessed monitoring video frames and outputs coordinate data representing the positions of each keypoint.

[0022] Based on the inference results of the deep learning keypoint detection unit, several heterogeneous keypoints of the aircraft in the monitoring video frames are identified. These heterogeneous keypoints are specifically selected as rigid structures least affected by load deformation, including but not limited to landing gear touchdown points, wingtips, engine inlet centers, and fuselage door corners. The identification results are output as a set of pixel coordinates based on a 2D image pixel coordinate system. Subsequently, the aircraft spatial pose calculation module establishes a one-to-one topological association between the identified pixel coordinates and the 3D spatial coordinates in the aircraft's 3D topology model library, forming a set of 2D and 3D point pairs. To overcome false detection interference caused by the on-site environment, the aircraft spatial pose calculation module introduces a robust estimation mechanism to filter the point pair set, eliminating outliers that do not conform to geometric constraints. By constructing and minimizing the reprojection error objective function, the six-degree-of-freedom pose parameters of the monitoring camera relative to the aircraft's world coordinate system are calculated. The system monitors the numerical stability of the calculation results in real time to ensure the physical reliability of the output pose parameters. Finally, based on the calculated intrinsic parameter matrix, rotation matrix, and translation vector, the aircraft spatial pose calculation module generates a perspective projection matrix from the three-dimensional world coordinate system to the two-dimensional image pixel coordinate system.

[0023] In step S2, the perspective operation corridor generation module uses a perspective projection matrix to map the virtual space constraints defined in the standard operating procedure onto the monitoring video frame at the current moment, realizing the visualization and digitization of physical rules. Within the aircraft world coordinate system, a three-dimensional operation bounding box or ground trajectory zone is predefined for specific apron operation types. The three-dimensional operation bounding box or ground trajectory zone is delineated according to relevant aviation ground service specifications, representing the vehicle movement space that conforms to the operating specifications. The perspective operation corridor generation module calculates the projected coordinates of the set of boundary vertices constituting the three-dimensional operation bounding box or ground trajectory zone using matrix multiplication operations via the perspective projection matrix. The polygonal area enclosed by the projected coordinates is the perspective remapping corridor. Since the projection process of the perspective remapping corridor directly depends on real-time pose calculation, the shape and position of the perspective remapping corridor on the monitoring video frame adaptively adjust in real time according to changes in the aircraft's parking attitude and the monitoring camera's viewing angle, ensuring that it always accurately conforms to the ground or operation space in the physical scene, eliminating detection area drift caused by viewing angle differences.

[0024] In step S3, the system extracts the motion features of the work vehicle within the generated perspective remapping corridor area. The target detection unit scans the monitoring video frames, and when a special vehicle is detected entering the perspective remapping corridor area, it locks onto the special vehicle and assigns a unique tracking ID. The system calculates the displacement vector of the special vehicle's center point in the monitoring video frame sequence, and combines the timestamp difference between consecutive frames to derive the special vehicle's instantaneous velocity vector and acceleration scalar. During the calculation process, the system performs logical verification on the validity of the time sampling to ensure the continuity and accuracy of the motion parameter calculation. Simultaneously, the system calculates the pixel Euclidean distance between the special vehicle's center point and the work docking target point in the two-dimensional image pixel coordinate system. The work docking target point is a coordinate point obtained by projecting the standard docking point in three-dimensional space onto the two-dimensional image pixel coordinate system using a perspective projection matrix.

[0025] In step S4, the dynamic tolerance gating module adjusts the stringency of the judgment logic in real time based on the distance between the special vehicle and the target work stop point, simulating the operation logic of human experts providing rough guidance from a distance and fine alignment at the near end. The dynamic tolerance gating module has a built-in distance-deviation threshold mapping function used to calculate the allowable speed vector direction deviation threshold. This distance-deviation threshold mapping function is configured with a mathematical relationship exhibiting non-linear convergence characteristics, which can smoothly simulate the transition characteristics from the free approach zone to the strictly controlled zone; that is, as the distance between the special vehicle and the target work stop point decreases, the allowable speed vector direction deviation threshold tightens non-linearly. The dynamic tolerance gating module calculates the angular deviation between the current instantaneous speed vector direction of the special vehicle and the tangent direction of the standard approach path. The tangent direction of the standard approach path is obtained by transforming the standard path vector in three-dimensional space to the two-dimensional image pixel coordinate system through a perspective projection matrix. The dynamic tolerance gating module outputs a compliant approach signal only when the special vehicle is located within the perspective remapping corridor area and the included angle deviation is less than or equal to the velocity vector direction deviation threshold corresponding to the current distance. Otherwise, the movement of the special vehicle is marked as an illegal approach or background interference, and the subsequent event judgment process is not triggered. This mechanism directly controls the logical judgment parameters through geometric constraints, achieving fine-grained verification of the operation trajectory.

[0026] In step S5, the timing state determination module receives motion features filtered by the dynamic tolerance gating module and infers the operation event using a finite state machine model. The finite state machine model includes states such as idle, approach, verification, and completion. To further eliminate false alarms caused by light and shadow interference or non-realistic targets, the timing state determination module introduces vanishing point consistency verification logic in the approach state. Based on perspective geometry principles, physically parallel lines should converge at a single point on the two-dimensional image plane. The timing state determination module uses the rotation matrix output by the aircraft spatial pose calculation module to calculate the global geometric vanishing point coordinates of the physical axis perpendicular to the ground in the two-dimensional image pixel coordinate system. Simultaneously, based on the texture features or edge contour information of the special vehicle in the monitoring video frame, the timing state determination module estimates the vertical vanishing point coordinates of the special vehicle itself using a geometric analysis algorithm. The timing state determination module calculates the distance deviation between the global geometric vanishing point and the vertical vanishing point of the special vehicle. If the deviation exceeds a preset range, the current target is determined to lack a three-dimensional structure conforming to physical perspective laws and is excluded. Once the special vehicle meets the compliant approach conditions and comes to a stable stop in the work area, the timing state determination module activates the geometric consistency verification unit. Based on the vanishing point convergence principle in projective geometry, it calculates the alignment confidence between the intersection of the extended lines of the special vehicle's contour and the theoretical vanishing point of the aircraft's axis. This confidence serves as a confirmation signal of compliant parking attitude, completing the output of the work start event. When it is detected that the special vehicle has left the perspective remapping corridor along the opposite direction vector of the standard path, the work end event is output.

[0027] The above scheme achieves reverse verification of 3D spatial operation specifications from a monocular monitoring perspective by using physical model registration to drive logic parameters, thus forming a closed-loop detection process.

[0028] See attached document Figure 1 and attached Figure 2 In order to support the subsequent high-precision spatial pose calculation, in step S1, the aircraft spatial pose calculation module establishes a precise correspondence between two-dimensional image observation data and three-dimensional physical model data. This process relies on the collaborative work of a pre-set aircraft three-dimensional topology model library and real-time depth feature inference.

[0029] In this embodiment, the aircraft 3D topology model library is stored in a non-volatile storage medium using a lookup table data structure. The aircraft spatial pose calculation module indexes and loads the corresponding rigid body structure data based on the aircraft identification code input by the flight plan or the automatic aircraft type identification module. For each specific aircraft type, the aircraft 3D topology model library defines the origin as a rigid point with a unique and obvious feature on the central axis of the aircraft's physical structure (usually the nose cone apex). The right-handed Cartesian coordinate system, also known as the aircraft world coordinate system. Under the definition of the aircraft world coordinate system, The axis extends along the longitudinal axis of symmetry of the aircraft fuselage and points towards the tail. The shaft extends along the span of the wing. The axis is perpendicular to the shaft and The reference plane formed by the axis points upwards from the back of the machine.

[0030] The set of key structural nodes recorded in the aircraft 3D topology model library Total number of key points A predefined three-dimensional physical space coordinate system. To avoid the matrix singularity problem that occurs in the subsequent perspective n-point algorithm (PnP) when solving for coplanar feature points, the selected set of key structural nodes is forcibly constrained to have a spatially heterogeneous distribution. Specifically, the set of key structural nodes... Not only includes Ground feature points with axis coordinate values ​​close to 0 (such as the nose landing gear tire contact point and the main landing gear axle center) must also include aerial feature points with significant height differences (such as the fuselage front door corner, the engine intake cowling center, and the wingtip navigation light position). This discrete distribution characteristic in the height dimension effectively eliminates the parallax error caused by homography transformation when processing non-planar objects, ensuring that the pose calculation matrix of the surveillance camera has full rank characteristics. (Corresponding to key point index) The three-dimensional physical space coordinates are defined by the aircraft world coordinate system. axis coordinates , axis coordinates and axis coordinates The three-dimensional position vector formed. Wherein, For the keypoint index used to uniquely identify the semantic category of aircraft keypoints, the total number of keypoints. The integer is greater than or equal to 6 to satisfy the minimum degree of freedom requirement for six-degree-of-freedom solution.

[0031] During the real-time processing phase, the video stream acquisition module outputs monitoring video frames. After preprocessing (including size normalization, mean subtraction, and standard deviation scaling), it is converted to the size... Image feature tensor The program then moves to the deep learning keypoint detection unit. This unit employs a convolutional neural network architecture containing multi-resolution parallel subnetworks. It maintains high-resolution representation and recovers spatial location information by performing repeated multi-scale fusion between feature maps of different resolutions. For the specific network layer structure design, convolutional kernel parameter configuration, and basic backpropagation algorithm of the convolutional neural network, those skilled in the art can refer to existing mature high-resolution network architectures (such as HRNet) for implementation. The basic model construction principles are well-known technologies in this field and will not be elaborated upon here. The convolutional neural network does not perform dimensionality reduction classification of the image; instead, it outputs a set of channels whose total number equals the total number of keypoints. Key point response heatmap tensor .

[0032] During the training phase of a convolutional neural network, mean squared error is used as the optimization objective to enable the network to learn the pixel-level keypoint distribution. For the keypoint indices in the training set... The ground truth coordinates of the labeled keypoints are used to generate a two-dimensional Gaussian distribution centered at these coordinates as a Gaussian distribution label heatmap. During training, the loss function is configured to calculate the mean of the sum of squares of the differences between the predicted keypoint response heatmap tensor and the Gaussian distribution label heatmap. This loss value is minimized through the backpropagation algorithm, enabling the convolutional neural network to learn the probability distribution response of the keypoints in the image space.

[0033] In the inference phase, to overcome the quantization error caused by the traditional maximum indexing method, the aircraft spatial pose calculation module uses differential expectation calculation to obtain the response heatmap tensor from the key point. The coordinates are decoded to sub-pixel precision. For the coordinates corresponding to the keypoint index... Key point response heatmap channel The horizontal coordinate in the corresponding two-dimensional image pixel coordinates and vertical coordinates It is obtained by calculation using the following formula: ; ; in: Keypoint Index The horizontal coordinates of the corresponding key points in the image pixel coordinate system. Keypoint Index The vertical coordinates of the corresponding key points in the image pixel coordinate system. This represents the summation operation symbol calculated for the molecular part, which covers all discrete pixel positions in the width and height directions of the heatmap. This represents the horizontal discrete pixel coordinate variable used for the weighted summation calculation of the numerator part. This represents the vertical discrete pixel coordinate variable used for the weighted summation calculation of the molecule part. This indicates the index corresponding to the key point. Key point response heatmap channels in discrete pixel coordinates The activation response value at that location. This represents the temperature coefficient, used to control the smoothness of the probability distribution. The preferred value range is between 0.1 and 1.0. Represented by natural constant The exponential function operator with base 0. Indicates index for key points The normalized denominator term of the calculation is specifically defined as follows: ,in This indicates the summation operation symbol for the denominator, which covers all discrete pixel positions in the width and height directions of the heatmap; This represents the horizontal discrete pixel coordinate variable used in the calculation of the normalization factor in the denominator. This represents the vertical discrete pixel coordinate variable used for calculating the normalization factor in the denominator. This indicates the index corresponding to the key point. Key point response heatmap channels in discrete pixel coordinates The activation response value at that location.

[0034] This calculation method utilizes the overall response information within the neighborhood of key points, so that the output coordinates are no longer limited to integer grid points, thereby improving the accuracy of subsequent geometric solutions.

[0035] To ensure the validity of the input data, the aircraft spatial pose calculation module calculates a keypoint confidence score for each decoded 2D image pixel coordinate. Only when the keypoint confidence score is greater than a preset confidence threshold (e.g., 0.6) is the keypoint marked as visible and included in subsequent calculations; otherwise, it is considered occluded or undetected. Based on the consistency of semantic labels, the aircraft spatial pose calculation module combines the filtered valid keypoint image pixel coordinates with the 3D physical space coordinates in the aircraft's 3D topology model library to form 2D and 3D point pair sets. This 2D and 3D point pair set serves as a bridge between the observed data and the physical ground truth, and is transmitted to the next-level module for iterative solution of pose parameters.

[0036] See attached document Figure 1 and attached Figure 2In the pose calculation stage of step S1, the aircraft spatial pose calculation module utilizes the aforementioned constructed set of 2D and 3D point pairs and, through the perspective n-point algorithm (PnP) framework, calculates the precise spatial position and attitude of the surveillance camera relative to the aircraft's world coordinate system. The surveillance camera pose calculation process is based on a pinhole camera imaging model, aiming to establish the geometric transformation relationship between the three-dimensional rigid body structure and the two-dimensional projection plane, and to eliminate the influence of observation noise on pose estimation through nonlinear optimization methods.

[0037] The aircraft spatial pose calculation module reads the intrinsic parameter matrix of the surveillance camera. and distortion coefficient vector Before performing geometric calculations, the aircraft spatial pose calculation module first performs distortion correction processing on the input two-dimensional image pixel coordinates, mapping the original observation coordinates to an ideal distortion-free normalized plane. For calculations based on distortion coefficient vectors... The specific mathematical operations for performing image distortion correction (including radial and tangential distortion coefficients) can be implemented by those skilled in the art by referring to classic literature such as the Brown-Conrady distortion model. This is a well-known technology in the field and will not be elaborated here.

[0038] To solve for the six-DOF pose of the surveillance camera, i.e., to determine the camera's orientation and orientation in the aircraft's world coordinate system, the aircraft spatial pose calculation module constructs a nonlinear least squares optimization problem. To ensure the stability of the numerical calculation, the aircraft spatial pose calculation module employs a coordinate normalization strategy, scaling the image pixel coordinates and 3D spatial coordinates to values ​​centered at the origin and with an average distance of [missing value]. and Within a unit sphere, to prevent the nonlinear least squares optimization problem from getting stuck in local minima or diverging due to excessive differences in data magnitude.

[0039] The core objective of nonlinear least squares optimization problems is to find the optimal rotation matrix. Translation vector This ensures that after rigid body transformation and perspective projection, the distance error between the reprojected position on the image plane and the actual observation point is minimized for all valid 3D physical space coordinates. To handle the rotation matrix... To solve the orthogonal constraint problem and reduce the number of optimization parameters, the aircraft spatial pose calculation module utilizes the Rodrigues transformation formula to transform the constrained... rotation matrix Convert to unconstrained Rotation vector (which mathematically corresponds to a Lie algebra) (elements in space). Through this parameterization method, the optimization process only needs to iterate over 3 rotation vector components, avoiding the orthogonality violation problem that occurs when directly optimizing 9 matrix elements.

[0040] The aircraft spatial attitude calculation module uses the reprojection error objective function defined as follows. Perform iterative solution: ; in: Let be the objective function for reprojection error, representing the total reprojection error energy that needs to be minimized. This represents the total number of valid matching point pairs participating in pose calculation in the current frame. This represents the point-to-point index number, with a value range of 100. to . The robust kernel function is represented, preferably the Huber kernel or the Tukey Biweight kernel; when the error residual is less than the preset robust threshold... (For example When the error residual is greater than 0, the squared error is used to ensure convexity; when the error residual is greater than 0, the squared error is used to ensure convexity. At this time, linear or truncation errors are used to suppress outliers from misleading the gradient descent direction. It represents the square of the L2 norm, and is a squared measure of the Euclidean distance. Indicates the first The key points are the two-dimensional pixel coordinate vectors actually observed in the distortion-free image. Indicates the first The three-dimensional physical space coordinate vectors of the key points in the aircraft world coordinate system. This represents the intrinsic parameter matrix of the surveillance camera, which includes the normalized focal length and principal point coordinates. express The rotation matrix, whose determinant value This describes the rotational transformation of the world coordinate system basis vectors relative to the surveillance camera coordinate system basis vectors. express The translation vector represents the displacement component of the origin of the world coordinate system in the coordinate system of the surveillance camera. This represents a projection mapping function from three-dimensional space to a two-dimensional plane.

[0041] The specific calculation logic of the projection mapping function is as follows: First, using the rotation matrix to be solved... Translation vector The three-dimensional physical space coordinate vector in the aircraft world coordinate system The vector is transformed into a three-dimensional coordinate vector in the coordinate system of the surveillance camera. Then, based on the principle of pinhole imaging, the aircraft spatial pose calculation module extracts the depth component (denoted as ) of this transformed three-dimensional coordinate vector in the surveillance camera coordinate system. The horizontal component of the three-dimensional coordinate vector (denoted as...) ) and vertical component (denoted as Divide by the depth component respectively This yields the normalized planar coordinates. Finally, the aircraft spatial pose calculation module multiplies the normalized planar coordinates by the intrinsic parameter matrix. By combining the focal length and principal point parameters, the predicted two-dimensional pixel projection coordinates of the model are calculated. .

[0042] After executing the above projection mapping function During the calculation process, the aircraft spatial pose calculation module introduced depth validity verification logic. If the calculated depth components in the monitoring camera coordinate system... Less than the preset depth cutoff threshold (For example If the distance is less than or equal to the optical center, the point is determined to be behind the imaging plane of the surveillance camera or coincides with the optical center, and is considered an invalid projection. In this case, the aircraft spatial pose calculation module sets the error term corresponding to this point to zero in the objective function or assigns a very large penalty value to avoid division by zero anomalies that could cause program crashes or generate infinite gradient values.

[0043] In solving the aforementioned nonlinear optimization problem, the aircraft spatial pose calculation module first employs the efficient PnP (EPnP) algorithm combined with a random sample consensus framework to generate the rotation matrix. Translation vector The initial estimate is then used for iterative optimization using the Levenberg-Marquardt algorithm. The specific matrix factorization steps of the EPnP algorithm, the interior point selection mechanism of the random sampling consensus framework, and the specific implementation of how the Levenberg-Marquardt algorithm dynamically adjusts the damping factor to switch between gradient descent and Gauss-Newton methods are readily available from standard mathematical libraries and are well-known techniques in the field, so they will not be elaborated further here. The iterative process continues until the reprojection error objective function is obtained. The change is less than the preset convergence threshold (e.g.) (or the number of iterations reaches the preset maximum number of iterations).

[0044] After the calculation is completed, the aircraft spatial pose calculation module performs a physical consistency check. The aircraft spatial pose calculation module checks the calculated height of the surveillance camera (derived from the rotation matrix). Translation vector The inverse transformation is used to check whether the result is within a reasonable range (e.g., 2 to 30 meters above ground) and whether the optical axis of the monitoring camera points to the ground (i.e., the pitch angle is negative). Only the solution that passes the physical consistency check is considered valid. The aircraft spatial pose calculation module outputs the final optimized rotation matrix. Translation vector And construct the perspective projection matrix : ; in: for A perspective projection matrix, through homogeneous coordinate transformation, can uniformly map points at infinity and ordinary points in the aircraft coordinate system to the image plane. Perspective projection matrix As the core geometric parameter connecting the three-dimensional physical space and the two-dimensional image space, it is passed to the subsequent perspective operation corridor generation module for perspective remapping of the standard operation trajectory in step S2. This mathematical description ensures that the perspective remapping corridor generated in subsequent steps strictly follows the perspective laws of the physical world geometrically, ensuring a close fit between the projected area and the ground texture even when the aircraft is not directly facing or is parked at a large angle.

[0045] See attached document Figure 3 Based on the perspective projection matrix calculated in step S1 In step S2, the system aims to establish a geometric bridge connecting the three-dimensional operation specifications and the two-dimensional monitoring screen, and to perform quantitative extraction of the motion features of special vehicles under this unified spatial reference. This process is led by the perspective operation corridor generation module, which aims to instantiate abstract spatial constraints into dynamic electronic fences on the image plane through projective geometric transformation.

[0046] The perspective operation corridor generation module connects to the standard operating procedure spatial constraint database. This database, based on the aircraft world coordinate system, stores standardized spatial definitions for various ground service support operations. Considering that the operation of special vehicles on the apron (such as refueling trucks and power supply vehicles) is mainly constrained by ground trajectories, the standard approach path is parameterized to lie on the ground plane (i.e., in the aircraft world coordinate system). A polygonal region on a plane. The spatial shape of the polygonal region is uniquely described by an ordered set of three-dimensional physical boundary vertices, where each three-dimensional physical boundary vertex... All include horizontal coordinates Vertical coordinates and the vertical coordinates that are forcibly constrained to 0 The database contains three-dimensional vectors. Furthermore, it clearly records the target points for work stoppages. The three-dimensional coordinates of the point, which physically correspond to the vertical projection of the aircraft service panel (such as the refueling port) on the ground, serve as an ideal reference for the final parking space of special vehicles.

[0047] The perspective corridor generation module calls the perspective projection matrix generated in step S1. The system performs perspective transformation operations on the aforementioned 3D physical boundary vertices. To ensure the numerical stability of the projection calculation, a depth verification mechanism is introduced during the calculation process. For the first... Three-dimensional physical boundary vertices The projection process is described as follows: ; in: This represents the homogeneous pixel coordinate vector after projection. The output of step S1 The perspective projection matrix. This represents the conversion of three-dimensional physical coordinates into homogeneous form. Input vector. The third component of the calculation result, namely the homogeneous scaling factor, physically corresponds to the depth value of the point in the coordinate system of the surveillance camera. and These are the horizontal and vertical coordinates of the image plane to be determined, respectively.

[0048] After obtaining the homogeneous coordinates, the system checks the homogeneous scaling factor. .like (here) Reuse the depth truncation threshold defined in step S1, for example If the vertex is located behind the imaging plane of the surveillance camera or at the optical center (in meters), the system will perform frustum clipping on that vertex. For vertexes that meet the following conditions... For valid vertices, the system performs a dehomogenization operation, which divides the first two components of the homogeneous coordinates by 1 / 2. This allows us to obtain the two-dimensional pixel coordinates of the vertex on the image plane. .

[0049] The perspective-based gallery generation module generates all projected 2D pixel coordinates according to the vertex index order. Connect sequentially, in the current monitoring video frame Generate a closed perspective remapping corridor above. .Should It is not a static layer, but rather a layer that changes with the perspective projection matrix. The dynamic area deforms in real time as it is updated. Even when the aircraft shifts position or the surveillance camera adjusts its focus, It consistently covers the corresponding work area on the physical ground with pixel-level precision, thus visually reproducing the physical constraints of three-dimensional space. Simultaneously, the work docks at the target point. Using the same projection logic, it is mapped to the work docking pixel in the two-dimensional image pixel coordinate system. .

[0050] See attached document Figure 1 and attached Figure 2 In step S3, after constructing the geometrically constrained environment, the system activates the target detection unit based on a deep convolutional neural network to identify and locate special vehicles entering the frame. In this embodiment, the target detection unit adopts a neural network architecture including a CSPDarknet backbone network, a path aggregation network neck structure, and a decoupled detection head. After adaptive scaling and normalization, the input image is fed into the backbone network to extract multi-scale feature maps. During network training, a composite loss function is used, including bounding box regression loss (such as CIoU Loss), confidence loss, and class probability loss (such as binary cross-entropy loss), and anchor boxes adapted to the aspect ratio distribution of special vehicles on the apron are preset based on the K-means clustering algorithm. For the specific convolutional layer structure of the CSPDarknet backbone network, the feature fusion mechanism of the path aggregation network, and the specific mathematical derivation of CIoU Loss, those skilled in the art can refer to existing open-source algorithm documentation in the field of target detection for implementation, which are well-known technologies in this field and will not be elaborated here.

[0051] Through forward inference, the target detection unit outputs a bounding box representing the position of the special vehicle. To eliminate the tilting effect caused by the special vehicle's own height under perspective projection (i.e., the projections of the roof and the bottom of the vehicle do not coincide in the image), the perspective operation corridor generation module does not directly use the geometric center of the bounding box, but instead extracts the center point of the bottom edge of the bounding box as the ground contact reference point of the special vehicle. This selection strategy is based on physical causal logic: special vehicles only contact the ground through their tires, and only the position of the ground contact point on the image can truly reflect the movement trajectory of the special vehicle on the physical ground plane, thus ensuring that it is on the same projection reference plane as the perspective remapping corridor defined on the ground plane.

[0052] The system uses the Kalman filter algorithm to determine the ground contact reference point of the special vehicle. Temporal correlation and smooth tracking are performed. Based on the timestamp difference between adjacent video frames, the perspective operation corridor generation module quantitatively calculates the instantaneous kinematic parameters of special vehicles.

[0053] First, calculate the remaining approach distance of the special vehicle to the standard operating point. This metric is defined as the ground contact reference point for special vehicles. With the job docking pixels The Euclidean distance (L2 norm) in the pixel space of a two-dimensional image is used to characterize the degree to which a special vehicle approaches a target.

[0054] Secondly, in order to obtain smooth and noise-resistant velocity information, the system calculates the instantaneous image velocity vector. The system employs a sliding window weighted averaging strategy to extract historical position data within a preset frame window (e.g., 5 to 10 frames). It calculates the position vector difference between adjacent frames and divides it by the corresponding timestamp difference to obtain the instantaneous velocity sample. Finally, it calculates the arithmetic mean of all instantaneous velocity samples within the window as the current instantaneous image velocity vector. During the calculation process, the system performs a non-zero check on the validity of the time difference.

[0055] The perspective operation corridor generation module uses the ray crossing method to determine the ground contact reference point of special vehicles in real time. Corridor with perspective remapping The topological relationship. Specifically, the system starts from... Emit a ray in any direction and count the ray's intersection with the target direction. The number of intersection points of the boundary line segments. If the number of intersection points is odd, then... Located within the region (including the boundary). For the specific programming implementation of the Kalman filter's state update equation and the ray-crossing method, those skilled in the art can refer to basic textbooks on computer vision and graphics; these are well-known techniques in the field and will not be elaborated upon here. The system only activates subsequent logic to calculate the remaining approach distance when the special vehicle is determined to be within the controlled operating area. and instantaneous image velocity vector The signal is then transmitted to the dynamic tolerance gating module. This mechanism acts as a spatial filter, effectively blocking false triggering signals from non-operational vehicles outside the corridor (such as passing shuttle buses), ensuring that the detection computing power is focused on high-risk approach operations.

[0056] See attached document Figure 4 After extracting the instantaneous motion parameters of the special vehicle in step S3, step S4 is executed by the dynamic tolerance gating module. It relies on the position-aware dynamic evaluation mechanism and aims to construct a virtual safety funnel on the two-dimensional image plane that converges with the approximation distance, thereby reproducing the operation logic of human experts’ rough guidance at the far end and fine alignment at the near end at the algorithm level.

[0057] In this embodiment, the dynamic tolerance gating module receives two key input variables from the perspective operation corridor generation module: the special vehicle ground contact reference point. Distance from the work stop pixel Remaining approach distance And instantaneous image velocity vectors of special vehicles To address the contradiction between oversensitivity at the far end and insufficient constraint at the near end caused by perspective compression in fixed-threshold judgment methods when handling scenes with great depth, the dynamic tolerance gating module incorporates a distance-to-deviation threshold mapping function. This function is not configured as a simple linear proportional relationship, but rather constructed as a nonlinear mathematical model with asymptotic convergence characteristics, used to determine the remaining approach distance. Real-time calculation of the allowable velocity vector direction deviation threshold at the current moment : ; in: The velocity vector direction deviation threshold represents the maximum angular tolerance value allowed by the system for the actual movement direction of a special vehicle to deviate from the standard approach path at the current time and distance. Its physical unit is radians. This represents the minimum hard tolerance boundary, i.e., the point at which a special vehicle contacts or is extremely close to the point where it stops for operation (i.e., The tolerance boundaries that must be met (e.g., a radian value corresponding to 5 degrees) are set according to the mandatory requirements for final docking accuracy in the Aircraft Ground Services Safety Specification. This represents the maximum relaxation tolerance boundary, which is the tolerance boundary allowed in the free approach zone far from the work stop pixel (e.g., a radius value corresponding to 45 degrees). This larger tolerance space allows special vehicles to make necessary attitude adjustments when initially entering the perspective remapping corridor without triggering an alarm. Represented by natural constant The exponential function operator with a base of 0x100 gives the threshold curve a non-linear characteristic of rapidly tightening at the near end and flattening at the far end. This represents the spatial convergence coefficient, used to control the rate at which tolerance tightening changes relative to the physical distance; in this embodiment, the spatial convergence coefficient... The preferred value range is between 0.05 and 0.2. The specific value needs to be calibrated in conjunction with the equivalent focal length of the monitoring camera and the physical scale of the working area to ensure that the threshold shrinkage curve matches the physical braking distance and minimum turning radius of the special vehicle. This represents the remaining approach distance input in step S3, in pixels.

[0058] After determining the current judgment criteria, the dynamic tolerance gating module needs to obtain the standard approach path tangent direction vector at the current position. Since the standard approach path is defined in three-dimensional space and includes curve segments, the dynamic tolerance gating module utilizes the perspective projection matrix calculated in step S1. The system maps the standard approach path vector in three-dimensional space to a two-dimensional image plane in real time. Specifically, the dynamic tolerance gating module searches for the standard approach path point closest to the current physical position of the special vehicle on the pre-stored three-dimensional standard approach path, extracts the tangent vector at that standard approach path point, and then maps it to the perspective projection matrix. Transform it into a standard approach path tangent direction vector in a two-dimensional image pixel coordinate system. .

[0059] Subsequently, the dynamic tolerance gating module performs a consistency check between the instantaneous image velocity vector and the standard approach path tangent direction vector. The system calculates the actual instantaneous image velocity vector of the special vehicle. tangent direction vector of standard approach path The deviation of the velocity vector between them Before performing the calculations, to avoid the risk of singularities in numerical computation, the system first processes the instantaneous image velocity vector. Length of the module Perform a non-zero threshold check. If If the noise level is below a preset static noise threshold (e.g., 0.5 pixels / frame), the special vehicle is determined to be stationary or in a state of slight movement. In this case, the instantaneous image velocity vector cannot be stably defined, and the system will skip the angle calculation step and retain the state determination result of the previous frame. If the modulus meets the calculation conditions, the system uses the vector dot product formula to calculate the inverse cosine of the dot product of the instantaneous image velocity vector and the normalized dot product of the standard approach path tangent direction vector, thereby obtaining the non-negative velocity vector angle deviation. .

[0060] Finally, the dynamic tolerance gating module performs a logic gating operation. The system will then calculate the velocity vector angle deviation. Deviation threshold between velocity vector direction and distance-dynamically generated velocity vector Perform a comparison. If the conditions are met... This indicates that although there is a slight deviation in the current motion posture of the special vehicle, it is within the safety margin coverage of the current distance and is an acceptable correction process. The dynamic tolerance gating module outputs a high-level Boolean signal representing a compliant approach, allowing data to flow to the next-level module. Conversely, if... This indicates that the movement trend of the special vehicle exceeds the safety tolerance allowed by the current distance, posing a risk of deviating from the perspective remapping corridor or colliding. The dynamic tolerance gating module outputs an illegal approach signal. This signal will immediately trigger an early warning mechanism, or mark the current object as an uncontrolled interference target in the event judgment logic, thereby preventing the false triggering of subsequent operation completion events. Through this mechanism where geometric constraints directly drive logic parameters, this embodiment achieves refined verification of the special vehicle's operating trajectory, effectively shielding non-critical jitter at the far end, while ensuring that the near-end docking process strictly follows standard operating procedures.

[0061] See attached document Figure 5 After the compliant motion data verified by dynamic tolerance gating is output in step S4, step S5 is executed by the timing state determination module. The timing state determination module does not use a single threshold trigger logic, but aggregates discrete, instantaneous frame-level data streams into continuous operation events with clear semantics, and introduces ultimate geometric constraints using the vanishing point characteristics in projective geometry to eliminate false positives when special vehicles arrive at the designated position but have slight deviations in their parking angle, thereby constructing a closed-loop determination for the entire operation lifecycle.

[0062] In this embodiment, the timing state determination module maintains a finite state machine in memory. This finite state machine models the timing behavior of special vehicles using a state transition matrix. The finite state machine contains four predefined core states: Idle state (IDLE): This represents the initial state in which a special vehicle has not entered the perspective remapping corridor or has been determined by the timing state determination module to be an irrelevant target. Approaching: This describes the process state in which the special vehicle is located in the perspective remapping corridor, has passed the dynamic tolerance check in step S4, and is continuously moving towards the work point. Verification state (VERIFYING): This indicates that the special vehicle has reached a preset radius near the operation and parking pixel, and its instantaneous image velocity vector magnitude has decayed to below the stationary threshold, entering the transition state of final position fine-tuning. COMPLETED: This indicates that the special vehicle has maintained the verification state for a sufficient period of time and meets the geometric constraint of vanishing point consistency. The timing state determination module officially confirms it as the termination state of the effective operation.

[0063] The timing status determination module reads the remaining approach distance output in steps S3 and S4 in real time, frame by frame. Instantaneous image velocity vector And the dynamic tolerance verification results. To address the issue of relying solely on the remaining approach distance... and instantaneous image velocity vector To address the technical challenge of accurately identifying the parking posture of special vehicles (i.e., whether they are precisely parallel to the aircraft fuselage), the timing state determination module introduces a vanishing point consistency constraint algorithm based on the principle of perspective convergence of parallel lines. This algorithm utilizes the projective geometry property that a group of parallel lines in three-dimensional space, after perspective projection, will inevitably converge to the same point (i.e., the vanishing point) on the two-dimensional image plane. If the special vehicle is parked parallel to the aircraft strictly according to the standard operating procedure, the intersection point of the extension lines of the special vehicle's own longitudinal contour lines (such as the edge of the roof and the side edge of the carriage) on the image (i.e., the vanishing point of the observed vehicle) should coincide with the theoretical vanishing point (i.e., the theoretical reference vanishing point) of the aircraft's longitudinal axis on the image within the allowable error range.

[0064] Based on this principle, the timing state determination module first uses the perspective projection matrix obtained in step S1. Reverse derivation of the theoretical reference vanishing point Due to the aircraft's world coordinate system The axis is strictly defined as the longitudinal axis of the fuselage, and its direction vector can be represented as an ideal point located at infinity in the homogeneous coordinate system. The timing state determination module performs the following matrix multiplication operation: ; in: The homogeneous coordinates of the theoretical reference vanishing point in the homogeneous coordinate system are expressed in the form of: This is used for subsequent dehomogenization calculations. express A perspective projection matrix of a dimension is used to map three-dimensional world coordinates to a two-dimensional image plane. This represents the homogeneous coordinate point at infinity along the direction of the aircraft's fuselage axis. express The specific numerical vector, of which the first three components The fourth component represents the unit vector of the fuselage's direction along the X-axis in the world coordinate system. This represents the point at infinity (direction vector).

[0065] After the calculation is completed, the timing state determination module determines the homogeneous coordinates of the obtained theoretical reference vanishing point. Perform dehomogenization, i.e., compute... (Assuming the third component is a scaling factor), the theoretical vanishing point in the two-dimensional pixel coordinate system is obtained. .

[0066] Meanwhile, the temporal state determination module needs to extract the actual vanishing points of the observed vehicles from the video stream. To reduce background clutter interference, the temporal state determination module uses the special vehicle bounding box output by the target detection unit as a basis to extract the region of interest. Within the region of interest, a straight line detection algorithm is used to extract all obvious straight line edges. Considering the structural characteristics of special vehicles, the temporal state determination module constructs a slope histogram, removes vertical lines perpendicular to the ground and irregular road surface textures, and filters out a set of candidate straight lines roughly parallel to the direction of travel of the special vehicles. Subsequently, a random sample consensus algorithm is used to robustly fit the candidate straight line set, and the optimal intersection point is calculated as the vanishing point of the observed vehicles. For the specific edge gradient calculation process of the line segment detection algorithm, and the random sampling and model evaluation iteration steps of the random sampling consensus algorithm, those skilled in the art can refer to classic textbooks or open-source library documents related to computer vision for implementation. These are well-known technologies in the field and will not be elaborated here.

[0067] To transform geometric deviations into quantifiable metrics usable for logical decision-making, the timing state determination module constructs a vanishing point alignment confidence level based on a Gaussian kernel function. Computational model: ; in: This represents the vanishing point alignment confidence level, which is... The normalized probability value of the interval, the closer the value is to 1, the higher the parallelism between the actual parking posture of the special vehicle and the aircraft fuselage axis. Represented by natural constant The exponential function operator with base 0.5 utilizes its nonlinear decay characteristics to enhance the discrimination of small errors. Representing vectors Norm (Euclidean norm) operations are used to calculate the length of a vector. It represents the square of the Euclidean distance between the observed vehicle vanishing point and the theoretical reference vanishing point on the image plane, and physically reflects the projection of the attitude angle deviation into perspective space. The standard deviation parameter, representing the vanishing point distribution, is used to adjust the sensitivity of the evaluation function to angular deviation; in this embodiment, The system is adaptively configured based on the image resolution and the field of view of the surveillance camera (for example, for a 1920x1080 resolution, it is preferably set to 50 to 80 pixels), and includes exception handling logic to prevent the denominator from being zero during the calculation process. This represents the constant term in the denominator of the exponential part of the Gaussian distribution probability density function, used to normalize the variance effect. The negative sign indicates a negative correlation, meaning that as the vanishing point distance (error) increases, the calculated confidence level should decrease.

[0068] The state transition logic of the finite state machine deeply integrates the aforementioned geometric constraints and spatiotemporal characteristics. The finite state machine will transition from the verification state to the completion state and lock the current timestamp if and only if the following three joint conditions are met. As the start time of the assignment: Kinematic rest condition: instantaneous image velocity vector The modulus length in continuous Within a time window of frames (e.g., set to 30 frames, corresponding to approximately 1 second), the value remains below the preset micro-motion and stillness threshold to eliminate transient jitter during emergency stops of special vehicles. Positioning conditions: Remaining approach distance The distance is less than the preset physical parking radius threshold (e.g., a pixel distance corresponding to 0.5 meters in physical space) to ensure that special vehicles have entered the effective operating radius; Geometric pose condition: calculated vanishing point alignment confidence. The orientation must be greater than the preset acceptable threshold (e.g., 0.85) to ensure that the parking position of special vehicles meets the parallelism requirements of the standard operating procedure.

[0069] Once the system enters the completion state, the timing status determination module will generate a structured event record containing the special vehicle ID, operation type, start time, and final docking accuracy, and synchronize it to the aircraft ground service management system via an external API interface. Conversely, if the vanishing point alignment confidence level is within the maximum allowable dwell time of the verification state... If the threshold is not reached, or if the special vehicle accelerates away again, the finite state machine will revert to the approach or idle state based on the latest position of the special vehicle, and trigger a maintenance alarm for abnormal parking posture. This multi-dimensional constraint fusion judgment mechanism effectively improves the accuracy and robustness of automated operation perception in complex apron environments.

[0070] See attached document Figure 6 and attached Figure 7 The following, in conjunction with the accompanying drawings and the foregoing content, describes the specific execution process of this embodiment in the refueling operation scenario of the B737-800 aircraft.

[0071] Scene initialization and data configuration. This embodiment runs on an industrial control computer deployed at the edge of the tarmac. The system has a pre-installed 3D topology model library for the B737-800 aircraft. This model library defines the origin of the aircraft's world coordinate system as being located at the apex of the nose cone. The monitoring camera resolution is set to 1920×1080, and the frame rate is 25fps.

[0072] Step S1: Aircraft pose calculation and projection matrix generation. The system acquires real-time video streams, and a deep learning keypoint detection unit identifies the pixel coordinates of the nose, landing gear, and wingtips in the image. Using a nonlinear least squares optimization algorithm integrating a Huber robust kernel function, the rotation matrix of the camera relative to the aircraft's world coordinate system is calculated. Translation vector .

[0073] Step S2: Mapping the perspective operation corridor to the target point. The perspective operation corridor generation module reads the standard ground trajectory data of the right wing refueling operation and uses the perspective projection matrix. Map the center of the ground refueling panel to the operation docking pixel on the image. .

[0074] Step S3: Special Vehicle Detection and Motion Parameter Calculation. The target detection unit identifies a refueling truck entering the frame. Using Kalman filtering for tracking, the distance to the refueling truck's stopping pixel is calculated in real time. Remaining approach distance and instantaneous image velocity vector .

[0075] Step S4: Compliance determination based on dynamic tolerance gating. (See attached document.) Figure 6 To resolve the conflict between long-range false alarms and the risk of close-range collisions, the system introduces a dynamic tolerance mechanism. For example... Figure 6 As shown, the horizontal axis of this coordinate system represents the remaining approach distance. (Unit: pixels), the vertical axis represents the allowable velocity vector direction deviation threshold. (Unit: degrees). Based on this coordinate system, the monitoring area is divided into three key logical segments: Free access zone: When the vehicle is in When the system operates within a pixel-level area, it is in a free approach zone. This allows for larger angular deviations, facilitating initial positioning adjustments for the vehicle.

[0076] Dynamic Transition Zone and Control Tightening Point: As the vehicle continues to approach, it enters the middle gray area, the dynamic transition zone. Notably, there is a critical control tightening point approximately 60 pixels from the target.

[0077] Strictly controlled zone: Once a vehicle crosses the control tightening point, it enters... Figure 6 The left side is a dark gray, strictly controlled zone. At this point, the allowable deviation threshold curve drops sharply and eventually converges to the minimum tolerance reference (5°) set by the system.

[0078] If the system detects that the vehicle's travel angle deviation in a strictly controlled area exceeds the minimum tolerance standard (5°), it will immediately determine it as a high-risk operation and trigger an emergency braking alarm.

[0079] Step S5: Vanishing point alignment confidence check and event output. When the refueling truck's speed drops to zero near the target point, the system initiates timing state analysis. (Refer to Appendix) Figure 7 The horizontal axis represents the job time series (unit: frames), and the vertical axis represents the normalized vanishing point alignment confidence. This diagram illustrates the monitoring logic for the entire operation process.

[0080] First, the phase boundary line (X=90) is clearly marked in the diagram. This vertical dotted line represents the 90th frame on the timeline. Using this as the boundary, the system strictly divides the entire operation into the approach phase (in motion) on the left and the verification phase (stationary) on the right. Second, the horizontal black dashed line at Y=0.85 in the diagram defines the system's compliance judgment threshold (0.85).

[0081] The system distinguishes between two typical operating conditions based on the aforementioned spatiotemporal coordinate system: Operating Condition A: Compliant operation (parallel docking). For example... Figure 7 As shown by the solid line (corresponding to the legend of compliant operation (parallel parking)). During the approach phase before the phase dividing line (X=90), the curve shows a fluctuating upward trend. After crossing the dividing line and entering the verification phase (after coming to a complete stop), the vanishing point alignment confidence level is high because the vehicle remains parallel to the fuselage. The value remained stable above 0.9. The system detected that the curve consistently exceeded the compliance threshold (0.85) during the verification phase, thus determining that the operation was compliant.

[0082] Condition B: Violation of operating procedures (angle deviation). For example... Figure 7 The dashed line indicates a violation (corresponding to the illustration of skewed angle). In this scenario, although the vehicle crossed the stage boundary line (X=90) and came to a stop, the calculated confidence score remained below the compliance threshold (0.85) due to the vehicle's skewed angle. Based on this, the system identified the risk and refused to confirm the operation as valid.

[0083] Reference Appendix Figure 6 and attached Figure 7 This embodiment verifies the technical advantages of the present invention in refined apron management through specific quantitative data and time-series curves: First, the tiered dynamic control strategy clearly defines the security boundaries. For example... Figure 6As shown, this invention discretizes the workspace by setting a control tightening point. Before the control tightening point, the system grants the driver ample freedom of movement; however, once the point is crossed and the vehicle enters a strictly controlled area, the system forces the vehicle's trajectory to converge to the minimum tolerance reference (5°). This design effectively prevents the risk of collision caused by the vehicle cutting in at a large angle at the last moment, balancing operational efficiency and safety.

[0084] Secondly, temporal feature analysis eliminates the blind spots of single-point detection. For example... Figure 7 As shown, this invention constructs a two-dimensional judgment coordinate system using a stage boundary line (X=90) and a compliance judgment threshold (0.85). Unlike traditional methods that only detect whether a vehicle has been reached, this invention mandates a secondary confirmation during the verification phase (after the vehicle has come to a complete stop) after crossing the stage boundary line (X=90). By comparing the distribution characteristics of the curves for compliant operation (parallel parking) and non-compliant operation (angle deviation) relative to the threshold line, accurate verification of the parking posture is achieved. This design ensures that the system only allows the vehicle to proceed when it has truly come to a complete stop and meets the posture requirements, reducing human error caused by observation angle errors.

Claims

1. An intelligent analysis and automatic detection method for apron video images, characterized in that, Includes the following steps: Process real-time video image sequences of the apron, establish the mapping relationship between the two-dimensional image pixel coordinate system and the aircraft world coordinate system, and generate a perspective projection matrix; Using the perspective projection matrix, the three-dimensional work space constraints defined in the aircraft world coordinate system are projected onto the two-dimensional image pixel coordinate system to generate perspective remapping corridors and work docking pixels; Determine the ground contact reference point of the work vehicle located in the perspective remapping corridor, calculate the remaining approach distance of the ground contact reference point relative to the work docking pixel, and the instantaneous image velocity vector of the work vehicle; Based on the remaining approach distance, the velocity vector direction deviation threshold is dynamically adjusted, and the velocity vector direction deviation threshold is used to verify the direction consistency of the instantaneous image velocity vector, thereby generating a compliant approach signal; The theoretical reference vanishing point is obtained by projecting the point at infinity of the longitudinal axis of the fuselage in the world coordinate system of the aircraft onto the plane of the two-dimensional image pixel coordinate system using the perspective projection matrix. In the real-time video image sequence, extract the edge line segments within the area where the work vehicle is located, and fit the intersection point of the line parallel to the direction of travel of the work vehicle as the observation vehicle vanishing point in the two-dimensional image pixel coordinate system; Based on the distance deviation between the observed vehicle vanishing point and the theoretical reference vanishing point, the vanishing point alignment confidence is calculated using a Gaussian kernel function. Based on the compliant approach signal, a finite state machine model is used to integrate the vanishing point alignment confidence, kinematic static state, and positional status of the work vehicle, and output a work event representing the intelligent analysis and automatic detection results of the real-time video image sequence.

2. The method for intelligent analysis and automatic event detection of apron video images according to claim 1, characterized in that, The specific steps of dynamically adjusting the velocity vector direction deviation threshold based on the remaining approach distance include: Set a minimum hard tolerance boundary and a maximum relaxation tolerance boundary, construct a distance mapping function, and set the distance mapping function to control the velocity vector direction deviation threshold to nonlinearly converge from the maximum relaxation tolerance boundary to the minimum hard tolerance boundary as the remaining approach distance decreases; Substitute the remaining approach distance into the distance mapping function to calculate the allowable velocity vector direction deviation threshold at the current moment.

3. The method for intelligent analysis and automatic event detection of apron video images according to claim 2, characterized in that, The distance mapping function adopts an exponential decay model, which calculates the exponential function value of the product of the remaining approach distance and the spatial convergence coefficient, and then weights the difference between the maximum relaxation tolerance boundary and the minimum hard tolerance boundary.

4. The method for intelligent analysis and automatic event detection of apron video images according to claim 1, characterized in that, The steps for verifying the directional consistency of the instantaneous image velocity vector using the velocity vector direction deviation threshold specifically include: The perspective projection matrix is ​​used to map the standard approach path tangent in the three-dimensional space under the aircraft world coordinate system to the standard approach path tangent direction under the two-dimensional image pixel coordinate system. The inverse cosine of the dot product between the instantaneous image velocity vector and the tangent direction of the standard approach path is calculated to obtain the directional angle deviation; Determine whether the directional angle deviation is less than or equal to the velocity vector direction deviation threshold. If the directional angle deviation is less than or equal to the velocity vector direction deviation threshold, then the directional consistency check is deemed to have passed.

5. The method for intelligent analysis and automatic event detection of apron video images according to claim 1, characterized in that, The specific steps for integrating the vanishing point alignment confidence, kinematic static state, and positional arrival state of the work vehicle using a finite state machine model include: Initialize the finite state machine model, and set the finite state machine model to include at least an approach state, a verification state, and a completion state; When the finite state machine model is in the verification state, determine whether the following three conditions are met simultaneously: Condition 1 is that the magnitude of the instantaneous image velocity vector remains below the micro-motion threshold within a preset time window; Condition two is that the remaining approach distance is less than the physical docking radius threshold; Condition 3 is that the vanishing point alignment confidence level is greater than the attitude qualification threshold; If all three conditions are met simultaneously, the finite state machine model is switched to the completed state, and a job start event is output as the job event.

6. The method for intelligent analysis and automatic event detection of apron video images according to claim 4, characterized in that, Also includes: If it is detected that the work vehicle leaves the perspective remapping corridor along the opposite direction vector of the standard approach path, the finite state machine model is reset and the work end event is output as the work event.

7. The method for intelligent analysis and automatic event detection of apron video images according to claim 1, characterized in that, The specific steps for generating the perspective remapping corridor and operation docking pixels are as follows: The three-dimensional work space constraints are defined as including the three-dimensional physical boundary vertices located in the aircraft world coordinate system to define the standard work area, and the three-dimensional docking target coordinates to define the theoretical stopping position. The three-dimensional physical boundary vertices are obtained and converted into homogeneous coordinates. The homogeneous coordinates are transformed, depth validity is checked and dehomogenized using the perspective projection matrix to obtain two-dimensional projection vertices located in the two-dimensional image pixel coordinate system. The two-dimensional projection vertices are connected according to the topological order to generate the perspective remapping corridor covering the real-time video image sequence. The perspective projection matrix is ​​used to perform perspective transformation on the coordinates of the three-dimensional docking target point, and the coordinates of the three-dimensional docking target point are mapped to the two-dimensional image pixel coordinate system to generate the operation docking pixel point.

8. The method for intelligent analysis and automatic event detection of apron video images according to claim 1, characterized in that, The steps for calculating the remaining approach distance of the ground contact reference point relative to the work docking pixel, and the instantaneous image velocity vector of the work vehicle are as follows: Detect the target object that is identified as the working vehicle and enters the perspective remapping corridor, and extract the center of the bottom edge of the bounding box of the target object as the ground contact reference point; The method of ray intersection is used to determine whether the ground contact reference point is located within the geometric range of the perspective remapping corridor; If the ground contact reference point is located within the geometric range, the Euclidean distance between the ground contact reference point and the work stop pixel is calculated as the remaining approach distance; The pixel displacement vector of the ground contact reference point between adjacent frames of the real-time video image sequence is recorded, and the instantaneous image velocity vector of the working vehicle is calculated by combining the frame rate of the real-time video image sequence.

9. The method for intelligent analysis and automatic event detection of apron video images according to claim 1, characterized in that, The specific steps for establishing the mapping relationship between the two-dimensional image pixel coordinate system and the aircraft world coordinate system are as follows: A deep learning network is used to extract the heat map response of key aircraft points from the real-time video image sequence, and the two-dimensional observation coordinates of the key aircraft points located in the two-dimensional image pixel coordinate system are obtained by decoding through differential expectation operation. The two-dimensional observation coordinates are matched with the three-dimensional physical coordinates in the aircraft world coordinate system in the preset three-dimensional topology model to construct a set of 2D and 3D point pairs. The camera's rotation matrix and translation vector are solved by minimizing the reprojection error, and the perspective projection matrix connecting the two-dimensional image pixel coordinate system and the aircraft world coordinate system is constructed by combining the camera's intrinsic parameters.

Images